Testing whether personalisation improves subscriber satisfaction

Compare a defined personalised-box offer with the existing box. Ask both groups the same satisfaction question after they have had a reasonable chance to receive and use it.

Decide before the test what improvement would matter. Record whether each intended box was actually delivered. Offering a preference form is not itself a satisfaction result.

Define the change and outcome

Test one manageable change, such as a choice between two eligible variants. Where possible, keep the core box, price, delivery schedule and survey timing the same. If those also change, attributing any difference to personalisation becomes harder.

Choose a primary question both groups can answer, such as ‘How well did this box suit your needs?’ Use identical wording and response options. Record supporting signals such as unwanted items, content-related support contacts and correction requests. Check the reason for any skip or cancellation; it is not automatically a verdict on the product choice.

Set the survey timing in advance. Some subscribers may not receive a box because of payment, stock or delivery problems. Record those events rather than silently removing those customers from the comparison.

Make the comparison fair

Where feasible and appropriate, randomly assign eligible subscribers to the existing and personalised offers, while honouring requirements that apply to every box. Random assignment helps reduce selection bias, but the test still needs enough participants and usable outcome data. Track missing survey answers for each group and report them alongside the results.

If random assignment is impractical, compare similar subscribers over the same cycles and describe the result as observational. Volunteers for personalisation may already be more engaged. A before-and-after change might also reflect a different product or delivery experience; neither pattern alone establishes that personalisation caused it.

Steps for a Fair Personalisation Test

Report assignment, delivery and response

For each group, report how many subscribers were eligible, assigned, received the intended box and answered the survey. Compare the assigned groups as the main test of the offered change, with non-receipt and missing answers visible. You can also describe satisfaction among recipients, but conditioning on receipt can make the groups less comparable if delivery problems differ.

Show the response distribution, not just an average. Check late parcels, unmet choices and unavailable variants.

If a preference did not reach packing, that is an implementation problem relevant to the offer's real-world effect; it does not reveal how a correctly personalised box would have been rated. Comments may explain a repeated complaint, but survey respondents may not represent everyone assigned.

Decide whether to keep, revise or stop the option using the observed satisfaction, delivery accuracy and added cost. State the sample size, response rate and limits beside any conclusion.