We change the photo, the headline, the audience and the offer. The result improves. Which of the changes mattered?
When you have changed many things and still do not know what mattered, it is natural to want a clearer conclusion. A well-defined test should help you make a decision, including when the result remains uncertain.
From this observation, taken alone, we cannot separate the contribution of each element. We can see that the new combination had a different result under those conditions. For a more precise conclusion, we need a properly prepared comparison.
Testing helps when it produces information that we can use in a future decision. The number of published variants says little about the quality of that information.
What do we want to find out?
“Which ad works better?” is a starting point. Be specific about the goal: more orders, more relevant inquiries, or learning which argument deserves further development?
For a fictional bag, we can ask whether showing the compartments helps people understand the product. Or we can compare a direction about organization with one about weekend trips. These are two different questions, needing different tests.
Element or complete concept?
The question determines the type of test.
Illustrative example. The conditions are explained in the article.
Message test
- Change the headline.
- Keep other conditions as comparable as possible.
- Assess the wording within that setting.
Concept test
- Change the message and visual direction.
- Compare two coherent packages.
- The conclusion concerns the package, not a single element.
| Type of comparison | What we change | What we can follow |
|---|---|---|
| Message test | Headline, keeping other conditions as comparable as possible | The difference associated with the two wordings in that setting |
| Concept test | Message and presentation as a coherent package | Which complete direction is best suited to the objective |
A concept test can legitimately change several elements. The conclusion must concern the complete versions compared. We cannot credit the photograph alone when the offer changed too.
What we set before publication
We note the hypothesis, the variants, the audience, the main indicator and the evaluation conditions. We clarify what we keep comparable and what differences we cannot control.
For an inquiry campaign, cost per form may be an intermediate measure. If the goal is suitable customers, include qualification, contact and sales progress. Otherwise, you may choose the version that produces cheap forms but few useful conversations.
We also set out the decision that will follow: continue, expand, rework or close the exploration. A test without a possible decision can consume budget without clarifying the project.
How long and how much data are enough?
The answer depends on the frequency of the result, its variation, the difference we want to detect and the test method. There is no universal number of days or conversions that validates every comparison.
One extra order in a small sample can be chance. Promotions, holidays or limited stock can also affect the interpretation. When there is too little data, “we do not know yet” may be the right conclusion.
We also check how the variants are distributed. The platform may deliver the assets unevenly, and audiences may differ. A simple report comparison does not provide the control of an experiment.
What a sample size calculator can and cannot tell you
A sample size calculator helps once we have defined the metric, the relevant starting level and the smallest difference we want to detect. We also set the statistical risks we accept: how likely we are to detect a real effect of the chosen size, and how much risk of a false positive we will tolerate. These choices affect the volume needed. Microsoft Research: statistical power and metric sensitivity
No calculator can work out the ideal commercial budget for every ad on its own. The sample estimate has to be read alongside the likely cost of traffic, the duration, the available audience and the limits of the project. Future costs remain estimates, and a precise calculation will not fix a poorly chosen metric.
The unit being compared matters too. One person can see an ad several times. The number of impressions is not automatically the same number of people or independent observations. The method has to match how participants are assigned and how the result is measured.
A detectable difference also has to be worth the decision
In a hypothetical example, a very small increase in conversion rate may require more data than the business can collect in the time available. If the possible economic benefit is also small, it may be worth exploring a more important problem first. Conversely, an apparently large difference in a small sample can remain very uncertain.
The absence of a statistically significant difference does not prove the variants are identical. We report the estimated effect and its uncertainty against the original question. The final choice can also weigh production cost, ease of implementation or risk, with those reasons kept separate from what the test showed.
How we work when volume is low
We can test fewer questions at the same time, extend the period if the situation stays comparable, or look directly at the difficulties people run into. Conversations with customers, a review of inquiries and a walk through the journey can suggest useful changes. We use them as a different kind of evidence, without presenting them as statistical confirmation.
A clear functional error can be fixed without a conversion experiment. For choices that are still uncertain, we record the hypothesis and the limits of the decision. That way, low traffic becomes neither a reason to stall nor an excuse for certainty the data does not support.
We don’t test a mandatory requirement as if it were a preference
Meeting format requirements and checking that the link works are preparation checks. The creative test starts once the variants are usable for the chosen comparison. Otherwise, a difference in results may come from an incorrectly displayed asset, without telling us anything about the argument we want to evaluate.
In a system that automatically combines and distributes assets, uploading two headlines does not itself create an A/B test of the wording. Choose an appropriate method and state whether you are assessing one element, a complete concept or observed delivery. The conclusion should never be more precise than the evidence.
What we keep after the test
We document the question, the variants, the period, the observations and the limits. If we resume exploration in a few months, this information helps us understand why we made a decision and what has changed in the meantime.
At NOT MORE., tests should build our understanding of the business and its audience. We explain what the evidence supports, what deserves another look and where it is too early for a firm conclusion.
Three questions for the next test
- What decision do we want to make at the end?
- What difference are we trying to understand, and what conditions could influence it?
- How do we proceed if the result remains unclear?
What would you like to learn from the next test?
Describe the options and the decision you want to make. We can help define a test question and the information needed, based on the volume and resources available.