Most A/B tests fail before a single visitor sees them. Not because the hypothesis was weak or the design was bad, but because nobody worked out whether the test could produce an answer at all.
The pattern is familiar. A team ships a variant, watches the dashboard, and after ten days sees the challenger up 6% with a p-value of 0.14. Someone says "it's trending positive, let's give it another week." Another week passes. Now it's up 4%, p = 0.09. The test gets called a "directional win" and shipped, or it gets killed as inconclusive. Either way, three weeks of traffic produced no knowledge — and that outcome was fully determined before launch by four numbers nobody checked.
This article is about those four numbers.
The four inputs
Sample size is not a judgement call. It falls out of four parameters, and every serious experimentation programme fixes them before launch rather than after.
| Input | What it means | Typical value |
|---|---|---|
| Baseline conversion rate | The current rate of the metric you're testing, measured on the exact page and audience the test will run on | Whatever your data says |
| Minimum detectable effect (MDE) | The smallest change you want the test to be capable of finding | 5–20% relative, depending on traffic |
| Significance level (α) | Your tolerance for declaring a winner that isn't real | 0.05 |
| Statistical power (1−β) | Your probability of detecting an effect that is real | 0.80 |
Give a calculator those four and it returns the visitors required per variant. Our free A/B test calculator does exactly this, and it takes about thirty seconds — which is the point. The reason teams skip this step is not that it's hard. It's that the answer is often unwelcome.
MDE is the input everyone gets wrong
Minimum detectable effect is the parameter that causes the most damage, because its name suggests something it isn't.
MDE is not the lift you expect. It is not a prediction, a target, or a forecast. It is a floor of detectability: the smallest true effect your test has an 80% chance of catching. Set MDE to 5% and you are saying "if the real improvement is smaller than 5%, I accept that this test will probably miss it."
Two consequences follow, and both matter.
A test can only answer the question you sized it for. If you size for a 20% MDE and the true effect is 7%, the test will most likely return "no significant difference." That is not evidence the variant does nothing. It is evidence your test could not see effects of that size — a completely different statement, and one that teams routinely mistranslate into "we tested it, it didn't work."
Relative and absolute MDE are not interchangeable. A baseline of 2.5% with a 10% relative MDE means detecting a move to 2.75% — an absolute difference of 0.25 percentage points. Enter 10 into a calculator field expecting absolute percentage points and you have just asked to detect a move from 2.5% to 12.5%, and the tool will cheerfully tell you 400 visitors will do. Check which convention your calculator uses. This single confusion invalidates more test plans than any other error we see.
A worked example
An e-commerce client wants to test a new product detail page layout. Baseline conversion on that page is 2.5%. They want to detect a 10% relative improvement — a move to 2.75%.
At α = 0.05 and 80% power, the requirement is roughly 64,000 visitors per variant, or about 128,000 in total.
That page gets 40,000 visitors a month. So the test needs a bit over three months to reach the sample size it was designed for.
This is the number that changes the conversation. Not because three months is impossible, but because it is now a decision made openly — before launch, with the trade-offs visible — rather than a slow disappointment discovered on day 21.
Now change one input. Keep the same baseline, but size for a 20% relative lift instead of 10%:
| MDE (relative) | Absolute difference | Visitors per variant | Time at 40k/month |
|---|---|---|---|
| 5% | 0.125 pp | ~256,000 | ~13 months |
| 10% | 0.25 pp | ~64,000 | ~3.2 months |
| 20% | 0.5 pp | ~16,800 | ~25 days |
| 30% | 0.75 pp | ~7,700 | ~12 days |
Halving the MDE roughly quadruples the sample. That relationship is not a quirk of these numbers — sample size scales with the inverse square of the effect size. It is the single most useful thing to internalise about experiment design, because it tells you where the leverage is: making the change bigger is far cheaper than gathering more traffic.
A timid variant that moves a button two pixels needs a sample size no mid-sized site can supply. A variant that restructures the page has a chance. This is why cautious testing programmes stall — not because the ideas are wrong, but because they are too small to measure.
Where the traffic actually is
The other lever hiding in the arithmetic is the baseline rate. Sample size requirements fall sharply as the baseline rises.
Take the same 10% relative MDE, but run it on the shipping step of the checkout, where 40% of arrivals continue to payment:
- Baseline 2.5% → ~64,000 per variant
- Baseline 40% → ~2,400 per variant
Same MDE, same confidence, same power. Twenty-six times less traffic.
That page sees far fewer visitors than the PDP, of course. But the requirement drops faster than the traffic does, which is why deep-funnel tests are frequently the only tests a mid-traffic site can actually complete. If your sitewide conversion is 2%, testing the homepage is a year-long commitment and testing the checkout is a fortnight.
The practical guidance: when the calculator returns an impossible number, before you widen the MDE, check whether there's a step further down the funnel where the same hypothesis can be tested against a higher baseline.
What actually goes wrong when you skip it
There are two failure modes here, and the second one is much less known than the first.
You miss real improvements. An underpowered test returns "no significant difference" for a change that genuinely works. The team concludes the idea was bad and abandons a direction that would have paid. This is the failure everyone knows about, and it's the less expensive of the two.
Your winners are inflated. This is the one that quietly corrupts a programme's numbers. In an underpowered test, the only way a result can clear the significance threshold is if random noise happened to push it far enough — which means the effects you do declare significant are systematically larger than the truth. The lower the power, the worse the exaggeration.
So an underpowered programme doesn't just miss things. It produces a test log full of winners with overstated uplifts, which then get multiplied by traffic to forecast revenue that never materialises. Six months later someone asks why the sum of the wins doesn't show up in the top-line number. This is usually the answer.
You cannot fix this in analysis. It has to be prevented at design time.
Per variant, not total
Calculator output is the requirement for each arm of the test. Two variants means double the number. This sounds obvious and is misread constantly, usually in the optimistic direction.
Multiple variants compound the problem twice over. An A/B/C/D test needs four times the sample of the single-arm figure, and three simultaneous comparisons against control inflate the chance of at least one false positive well beyond your nominal 5%. Correcting for that pushes the requirement higher again.
For most sites, the practical rule is: unless you have traffic to spare, run two arms. A four-way test on a page that can barely power a two-way test is not four times the learning. It is zero times the learning, delivered more slowly.
Duration is a separate constraint
Reaching your sample size is necessary. It is not sufficient.
A test that hits its number in four days has sampled four days of behaviour. Weekday and weekend traffic convert differently. Payday cycles matter in e-commerce. Paid campaigns launching mid-test change the audience mix underneath you. New designs also produce a novelty effect that decays — early engagement with an unfamiliar layout is not a stable estimate of steady-state performance.
Run for at least one full week, and preferably two, regardless of what the sample size calculation permits. Whichever constraint is longer — sample size or business cycle — is the one that governs. And set the end date in advance rather than checking each morning and stopping when the numbers look good, which is a separate and very effective way of manufacturing false winners.
When the number is impossible
Sometimes the honest answer is that the test cannot be run. That is useful information, and there are five reasonable responses.
Test a bigger change. The quadratic relationship makes this the highest-leverage option by a wide margin. Redesign the flow rather than the button.
Move to a higher-baseline metric. Test against add-to-cart or checkout entry rather than purchase. The effect size is measured against a larger base and the sample requirement collapses. Accept that you're measuring a proxy and validate the link to revenue separately.
Test further down the funnel, as above.
Group similar pages. Running one test across all category pages rather than a single category multiplies available traffic, provided the pages are behaviourally similar enough that pooling is defensible.
Stop testing and start researching. If a site cannot power an experiment, A/B testing is the wrong instrument for it. Usability testing, session recordings, and heuristic review will find the broken things faster and cheaper. Quantitative validation comes later, once the changes are large enough to be measurable. There is no shame in this — running underpowered tests to feel rigorous is worse than not running them.
Pre-flight checklist
Before any test launches:
- [ ] Baseline measured on the actual page and audience the test will run on, not sitewide
- [ ] MDE stated explicitly, in writing, with relative-or-absolute specified
- [ ] Sample size calculated per variant and multiplied by the number of arms
- [ ] Required duration compared against available traffic — and the answer accepted before launch
- [ ] End date fixed in advance
- [ ] Minimum one full business cycle, whatever the sample maths says
- [ ] Randomisation unit matches analysis unit (visitor-level assignment, visitor-level metrics)
- [ ] Traffic split verified in the first hours — a 50/50 test delivering 53/47 means an instrumentation problem, and the results are unusable until it's found
Run the numbers before you build the variant
The whole calculation takes half a minute and it changes what you build. Teams that size their tests first design bolder variants, because they can see what timid ones would cost.
Enter your baseline conversion rate, the effect you want to be able to detect, and your confidence and power settings. If the number that comes back is larger than your traffic, that is not a reason to launch anyway — it is the test telling you to change the plan.
Frequently asked questions
How many visitors do I need for an A/B test? There is no universal number. It depends on your baseline conversion rate, the smallest effect you want to detect, and your chosen significance and power levels. A page converting at 2.5% needs roughly 64,000 visitors per variant to detect a 10% relative improvement; a checkout step converting at 40% needs about 2,400 for the same relative effect.
What is a good minimum detectable effect? The lowest one your traffic can support within a reasonable timeframe. Start from your available traffic and a target duration of two to four weeks, then solve backwards for the MDE that fits. If that MDE is larger than the change you're testing could plausibly produce, redesign the test.
Can I stop a test early if it's clearly winning? Not with a standard fixed-horizon test. Repeatedly checking and stopping at the first significant result inflates your false positive rate far above 5%. If you need the ability to stop early, use a method built for it — sequential testing or a Bayesian framework — and decide that before launch, not during.
What if my test reaches significance before the sample size is met? Treat it with suspicion and keep running to the planned end. Early significance in an underpowered window is exactly the pattern that produces inflated, non-reproducible winners.
Is 80% power enough? It's the conventional default and it's a reasonable starting point, but it means accepting a 20% chance of missing a real effect. For high-stakes decisions — a checkout redesign, a pricing change — 90% power is worth the extra sample. For low-cost, reversible changes, 80% is fine.
Not sure your testing programme is producing real answers?
If your test log is full of small wins that never show up in the top-line number, the problem is usually upstream of the analysis. We review experimentation programmes and tell you plainly where the design is failing.
