Your program reports a winner about once a month. Add twelve wins up and the annual deck says conversion should be 18% higher than last year. Finance looks at the P&L and says it isn't. Usually the deck is wrong.
On Microsoft's experimentation platform, only about one third of tested ideas improved the metric they were designed to improve, and the ones that get launched are selected precisely for having looked good, which is where the arithmetic starts to drift (Kohavi and Longbotham, 2015). A holdout group is the cheapest instrument that measures the program rather than the individual test.
This guide is for CRO leads and analytics owners shipping at least one experiment a month. By the end you will be able to size a holdout that can actually detect your program's effect, pick a duration and analysis window, and recognise the failure modes that void the result before you present it.
What a holdout group measures that a sum of wins does not
A holdout is a randomly selected slice of users kept on the pre-change experience while everyone else receives every launched winner. After some months you compare the two groups. The difference is the cumulative, still-alive effect of everything you shipped — not twelve short-window estimates stapled together.
Netflix uses the same instrument, describing "holdback AB tests" as a way to "understand long term effects, retest old assumptions on newer members, and measure cumulative value" (Netflix Technology Blog).
Why summed test wins overstate what happened
Four mechanisms pull the sum above the truth, and they compound:
- Winner's curse. You only launch variants that crossed significance, so the launched set is enriched with upward noise. Lee and Shen at Airbnb formalised this selection bias and published a correction (KDD 2018).
- Novelty and primacy decay. Short windows capture reactions to newness. In a Microsoft News example, an initial 28% lift in clicks on a new button fell away day over day inside the experiment window (Sadeghi et al., 2022).
- Interaction and diminishing returns. Two independently tested wins on the same funnel step frequently do not add; they compete for the same marginal user.
- Analysis slack. Peeking, metric switching and selective reporting each shift the launched set further upward (Eppo).
None of this means your tests were fake. It means the sum is an upper bound, not an estimate.
Size the holdout before you build it
This is the step nearly everyone skips, and it is where most holdouts die. A holdout is a two-arm test with a deliberately lopsided split, and precision is governed by the smaller arm. Required total sample scales with 1 / (p × (1 − p)), where p is the holdout share. A 1% holdout therefore needs roughly 25 times the traffic of a balanced test to see the same effect.
Total users needed across the holdout period, at a 2% baseline conversion rate, 80% power and 5% two-sided significance:
| Holdout share | Users needed to detect +5% relative | Users needed to detect +10% relative |
|---|---|---|
| 1% | 15,500,000 | 3,900,000 |
| 5% | 3,250,000 | 814,000 |
| 10% | 1,720,000 | 432,000 |
| 20% | 971,000 | 245,000 |
| 50% | 630,000 | 161,000 |
Now read that against the vendor defaults. Statsig suggests "a low single-digit holdout percentage, such as 1%–2%" (Statsig); Optimizely warns above 5% (Optimizely); GrowthBook reports Facebook and X teams running 5% or less (GrowthBook).
Those numbers are right for their authors and wrong for most mid-market sites. A regional e-commerce site doing 400,000 monthly users accumulates 1.2 million over a quarter. At a 1% holdout it would need roughly a decade. At 10% it still falls short of a +5% read. At 20% it detects +5% in a quarter and +10% in about a month. Pick the share from your traffic, not from a vendor's default. If the honest answer is 20%, put that trade-off to the business plainly: one fifth of users wait a quarter for new features, in exchange for knowing whether the quarter was worth anything.
Choose the duration and the analysis window
Statsig recommends three to six months before releasing a holdout; GrowthBook recommends an active period of half a month to a quarter, then a two to four week analysis period so effects settle and lookback windows fill.
Our working rule: the active period is the longer of what your sizing table requires and one full business cycle. In this region that means covering at least one campaign peak and one flat month, so the result is not decided by 11.11 alone. Then freeze the flags and wait out the analysis window before anyone opens the dashboard.
Build it: seven things to get right
- Randomise on a persistent, server-side user or account identifier. A cookie that a consent banner blocks or a user clears leaks people between arms.
- Assign the holdout once, up front, before the launches you want to measure. A holdout created after the fact is not a holdout.
- Route every launched winner through the same flag system, so "holdout = off" genuinely means the old experience. Held-out users get default gate and layer values, as Statsig's implementation does.
- Keep an explicit exclusion list: security fixes, legal and accessibility compliance work, and pricing changes ship to everyone. Document each exclusion with its date.
- Pick two to four durable metrics (revenue per user, retention, order frequency) plus your primary conversion rate. Do not add a fifth.
- Keep a dated launch register of every change shipped in the window with its measured lift. That is what you compare the holdout against.
- Tell engineering the flags stay in the codebase until the analysis window closes. Holdouts die from premature flag cleanup more often than from anything statistical.
Read the result honestly
Compute the ratio of holdout-measured lift to summed lift in your launch register. Our working thresholds — judgement calls, not published benchmarks:
| Holdout lift ÷ summed lift | Reading | Action |
|---|---|---|
| ≥ 0.6, interval excludes zero | Program is roughly honest | Report the holdout number externally, keep current forecasting |
| 0.3 – 0.6 | Normal shrinkage from selection and interaction | Apply the observed ratio as a haircut to future business cases |
| < 0.3, or interval includes zero | The sum is not evidence | Stop quoting summed lift; re-examine win criteria and stopping rules |
| Negative point estimate | Something shipped is costing money | Reverse-test the largest launches one at a time |
One regional caution about the alternative teams reach for first. If you plan to skip the holdout and compare this quarter with last year, note that Türkiye's annual consumer price inflation was 31.51% in August 2026 (TÜİK figures, published 3 September 2026). Revenue per user in lira is not comparable across periods at that rate, and deflating it adds its own assumptions. Concurrent randomisation sidesteps this: both arms live in the same economy on the same day.
[INTERNAL DATA NEEDED: a Switas client example — program length, traffic scale, summed test lift versus holdout-measured lift, and the holdout share used.]
Six ways a holdout quietly breaks
- Contamination. Shared devices, logged-out sessions and identifier resets move users across arms. Measure crossover; above roughly 2% the estimate is not trustworthy.
- Silent leakage. A change shipped outside the flag system (a CDN rule, a CMS edit, a new template) reaches the holdout too and shrinks the gap toward zero.
- Exclusion creep. Every "this one has to go to everyone" decision removes a launch from what you are measuring. Ten of them and you are measuring nothing.
- Sample ratio mismatch. The split drifts from its intended share. Check weekly; a holdout with SRM is as void as any other test with SRM.
- Premature release. Optimizely's native global holdouts cannot be paused, only ended. Take that as the rule: ending early forfeits the window.
- Underpowered from day one. The commonest failure. A null result from a 2% holdout tells you nothing about your program and everything about your sizing.
Where this method breaks down
A holdout gives the size of the combined effect, not which launch produced it. Attribution still needs the individual tests. It cannot measure anything that must ship to everyone, so compliance work sits outside it. It is a poor fit where the same users do not return within the window, and it says nothing about brand or offline spillover between arms. Below roughly 100,000 monthly users there is no share that is both detectable and defensible; at that scale, carefully controlled before-after reads plus qualitative validation are the honest fallback, with their weaknesses stated out loud.
Frequently asked questions
How big should a holdout group be?
Large enough to detect the cumulative effect you expect, which is a calculation rather than a convention. At a 2% baseline and a +5% relative target, a 5% holdout needs around 3.25 million users while a 20% holdout needs around 971,000.
How long should a holdout run?
Vendors suggest three to six months of active period plus a two to four week analysis window. Add the constraint that the window covers at least one campaign peak and one ordinary month, so a single sales event does not decide the result.
Why is my holdout lift smaller than the sum of my test wins?
That is the expected result, not a bug. Winner's curse, novelty decay, interaction between changes and analysis slack all inflate the sum. The holdout number is the one closer to reality.
Can I create a holdout after I have already launched things?
Not for those launches — everyone already received them. A holdout only measures changes gated from its start date. You can still start one today for the next two quarters.
Does holding users back hurt revenue?
Slightly, if your launches genuinely work; that cost is bounded by the holdout share times the true lift. If they do not work, the holdout costs nothing and saves you from repeating them.
What if the holdout result is flat?
Check power against your sizing table first, then crossover and leakage. A genuinely flat, adequately powered result means two quarters of shipping did not move the metric: expensive to learn, more expensive not to.
Next step
If you run experiments and cannot say what a year of wins was worth, size a holdout against your own traffic this week — or have us size, build and read it as part of a CRO engagement.
Sources
- Kohavi & Longbotham, "Online Controlled Experiments and A/B Testing" (2015)
- Lee & Shen (Airbnb), "Winner's Curse", KDD 2018
- Sadeghi et al. (Microsoft), "Novelty and Primacy", Technometrics, 2022
- Netflix TechBlog, "Causal Inference Applications at Netflix"
- Statsig Docs, "Holdouts"
- GrowthBook, "Holdouts in GrowthBook"
- Optimizely Support, "Global holdouts"
- Eppo, "The Whole Is Less Than the Sum of Its Parts"
- Directorate of Strategy and Budget — TÜİK CPI, August 2026
