Confidence Interval

A confidence interval is a range of values, calculated from experiment data, that is likely to contain the true effect of a change. Where a significance test returns a binary verdict, a confidence interval communicates both the direction and the plausible magnitude of an effect, along with the precision of the measurement. A result reported as "a 6 percent relative lift, 95 percent confidence interval from 1 percent to 11 percent" carries far more decision-relevant information than "significant at 95 percent."

The width of the interval is the key signal. A narrow interval means the experiment measured the effect precisely and the business can plan around the number. A wide interval that still excludes zero means the change was probably positive but its size remains genuinely uncertain, which matters when the decision to ship depends on the effect being large enough to justify implementation cost. An interval that spans zero means the data is compatible with the change helping, doing nothing, or hurting, and no confident claim in any direction is supportable.

Confidence intervals correct a specific and expensive habit: reporting the observed point estimate as though it were the truth. When a test reports an 18 percent lift with an interval from 2 percent to 36 percent, the headline number that reaches the executive summary is almost always 18 percent, and that number then appears in forecasts, business cases, and annual projections. When the change is rolled out and delivers 4 percent, the testing program is blamed for over-promising. Presenting the interval from the outset sets accurate expectations and protects the credibility of the practice.

The formal interpretation is more subtle than the everyday one. A 95 percent confidence interval does not mean there is a 95 percent probability that the true effect lies inside this particular interval. It means that a procedure producing intervals this way would capture the true value in 95 percent of repeated experiments. In practice, most teams reason about it in the everyday way and rarely come to harm, but the distinction matters when someone tries to combine intervals across tests or to reason about a specific result in probabilistic terms. Bayesian methods, which report credible intervals with the intuitive interpretation, are often preferred for exactly this reason.

Intervals also expose the cost of segment analysis after the fact. Slicing an experiment into mobile and desktop, new and returning, paid and organic, produces intervals that widen sharply as each subset shrinks. Segments with dramatic-looking point estimates and intervals spanning from strongly negative to strongly positive are the raw material of most false learnings in optimization work. Displaying the interval next to every segment figure makes the weakness of those readings immediately visible to non-specialists, which is more effective than a written caveat that nobody reads.

Communicating intervals to commercial stakeholders requires translating them into the terms of the decision rather than presenting them as statistical notation. The effective framing expresses the range in business units: rather than reporting a relative lift interval, state that the change is expected to produce between a lower and an upper figure in additional monthly revenue, given current traffic. Planning can then use the conservative end of the range, which produces forecasts that hold up, while the optimistic end sets expectations about the best realistic case. Where the interval includes zero, the honest statement is that the experiment could not establish whether the change helps or hurts, and the follow-up question is whether the business wants to spend more traffic narrowing it or to make the decision on other grounds such as brand, accessibility, or maintenance cost.

Reporting practice in a mature program therefore leads with the interval rather than the verdict. A data analytics function that standardizes on interval reporting across experiments, marketing measurement, and forecasting builds a shared organizational vocabulary for uncertainty, which in turn makes it easier to have honest conversations about which results are solid enough to build a strategy on. In growth management work, that honesty compounds: forecasts built on interval-aware estimates survive contact with reality far better than those built on cherry-picked point estimates from the most flattering test in the last quarter.