Test Peeking

Test peeking is the practice of repeatedly checking an experiment's results while it is still running and stopping it as soon as a significance threshold is crossed. It is the most common way that valid statistical methods are turned into invalid conclusions, and it is almost always done with good intentions by people who believe they are simply monitoring progress. The mathematics of the fixed-horizon test assumes a single analysis at a pre-calculated sample size; every additional look is another chance for random fluctuation to cross the line, and the real false positive rate climbs with each one.

The scale of the distortion surprises people. A test intended to operate at a 5 percent false positive rate, checked daily over a few weeks with a stopping rule of "significant means done," can end up with a real false positive rate several times higher. The program then accumulates a portfolio of wins, a substantial share of which are noise, and the aggregate business impact fails to match the sum of the reported effects. Because each individual test looked properly analyzed, the cause of the shortfall is rarely identified, and the usual conclusion is that optimization simply does not work at this company.

Peeking also biases the size of the effects that survive. A test stopped at the moment significance is reached is stopped at a moment when random variation happens to favor the variant, so the observed lift at that point systematically overstates the true effect even when the effect is real. Programs that stop early therefore report larger wins than they deliver, which damages credibility over time in a way that is difficult to recover from once finance has noticed the gap.

The pressure to peek is structural rather than personal. Testing platforms display live significance figures on their dashboards. Stakeholders check them. A variant showing a large early lead creates real commercial pressure to ship immediately, and a variant showing an early decline creates pressure to stop and cut losses. Telling people not to look is ineffective; the practical solutions are procedural and technical. Fix the sample size and end date in writing before launch. Use sequential or Bayesian methods that are designed for continuous monitoring so that looking is legitimate. Configure dashboards to hide significance until the planned sample is reached. Reserve early stopping for genuine emergencies detected by guardrail metrics, with a named decision-maker and a documented rationale.

There is an important distinction between peeking and monitoring. Watching error rates, page performance, sample ratio, and severe guardrail breaches during a test is not only permissible but necessary, because a broken or damaging variant should be stopped immediately. What corrupts the analysis is using the primary business metric as a stopping criterion outside the pre-registered plan. Separating operational monitoring from outcome analysis, with different dashboards and different permissions, is often enough to resolve the problem in practice.

When a stakeholder demands to know how a test is performing before it concludes, refusing outright is usually counterproductive and giving the number undermines the protocol. The workable middle position is to report on operational health rather than on outcome: confirming that the test is running correctly, that traffic is splitting as configured, that no guardrail has been breached, and how many days remain until the planned analysis. This answers the legitimate concern behind the request, which is normally about risk rather than curiosity, without exposing a number that will be acted upon. Agreeing this arrangement in advance, as part of the experiment protocol, is far easier than negotiating it in the middle of a test that appears to be winning, when the commercial pressure to ship immediately is at its highest and the argument for patience is at its least welcome.

Establishing this discipline is one of the least technical and most valuable interventions available to an optimization program. In a typical CRO service engagement, the analysis protocol, the stopping rules, and the emergency-stop authority are documented before the first test launches, and results are published on a fixed cadence rather than on demand. Where an organization needs continuous visibility for legitimate commercial reasons, the answer is to adopt a sequential design that supports it rather than to run fixed-horizon tests and hope that nobody looks.