Sequential Testing

Sequential testing is a family of experiment designs that allow results to be evaluated continuously, or at planned intervals, while controlling the error rate that repeated evaluation would otherwise inflate. In a conventional fixed-horizon test, the analysis is valid only at the single, pre-calculated sample size; every additional look increases the chance of a false positive. Sequential methods adjust the decision boundary to account for the fact that multiple looks are happening, making it legitimate to stop a test early when the evidence is strong enough.

The commercial appeal is obvious. A fixed-horizon test that requires four weeks of traffic must run for four weeks even if the variant is severely damaging revenue in week one, unless the team is willing to abandon the statistical validity of the design. Sequential approaches allow a genuinely disastrous variant to be stopped quickly and a genuinely large winner to be shipped early, which both reduces the cost of testing and increases the number of experiments a site can run in a year. For businesses where a bad variant carries real revenue risk, this is not a refinement but a requirement.

Several techniques fall under the heading. Group sequential designs specify a small number of interim analyses with adjusted thresholds at each. Alpha-spending functions distribute the total allowable false positive rate across those looks. Always-valid inference and sequential probability ratio tests permit continuous monitoring with boundaries that tighten as data accumulates. Bayesian approaches sidestep the fixed-horizon problem by construction, though they carry their own considerations around stopping bias. Most commercial testing platforms now implement one of these under the surface, which is why their reported confidence figures no longer behave the way naive repeated significance testing would.

The important caveat is that sequential validity does not eliminate every risk of stopping early. A test halted after five days is still a test that has not seen a full weekly cycle, has not accumulated returning visitors, and has not outlived the novelty effect that a visible design change produces in its first days of exposure. The statistics may be valid while the business conclusion is still wrong, because the sample is unrepresentative of normal operating conditions rather than too small. Minimum runtime rules, typically at least one and preferably two full business cycles, sit alongside sequential boundaries rather than being replaced by them.

Early stopping also biases the estimated effect size. Tests that stop as soon as a boundary is crossed do so at moments when random variation happens to favor the variant, so the observed lift at the stopping point tends to overstate the true effect. Where the decision is simply whether to ship, this rarely matters. Where the number is going into a revenue forecast, it matters a great deal, and the reported figure should be discounted or re-estimated rather than quoted directly.

Because implementations differ, it is worth establishing what a testing platform actually does before relying on its verdicts. Vendor documentation should state whether the reported confidence is valid under continuous monitoring, which method is used, and whether the reported effect size is corrected for early stopping. Some platforms apply always-valid inference throughout, some apply group sequential boundaries at fixed checkpoints, and some still report naive fixed-horizon statistics on a live dashboard, which is the arrangement that produces the most false positives in practice. The distinction is not academic: the same data can yield a confident verdict under one method and no conclusion under another, and teams that have switched platforms sometimes discover that a large share of their historical wins would not have been declared under the new tool's methodology.

For a consultancy running a high-cadence program, sequential designs are what make an aggressive experiment velocity compatible with statistical discipline. In a conversion rate optimization engagement, the rules are normally documented up front: which boundary method applies, the minimum runtime regardless of boundary, the guardrail metrics that can trigger an emergency stop, and who has the authority to call one. That governance is particularly important in e-commerce environments, where a variant that damages checkout during a peak trading week can cost more in three days than the entire testing program returns in a quarter.