Statistical Power

Statistical power is the probability that an experiment will detect a genuine effect of a given size, if that effect actually exists. A test with 80 percent power, the most widely used convention, will correctly identify a real improvement of the specified size four times out of five and will miss it the remaining one time in five. Power is the counterpart of the significance level: significance controls how often the test cries wolf, power controls how often it fails to notice the wolf.

Power is decided in advance, together with the significance level, the baseline conversion rate, and the minimum detectable effect, and together these determine the required sample size. Raising the power requirement from 80 to 90 percent increases the traffic needed substantially, which is why 80 percent has become the default in commercial testing even though higher standards are common in clinical research. The choice reflects a business trade-off between the cost of missing a good idea and the cost of the traffic spent looking for it.

Underpowered testing is the most widespread and least visible defect in optimization programs. Tests that run for a fixed calendar period rather than to a calculated sample size are underpowered by default whenever traffic is lower than assumed. The consequence is not just wasted effort, but systematically misleading conclusions: an underpowered program produces a stream of inconclusive results that get recorded as "no effect," which over time creates a false institutional belief that a whole category of changes does not matter. Real opportunities are abandoned because the measurement instrument was never sensitive enough to see them.

There is a second, more insidious consequence. In an underpowered experiment, any result that does reach significance must, by arithmetic, show a large observed effect. This inflates the apparent size of every win the program reports, a phenomenon sometimes called the winner's curse. A program running underpowered tests will therefore report a portfolio of dramatic wins, none of which reproduce at the reported magnitude when rolled out, and the gap between projected and realized revenue is what eventually destroys trust in the practice.

Improving power does not always mean waiting for more traffic. Choosing a metric higher in the funnel with a larger base rate increases power for the same number of visitors. Reducing variance by using a covariate-adjusted analysis, or by restricting the test to a more homogeneous audience segment, can meaningfully shorten runtime. Testing bolder changes with larger expected effects raises the effect size the test needs to detect and therefore lowers the sample requirement. Each of these is a design decision made before launch, and each is more productive than extending an already inconclusive test in the hope that the numbers turn.

Variance reduction deserves more attention than it usually receives, because it improves sensitivity without requiring more traffic. The best-established technique uses pre-experiment data about each visitor, such as their behavior in the weeks before the test, as a covariate in the analysis, removing variation that has nothing to do with the treatment. Where a substantial proportion of traffic is returning visitors with history, this can reduce the required sample meaningfully, which shortens runtimes across an entire programme rather than improving a single test. Choosing a less variable metric has a similar effect: binary conversion is less noisy than revenue per visitor, which is dominated by a small number of large orders, so a test powered on conversion and reporting revenue as a secondary measure often reaches a conclusion far sooner than one powered on revenue directly.

In consultancy practice, power analysis is where a testing roadmap becomes honest about what it can deliver. Before a conversion rate optimization program starts, calculating the achievable power at current traffic tells the client how many experiments per quarter are realistic and which questions will need a different method entirely. For sites where the arithmetic does not support a continuous experimentation cadence, the more effective route is usually structured research and expert evaluation, delivered through a UX audit, with experimentation reserved for the small number of high-traffic, high-value flows where the numbers do work.