P-Value

A p-value is the probability of observing a result at least as extreme as the one measured, assuming that the change being tested had no real effect. It is the number underlying most significance verdicts in conversion experiments: a p-value below 0.05 is conventionally treated as evidence against the assumption of no effect, and the result is labelled significant at the 95 percent level.

The definition contains a conditional that is routinely dropped in practice, and dropping it causes most of the confusion around the concept. A p-value of 0.03 does not mean there is a 3 percent chance the change had no effect, and it does not mean there is a 97 percent chance the variant is better. It describes how surprising the data would be in a world where the change did nothing. Translating that into a probability about the hypothesis itself requires additional information, specifically how plausible the hypothesis was before the test, which is precisely what Bayesian methods make explicit and frequentist p-values leave out.

This matters commercially because it changes how results should be weighted. A p-value of 0.04 on a well-reasoned hypothesis grounded in observed user behavior is much stronger evidence than the same p-value on an arbitrary change tested on a whim, even though the two numbers are identical. Programs that test large volumes of low-quality ideas will accumulate significant results at roughly the rate the threshold implies, one in twenty by chance alone, and will then struggle to understand why their aggregate business impact does not match the sum of their reported wins.

The 0.05 threshold itself has no scientific basis; it is a convention inherited from early twentieth-century statistical practice and adopted wholesale by the optimization industry. Treating it as a bright line encourages a particular kind of bad reasoning where a p-value of 0.049 is a win and 0.051 is a failure, when the two results are practically identical. Better practice is to report the p-value alongside the effect size and confidence interval and let the decision reflect the full picture, including the cost of the change and the consequences of being wrong.

Several common practices inflate p-values in ways that are invisible in the final report. Repeatedly checking results and stopping when the threshold is crossed dramatically raises the real false positive rate. Testing many metrics and reporting the one that reached significance does the same. Slicing into segments after seeing the totals is the same error again, at a larger scale. Correction methods exist for multiple comparisons, and sequential testing designs exist for continuous monitoring, but the most reliable protection is a pre-registered analysis plan that fixes the primary metric, the segments, and the stopping rule before any data arrives.

Examining the distribution of p-values across a programme's history is a revealing diagnostic that few teams perform. If experiments were being run and analyzed correctly, and if most tested ideas had no real effect, p-values across the portfolio would be roughly uniformly distributed, with about one in twenty falling below the conventional threshold. A distribution heavily clustered just below the threshold is a strong indicator of stopping rules driven by the numbers rather than by the plan, since tests halted the moment they cross a line accumulate exactly at that line. Reviewing a year of past experiments this way costs a few hours and frequently explains a discrepancy that has puzzled a business for much longer, namely why a portfolio of reported wins has not moved the metrics that leadership actually watches.

For a consultancy, the useful contribution is usually less about calculating p-values correctly, which every testing platform does automatically, and more about governing how they are used. A CRO service engagement normally establishes a written analysis protocol, defines primary and secondary metrics per experiment, and reports outcomes in a consistent format that includes the effect size and interval rather than a lone verdict. In larger organizations, this protocol becomes part of how the data analytics function certifies results before they are quoted in board reporting, which is what prevents a portfolio of statistically fragile wins from being presented as a reliable revenue forecast.