Sample Size

Sample size is the number of visitors, sessions, or users included in each group of an experiment, and it is the single most important determinant of whether a test can produce a trustworthy answer. It is calculated before a test launches, from four inputs: the baseline conversion rate of the control, the minimum effect size the business would consider meaningful, the desired statistical power, and the significance level. Change any of those inputs and the required sample size moves, often dramatically.

The relationship between these inputs is not intuitive, and this is where most testing programs go wrong. Detecting a small effect requires far more traffic than detecting a large one, and the requirement grows roughly with the inverse square of the effect size. Halving the effect you want to detect quadruples the traffic you need. A site converting at 2 percent that wants to reliably detect a relative improvement of 5 percent needs tens of thousands of visitors per variant; the same site trying to detect a 20 percent improvement needs a small fraction of that. Similarly, low baseline conversion rates demand more traffic than high ones, which is why lead generation sites and high-consideration e-commerce categories run slower testing programs than high-frequency retail.

Running a test without a sample size calculation produces one of two failure modes. The underpowered test never reaches a conclusion, consumes weeks of traffic, and is abandoned as inconclusive, or worse, is read as evidence that the change did nothing. The test stopped early, when a lucky run of conversions crossed a significance threshold, reports a large effect that mostly disappears when the change is rolled out to everyone. Both failures cost the same thing: the credibility of the testing program inside the organization.

Sample size requirements also constrain what a site can realistically test. A site with limited traffic cannot run a portfolio of small refinements, because each one would take months to resolve. The appropriate strategy at that traffic level is to test fewer, bolder changes with large expected effects, to test higher up the funnel where volumes are greater, and to lean more heavily on qualitative research and heuristic analysis to decide what to ship without an experiment. Recognizing this constraint honestly is far more productive than running a program of tests that mathematically cannot succeed.

Practical calculation also has to account for the difference between the statistical unit and the business unit. If the randomization unit is the visitor but the metric is measured per session, the arithmetic breaks down and the reported confidence is overstated. Tests measuring revenue per visitor rather than a binary conversion need larger samples still, because revenue is a high-variance metric with a long tail of large orders. Segment-level analysis compounds the problem: a test powered for the whole audience is not powered for the mobile subset, and reading segment results from an experiment sized for the total is one of the most common sources of false conclusions in the field.

Two practical questions follow from the calculation and are frequently answered wrongly. The first is whether traffic should be split evenly. An uneven allocation, such as ninety percent control and ten percent variant, is sometimes proposed to limit exposure to a risky change, but it extends the required runtime substantially, because the smaller group determines the precision of the comparison. Limiting risk is better achieved through a short initial ramp with guardrail monitoring, followed by an even split. The second is what to do about running several tests at once. Concurrent experiments on unrelated parts of a site are generally acceptable and are how most programmes achieve reasonable velocity, but tests that interact, such as two changes affecting the same funnel step, either need to be sequenced or analyzed as a combined design, since visitors experiencing both cannot be attributed cleanly to either.

Within a structured conversion rate optimization program, sample size planning happens during prioritization, not at launch. Knowing that a proposed test needs six weeks of traffic changes its priority relative to an idea that resolves in ten days, and that trade-off between expected impact and time to answer is one of the core inputs to the roadmap. For lower-traffic sites, a UX audit often delivers a better return than an experimentation backlog, because it identifies defects severe enough that fixing them does not require statistical proof.