Sample Ratio Mismatch (SRM)

Sample ratio mismatch, commonly abbreviated SRM, occurs when the actual split of traffic between experiment variants differs from the intended split by more than random variation can explain. A test configured for an even 50/50 division that delivers 51,200 visitors to the control and 48,800 to the variant has not simply been unlucky; at that volume, a gap of that size is statistically implausible under correct randomization. SRM is detected with a chi-squared test against the expected allocation, and any test showing it should be treated as invalid until the cause is found.

SRM matters because it signals that the two groups are no longer comparable. The entire logic of a controlled experiment rests on random assignment producing groups that differ only in the treatment applied. If some systematic process is diverting particular visitors into one group, or dropping visitors from one group before they are counted, then any observed difference in conversion may reflect that process rather than the change being tested. The reported result is not merely imprecise; it is measuring something other than what the team believes it is measuring.

The causes are almost always technical rather than statistical. Slow-loading variant code causes visitors to leave before being counted, systematically removing the least patient users from one group. Redirect-based tests lose traffic during the redirect, and the loss is not evenly distributed across devices or connection speeds. Bot filtering applied inconsistently across variants, caching layers serving one version disproportionately, tracking calls firing before rendering completes in one variant but after in the other, and interactions between concurrently running experiments all produce SRM. Each of these is a real defect that would corrupt results whether or not anyone checked the ratio.

The appropriate response is to stop the test, diagnose the cause, fix it, and restart from zero. Attempts to salvage an SRM-affected test by reweighting the groups or by analyzing a subset are rarely defensible, because the missing or misallocated traffic is not missing at random. The instinct to rescue a test that has already consumed three weeks of traffic is strong, especially when it appears to show a win, but a result from a broken randomization has no evidential value and shipping on the basis of one is worse than not testing at all.

Because SRM is invisible unless someone looks for it, the check has to be automated. Mature programs run an SRM test on every experiment as part of the standard analysis pipeline, flag any deviation beyond a strict threshold, and refuse to publish results until the flag is cleared. Retrospective audits of past experiments frequently find a meaningful share of historical tests affected, which is one of the more uncomfortable discoveries a testing program can make, and one of the most valuable, because it explains why certain past wins never materialized in the business results.

Investigating a mismatch is a systematic process rather than a hunt. The first step is to check whether the imbalance is present in the testing tool's own assignment records or only in the analytics platform's, which distinguishes a randomization problem from a tracking problem. The second is to segment the ratio by device, browser, geography, and traffic source, since the cause frequently affects one group disproportionately and the segment showing the largest deviation usually points directly at the mechanism. The third is to check the timing, since a ratio that was correct initially and drifted later often indicates a deployment, a caching change, or a third-party script introduced mid-test. Documenting each investigation builds a catalogue of recurring causes, and most organizations find the same two or three mechanisms responsible for the majority of their incidents.

Introducing an automatic SRM check is usually one of the earliest technical recommendations in a CRO service engagement, because it costs very little to implement and immediately raises the reliability of everything the program produces afterwards. Where SRM traces back to variant rendering delay or redirect latency, the remedy sits with web performance work rather than with the testing tool itself, and fixing the underlying speed problem tends to improve both the validity of experiments and the conversion rate they are measuring.