Diagnose a Sample Ratio Mismatch in Five Branches Before You Throw the Test Away

About 6% of experiments at Microsoft show a sample ratio mismatch, which at a product running 10,000 experiments a year works out to roughly one detection a day (Fabijan et al., KDD 2019). Teams running far fewer tests rarely see that number, because the check their platform runs cannot reach significance at their traffic volume. The test looks clean, reports a 4% lift, and is wrong anyway.

This guide is for the person who reads the experiment readout and has to decide whether to ship. By the end you will be able to work out whether your test is large enough for the mismatch check to mean anything, run five branches of diagnosis in the order that finds the cause fastest, and separate an imbalance that voids the readout from one you can segment around and keep.

What an SRM alarm actually tells you

A sample ratio mismatch is a gap between the split you configured and the split you observe: you set 50/50 and got 50.8/49.2. The standard check is Pearson's chi-square goodness-of-fit test on the exposure counts, one degree of freedom for a two-arm test.

It matters because the mechanism that lost users is almost never neutral with respect to your metric. If a redirect drops slow connections from the variant, the survivors are the ones on better connections, and they convert better. The lift is real in the data and absent in reality.

Here is the first thing most teams get wrong: there is no industry threshold. Platforms disagree by more than an order of magnitude.

Two consequences. A test flagged on one platform passes on another with identical data, so "our tool did not warn us" is not evidence of a clean split. And a 0.01 threshold flags one in a hundred healthy tests. If you eyeball the p-value daily across a three-week test, the naive independent-peek bound on a false alarm is 19%; the true figure is lower because consecutive days are strongly correlated, but it is not 1%. That is why AB Tasty runs a sequential procedure. If your platform runs a fixed-horizon chi-square, look once, at the end, on cumulative counts.

Check whether your test can detect an imbalance at all

For a 50/50 split the chi-square statistic simplifies to 4n(p̂−0.5)², where n is total exposures. Invert it and you get the smallest imbalance your check can flag:

Read it as a limit, not a licence. At 10,000 exposures anything smaller than a 1.3 point imbalance is invisible to the check, and a 1.3 point selection effect is more than enough to manufacture a small conversion lift. That is the structural problem across most of the region. Turkey recorded about 6 billion e-commerce transactions across roughly 634,000 businesses in 2025 (Ministry of Trade, 12 May 2026), an average near 9,500 a year each, and that average is pulled up hard by a handful of marketplaces. A mid-market shop testing a checkout step works with tens of thousands of exposures, not millions, so its SRM check catches gross breakage and nothing subtler.

At that volume, do not rely on the alarm. Pull raw exposure counts by day and by segment yourself, and treat a persistent same-direction drift as suspicious even when the p-value is unremarkable. If you are not sure where your platform exposes those counts, the SRM Checker documentation maps them per tool.

The five branches, in the order that finds the cause fastest

The KDD taxonomy sorts SRM causes by lifecycle stage: assignment, execution, log processing, analysis, interference. Diagnose them in a different order, cheapest and most common first.

1. Analysis and triggering. Symptom: the split is clean in the raw exposure log and broken in the scorecard. Check: recompute the ratio without your trigger condition, date filter and metric-level joins. Confirms it if the ratio returns to expectation. Fix: evaluate triggering identically in both arms, before any variant-specific behaviour can change eligibility. Most common false alarm, ten minutes to rule out.
2. Log processing. Symptom: exposure counts differ between two pipelines, or the gap appears after the ETL step. Check: compare the SDK-side count, the warehouse count and the reporting count for the same window, then compare bot-filter exclusion rates per arm. Confirms it if the arms are equal upstream and unequal downstream. Fix: filter deterministically and identically. Automated traffic reached 53% of web traffic in 2026, up from 51% in 2024 (Imperva Bad Bot Report, 29 April 2026), so a heuristic filter keyed on load timing or user-agent behaviour can exclude at different rates per variant. Consent-driven loss sits here too: when ad_storage is denied, Google's tag platform stops setting advertising cookies (Google tag platform consent docs), so if denial rates differ across the geographies in one test, any assignment leaning on those identifiers loses users unevenly.
3. Execution. Symptom: the imbalance grows with page weight, is worse on mobile, or starts at a deploy. Check: split the ratio by device, browser, connection type and day; compare time-to-exposure between arms. Confirms it if the deficit concentrates on slow segments or in one browser. Fix: this is where redirect tests die. A redirect adds a full navigation to one arm only, and everyone who abandons during it is a selective loss. Render both variants server-side at one URL, or accept that the arms are not comparable.
4. Assignment. Symptom: the ratio is off from the first hour and stays off by a constant factor. Check: run an A/A on the same allocation and verify the hash salt was not reused from an earlier experiment on this population. Confirms it if the A/A shows the imbalance too. Fix: reassign with a fresh salt and restart. Carryover from a prior test is the quiet version of this, and only the A/A exposes it.
5. Interference. Symptom: the imbalance is concentrated in a small number of high-activity identifiers, or in internal IP ranges. Check: the exposure count distribution per user, and the top 1% of identifiers by event volume. Confirms it if removing them restores the ratio. Fix: exclude internal and scripted traffic by documented rule, not by hand, so the exclusion is identical next time.

A worked walkthrough

The numbers below are illustrative, not client data. A checkout test reports 61,480 control and 58,520 variant exposures: 51.2/48.8 on 120,000 users, chi-square p below 0.001. Branch 1: dropping the trigger condition returns 60,020 / 59,980, so assignment is fine and the loss happens after exposure. Branch 2: pipeline counts agree, bot exclusions 0.9% in both arms. Branch 3: desktop sits at 50.1/49.9, mobile at 52.4/47.6, and the variant's median time-to-exposure is 900 ms higher on mobile. Diagnosis: the variant ships an extra render-blocking request, mobile users abandon during it, survivors have better connections. The readout is void. The fix is a performance fix, and the test is rerun, not reinterpreted.

What you can salvage and what you cannot

• Void the readout when the loss mechanism could plausibly correlate with the metric. Every execution and assignment cause qualifies by default.
• Salvage by segment only when the imbalance is confined to a segment you can cleanly exclude, that segment was defined before you looked at the metric, and the remainder passes its own SRM check. Two of three is not enough.
• Keep the test when the cause was purely in analysis and the corrected scorecard passes. You are re-reading the same data with a fixed filter, not fishing.
• Log it either way. Record the cause category on the experiment. The taxonomy pays off cumulatively: if three of your last ten tests died in branch 3, your problem is not experimentation, it is front-end delivery. [INTERNAL DATA NEEDED: share of Switas-run experiments in the last 24 months that failed an SRM check, split by taxonomy branch, to replace the Microsoft 6% benchmark with a figure from this market]

Where this breaks down

This tree assumes a two-arm test with a fixed, known allocation and user-level randomization. It does not cover multi-arm tests, which need the full chi-square across all arms plus a follow-up on which pair is off, or ramped allocations, where the expected ratio moves during the test and a cumulative chi-square is the wrong test outright. It does not apply to cluster or geo-randomized designs, where the unit is not the user and the imbalance that matters is in cluster characteristics. It says nothing about whether your metric definition is right. And it cannot help when exposures are not logged and you are inferring assignment from a metric event, which is not a split check but a guess.

FAQ

Is a 51/49 split always a problem? It depends entirely on n. At 5,000 exposures it is ordinary noise; at 500,000 it is a near-certain defect. Judge the ratio against the detectability table, not against a fixed percentage rule of thumb.

Can I just reweight the arms statistically instead of rerunning? No. Reweighting corrects the counts and leaves the selection intact. The users you lost are gone, and they were not a random sample of the arm.

How often should I look at the SRM p-value? Once, at the end, on cumulative counts, unless your platform runs a sequential procedure. Daily peeking at a fixed-horizon chi-square inflates false alarms substantially.

Does SRM invalidate guardrail metrics too? Yes, for the same reason. If the arms hold different populations, a flat guardrail is no more trustworthy than a moving primary metric.

What if an A/A test also shows a mismatch? Then the fault is in assignment or in the logging pipeline, not in your treatment. Fix it before running anything else, because it is contaminating every test on that platform.

Do redirect tests always cause SRM? Not always, but they carry a structural risk no other design does, because the added navigation exists in one arm only. If you must run one, check the ratio by connection speed and device before reading any metric.

My traffic is too low for the check to mean anything. What then? Monitor raw daily exposure counts for persistent drift, run a periodic A/A on the assignment layer, and pre-register a longer list of sanity checks. You are substituting process for statistical power.

Should bot filtering happen before or after assignment? After exposure logging, before analysis, using deterministic rules applied identically to both arms. Heuristic filtering inside the SDK is how the filter itself becomes the mismatch.

Run the check on your own experiment programme

If you want a second pair of eyes on the split health of the tests you are running now, or a review of the assignment and logging layer they all depend on, that is what our CRO and experimentation service does.

Sources

• Fabijan, Gupchup, Gupta, Omhover, Vermeer, Dmitriev, "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments: A Taxonomy and Rules of Thumb for Practitioners", KDD 2019
• AB Tasty documentation, Sample ratio mismatch (sequential S-SRM, threshold 0.01)
• Statsig, "A quick guide to sample ratio mismatch", 31 August 2023 (chi-square, threshold 0.01)
• GrowthBook documentation, Experimenting (chi-square, warning below 0.001)
• Imperva, "Bad Bot Report 2026: Bots in the Agentic Age", 29 April 2026
• Republic of Türkiye Ministry of Trade, 2025 e-commerce figures, 12 May 2026
• Google tag platform, Consent mode and storage behaviour
• SRM Checker documentation and FAQ, Lukas Vermeer