Implementation worksheet · 5 min read
A Sample-Ratio Mismatch Investigation Checklist
A sample-ratio mismatch means the observed split differs from the intended one by more than chance explains, and it invalidates the experiment rather than merely adding noise — because whatever caused the imbalance probably also affected the outcome. Investigate six causes in order: assignment happening after an action that itself differs by arm, filtering applied to one arm only, duplicate or missing assignment events, bots and internal traffic hitting one arm disproportionately, a redirect or latency difference between arms, and analysis-time joins dropping rows. Do not rebalance the data. Reweighting hides the mechanism and produces a confidently wrong result — find the cause, fix it, and rerun.
SRM is the most reliable single signal that an experiment is broken, and the most commonly ignored because the imbalance looks small. A 52/48 split on 40,000 users is not a rounding difference; it is a message that assignment and exposure did not work the way the design assumed.
Put it into practice
1. Check the numbers before anything else
Run the chi-square test on observed versus intended counts. Treat a p-value below roughly 0.001 as a genuine mismatch rather than bad luck — this threshold is deliberately strict because the cost of investigating a false alarm is far lower than the cost of trusting a broken experiment.
2. Check where assignment happens
If users are assigned after an action — a page load, a login, an event — and that action differs by arm, the imbalance is structural. Assignment must precede anything the treatment can influence.
3. Check for one-sided filtering
An exclusion applied in one arm's query but not the other's, or a treatment that fails silently for a subset. Compare the filter logic for both arms line by line; this is where most mismatches live.
4. Check bots, internal traffic and duplicates
Automated traffic does not distribute evenly, and duplicate assignment events inflate one arm. Deduplicate on user and assignment id before concluding anything.
5. Check the analysis joins
Rows dropped by an inner join to a table that is incomplete for one arm is a quiet and common cause — the experiment was fine and the analysis broke it.
SRM investigation order
Copy this structure into your review document and record your observed result for each row.
| Cause | Signal | Check | Ruled out |
|---|---|---|---|
| Assignment after a differing action | mismatch grows with exposure | where is assign() called | |
| One-sided filtering | arms differ before treatment | diff the two queries | |
| Duplicate / missing events | counts exceed uniques | dedupe on user + assignment | |
| Bots / internal traffic | odd user agents, office IPs | exclude and recount | |
| Latency or redirect difference | one arm slower to record | compare timing distributions | |
| Analysis joins | mismatch only in the report | count at each pipeline stage |
A failure worth checking
The reweighted result: the mismatch is noticed, someone applies weights to restore 50/50, and the experiment is reported as a win. The cause was a treatment that failed to load for slow connections — so the treatment arm systematically lost its slowest users, who convert worst. Reweighting preserved the ratio and the bias, and the 'win' was the missing users, not the feature.
Common questions
Is a small mismatch ever acceptable?
Only when the test says it is consistent with chance. The size of the imbalance matters less than whether it is explicable — a 51/49 split on millions is a bigger red flag than 55/45 on a few hundred.
Can I salvage an experiment with SRM?
Occasionally, if you can prove the cause is unrelated to the outcome — for example a logging bug affecting assignment records but not user experience. The burden of proof is on salvage, and the default is to fix and rerun.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.