Implementation worksheet · 5 min read
An Activation Experiment Guardrail Checklist
Before launching any activation experiment, lock five guardrails in writing: a pre-registered primary metric (one, with its measurement window), fixed assignment (users keep their arm for the full window — no reassignment on upgrade or complaint), a minimum sample and duration you commit to before reading results, a no-mid-flight-edits rule (copy, timing and audience frozen; a needed fix means restart, not patch), and a decision rule written in advance (what result ships the variant, what result kills it). Experiments without guardrails produce vibes with confidence intervals.
Lifecycle experiments are uniquely easy to corrupt because the people running them can see partial results and edit live campaigns. None of the corruption feels like corruption at the time — it feels like iterating. The checklist exists to make the corruption visible before it happens.
Put it into practice
1. Pre-register the primary metric
One metric, one window, written before launch: 'value-event rate within 14 days of entry'. Secondary metrics are for hypotheses, not verdicts — the verdict metric is chosen blind.
2. Fix assignment at entry
A user's arm is decided once, at journey entry, and survives everything short of deletion. Reassigning upgraded users 'because they're engaged now' launders the treatment effect into the control.
3. Commit to sample and duration
Compute the minimum sample for the effect size you'd act on; commit to the duration that reaches it. Reading at day 3 of 14 because 'it looks significant' is the peeking that manufactures false winners — significance at a peek is what noise looks like.
4. Freeze the experiment surface
Copy, send times, audience definition: frozen at launch. Discovered a typo? Log it, let it ride, fix in the next run. A patched experiment measures the patch boundary, not the variant.
5. Write the decision rule in advance
'Ships if primary metric improves by ≥X with the agreed confidence; dies otherwise; inconclusive reruns once at larger sample.' Deciding after seeing results converts every outcome into a ship decision — which is why the rule predates the data.
Guardrail checklist
Copy this structure into your review document and record your observed result for each row.
| Guardrail | Written down? | Where |
|---|---|---|
| Primary metric + window | ||
| Fixed assignment policy | ||
| Min sample / duration commitment | ||
| Freeze (copy, timing, audience) | ||
| Advance decision rule |
A failure worth checking
The mid-flight edit: day 4, variant B's second email has a broken button; someone fixes it and the experiment continues. Reasonable — and now arm B is two different treatments pooled, the effect is unattributable, and whatever ships, ships on corrupted evidence. The guardrail isn't against fixing bugs; it's against pretending the fixed experiment is the same experiment.
Common questions
Do small teams really need this rigor?
Small teams need it most — they have the least traffic, so every corrupted experiment burns weeks of sample they can't spare. The checklist costs one page; a false winner shipped to production costs a quarter.
What about holdouts vs A/B arms?
Different jobs: the holdout (no treatment) measures whether the campaign should exist; A/B arms measure which version wins. Mature programs run both — the guardrails above apply identically to each.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.