Implementation worksheet · 2 min read
A holdout design worksheet for a lifecycle campaign
To estimate a lifecycle campaign's incremental effect, assign eligible users or accounts to a treatment group and a concurrent holdout before exposure. Keep assignment stable, define the primary outcome and observation window, and compare outcomes for the originally assigned groups. Report uncertainty and implementation failures alongside the result.
This is a design worksheet, not a claim that a campaign produced uplift. A before-and-after comparison can mix campaign effects with changes in traffic, seasonality or product behavior. The appropriate sample size and inference method depend on your baseline rate, effect of interest and randomization unit.
Put it into practice
1. Choose the unit
Randomize at the level where interference is limited. If teammates share a workspace and influence one another, account-level assignment may be more suitable than assigning individuals independently.
2. Define eligibility and assignment
Specify entry conditions, exclusions and a stable assignment mechanism. Record assignment before the send or action occurs. Preserve a holdout even if the treatment fails to deliver, so the analysis does not select only successful sends.
3. Choose outcomes in advance
Name one primary outcome, its observation window and the guardrails that could stop the test. Estimate the sample required for a useful result. If traffic is insufficient, state that limitation before launching.
4. Check implementation
Compare intended and observed allocation, duplicate exposures and cross-channel leakage. A holdout that receives the same campaign through another workflow is not a clean control for that intervention.
5. Report the full result
Show group sizes, conversion counts, rates, absolute difference and uncertainty from the chosen method. Distinguish percentage points from relative lift. Record whether the experiment was stopped early and why.
Holdout planning worksheet
Copy this structure into your review document and record your observed result for each row.
| Decision | Record before launch | Review |
|---|---|---|
| Population | Eligible cohort and exclusions | Reproducible |
| Assignment | Unit and allocation | Stable |
| Primary outcome | Event and window | Unambiguous |
| Guardrails | Complaints and adverse outcomes | Owned |
| Analysis | Method and stopping rule | Chosen in advance |
A failure worth checking
Comparing people who opened an email with people who did not open it does not isolate the email's effect. Those groups selected themselves after assignment. Analyze the original randomized groups for the intended causal comparison.
Common questions
Can I claim uplift from a few extra conversions?
Only with an analysis that accounts for uncertainty and the design. A small observed difference may be compatible with chance.
Must a holdout last forever?
No. Its duration should match the predeclared observation window and operational constraints. Do not repeatedly change it to obtain a preferred result.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.