Implementation worksheet · 5 min read
An Incremental Conversion Reporting Template for Lifecycle Teams
Report four numbers per campaign instead of one: treatment conversion rate, holdout conversion rate, the difference (incremental rate), and the incremental count applied to the eligible population. Attributed conversions answer 'how many converters touched this message', which counts people who would have converted anyway; incremental conversions answer 'how many extra converted because of it', which is the only number that justifies spend. Expect the incremental figure to be a fraction of the attributed one — that gap is not a failure, it is what attribution was hiding, and reporting both side by side is how a lifecycle team keeps its credibility when someone finally runs the holdout.
Lifecycle programmes are usually judged on attributed conversions because that number is available by default and always looks good. Switching to incremental reporting is uncomfortable exactly once, and it is the difference between a programme that survives scrutiny and one that does not.
Put it into practice
1. Define the eligible population first
Everyone who met the entry criteria, not everyone who received the message. Reporting on recipients excludes people the system failed to reach and quietly inflates every rate that follows.
2. Hold out a random slice, consistently
Typically 5-10%, randomly assigned at eligibility and held for the whole measurement period. Assigning at send time rather than at eligibility is the most common way a holdout silently stops being random.
3. Report the four numbers together
Treatment rate, holdout rate, difference, incremental count. Publishing the difference without both rates invites a debate about whether the holdout was comparable, which the raw rates settle.
4. State the uncertainty honestly
A three-point difference on 400 people per arm is not a result. Report the interval or at least the arm sizes, so a reader can judge whether the difference is worth acting on.
5. Keep the attributed number visible
Show both, labelled clearly. Removing the attributed figure entirely reads as hiding something; showing both teaches the organisation the difference, which is the durable win.
Incremental conversion report
Copy this structure into your review document and record your observed result for each row.
| Metric | Treatment | Holdout | Difference |
|---|---|---|---|
| Eligible population | — | ||
| Conversion rate | |||
| Incremental rate | — | — | |
| Incremental conversions | — | — | |
| Arm sizes (for uncertainty) | — |
A failure worth checking
The first honest report: a programme reporting 1,200 attributed conversions a month runs its first holdout and finds 180 incremental. Nothing changed except the measurement, but it reads as a collapse to anyone seeing both numbers for the first time. Introducing incremental reporting alongside the attributed number from the start — rather than switching after a year of headline figures — is the difference between a methodology improvement and a credibility event.
Common questions
Is a permanent holdout worth the lost conversions?
At 5-10% the forgone conversions are small and the alternative is spending indefinitely on programmes nobody has tested. Rotate which cohort is held out if the loss concentrates on the same users.
What if the incremental effect is zero?
That is a real and valuable finding: the programme is reaching people who would have converted anyway. Redirect the effort rather than defending the attributed number.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.