Implementation worksheet · 8 min read
A Customer Activation Case-Study Evidence Template
Write the sentence you want to publish first, then fill one evidence record for it: the activation definition frozen before anyone saw results, the eligible population, the comparison design, counts and rates for each arm, the absolute difference with an interval, how credit was split across email, push, in-app and SMS, what else changed during the window, and the customer's written sign-off. The comparison design decides the verb. A random holdout supports 'increased'; a before-and-after supports 'coincided with'; no comparison supports a story about what was built and no outcome number at all. Channel percentages from attribution describe credit, not effect, unless each channel had its own holdout.
Scope: the lifecycle or customer marketer turning a customer's multi-channel program into a published case study. Starting state: a dashboard and an enthusiastic customer. Boundary: one claim about activation, caused by lifecycle messaging, published by you about someone else's results. Intended outcome: a claim sentence that a skeptical reader, or the customer's own analyst, could reproduce from the record.
Put it into practice
1. Write the claim sentence before the record
Every word in a case-study headline implies evidence. 'Increased' implies a comparison group. '40%' implies a denominator and a choice between relative and absolute. 'Because of the journey' implies a design that rules out everything else that happened that quarter. Draft the sentence, underline each implied claim, and the record below becomes the list of what you now need.
2. Freeze the activation definition, with a date
Name the event, the unit (user, workspace or account), the window and the date the definition was fixed. If it changed after anyone looked at results, publish both versions or neither. A metric chosen after the data is in can be chosen to flatter, even when nobody intends it to.
3. Record the comparison design and let it choose the verb
A random holdout assigned at eligibility supports 'increased'. A staggered rollout by region supports 'associated with', with the regional differences stated. A before-and-after supports 'coincided with'. No comparison group supports a description of the program and nothing about outcomes. Write the verb into the record so the headline can't drift upward during editing.
4. Separate channel credit from channel effect
Attribution splits the credit for each activation among the channels a customer touched. It doesn't estimate what any channel added. Unless each channel had its own holdout, the case study can report the program's effect plus each channel's attributed share, labeled as attributed share, but not that 'push drove a third of the lift'.
5. Log what else changed in the window
Pricing changes, an onboarding redesign, a sales push, a seasonal peak. A random holdout protects the comparison from most of these, because both arms lived through them; say so explicitly. A before-and-after has no such protection, which is why its change log is the difference between evidence and an advertisement.
6. Report the result in full, and check durability
Counts and rates for each arm, the absolute difference in percentage points, a 95% interval, and relative lift only beside the absolute figure. Then split the window in half. Sadeghi and colleagues describe treatment effects in online experiments that fade (novelty) or grow (primacy) over time, so one pooled number can hide either pattern.
7. Get the customer's sign-off and state what is typical
The customer approves the numbers and the wording in writing. If you publish in the US, the FTC's Endorsement Guides say an advertisement relating a consumer's experience with a key attribute is likely to be read as typical, so the advertiser needs substantiation for that or a clear disclosure of generally expected results (16 CFR 255.2(b)). Ask counsel how that applies to a business customer's story.
8. Worked example (illustrative, synthetic numbers)
A scheduling-software customer ran a 14-day activation journey: three emails, a day-3 push and an in-app checklist. Over eight weeks 9,000 new trial workspaces were eligible, and 10% were held out at random at signup. Activation, frozen before launch as 'first booking received within 14 days', was 2,511 of 8,100 (31.0%) with the journey and 243 of 900 (27.0%) without: +4.0 points, 95% interval roughly +0.9 to +7.1, about 15% relative. The sales draft said '42% lift'. The publishable sentence is the 4 points, with its interval.
Case-study evidence record
Copy this structure into your review document and record your observed result for each row.
| Field | What to record | Illustrative entry | Blocks publication if |
|---|---|---|---|
| Claim sentence | Exact wording to be published | The journey raised 14-day activation by 4 points | The verb is stronger than the design |
| Activation definition | Event, unit, window, date frozen | First booking received per workspace within 14 days; frozen before launch | It changed after results were seen |
| Eligible population | Entry rule, dates, count | New trial workspaces over eight weeks: 9,000 | Only message recipients are counted |
| Comparison design | Holdout, staggered rollout, before-and-after, or none | Random 10% of workspaces held out at signup | There is no comparison but an outcome number is claimed |
| Result per arm | Counts and rates | 2,511 of 8,100 (31.0%) vs 243 of 900 (27.0%) | Rates appear without counts |
| Effect and interval | Absolute difference and 95% interval | +4.0 points, roughly +0.9 to +7.1 | The interval is missing |
| Relative lift | Only beside the absolute figure and the base rate | About 15% relative, from a 27.0% base | It appears on its own |
| Channel credit | Attribution rule and share per channel, labeled as credit | Last touch: email 61%, push 22%, in-app 17% (attributed share) | It is presented as each channel's effect |
| Concurrent changes | Anything else that moved in the window | Pricing page redesign in week 5; both arms exposed | The log is empty for a before-and-after |
| Durability | Effect in each half of the window | Weeks 1-4: +4.6 points; weeks 5-8: +3.4 points; each about ±4.3, so not distinguishable | Only the stronger half is shown |
| Approval and typicality | Customer sign-off; statement of generally expected results | Signed; 'results depend on trial volume and setup' | Either is missing |
A failure worth checking
The clicker comparison. The sales draft's '42% lift' compared workspaces that clicked a journey email (38.1% activated) with workspaces that received the journey and never clicked (26.8%). That measures who clicks, not what the journey did: workspaces that were already engaged clicked more and would have activated more without any email. The same data, analyzed by the randomly assigned arms, gives 4 points. To verify any case-study number, have someone who didn't run the program recompute it from raw event data, starting from every eligible workspace in the originally assigned arms, including those whose messages were never delivered.
Common questions
Can we publish a case study without a holdout?
Yes, with the verb the design supports. A before-and-after can be published as 'coincided with', with the weekly series and the change log attached. What you can't do is borrow causal language from a holdout you didn't run.
The customer prefers the relative number because it's bigger. Can we lead with it?
Put it beside the absolute difference and the base rate. '15% relative, from 27.0% to 31.0%' can be checked by a reader; '15% lift' on its own can't, and it reads larger than the change most readers picture.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.