Implementation worksheet · 5 min read
A Predictive Marketing Vendor Evidence Checklist
Before crediting any 'predictive' claim, demand six pieces of evidence in writing: what exactly is predicted (event, horizon, population), the model's measured performance on held-out data with the metric named, how performance is monitored after deployment, what happens when the model is wrong, whether predictions are explainable at the user level, and what data the model needs from you before it predicts anything. A vendor who can't produce item two is selling a heuristic with a hat on.
'AI-powered predictions' has become table-stakes vocabulary in engagement platforms, which means the phrase itself now carries zero information. The evidence checklist restores the signal.
Put it into practice
1. Pin the prediction down
Predicting what — churn within 30 days? purchase within 7? — for whom, updated how often. Vague answers ('user propensity') mean the product is a score, not a prediction.
2. Demand held-out performance
The metric (AUC, precision at k, calibration) on data the model didn't train on, with the evaluation population described. Accept ranges from real deployments; reject single cherry-picked case numbers with no denominator.
3. Ask how it degrades
Data drift, cold-start users, sparse events: what does the model do — abstain, fall back to rules, keep scoring confidently? Confident scoring on insufficient data is the dangerous default.
4. Check the action wiring
A prediction that can't gate a journey, adjust a send, or exclude a segment is dashboard decoration. Trace one prediction to one automated action in the live product.
5. Verify your data prerequisites
Which events, how much history, what volume before predictions activate. This is also your onboarding timeline in disguise — get it in writing.
Predictive evidence checklist
Copy this structure into your review document and record your observed result for each row.
| Evidence item | Provided? | Quality | Notes |
|---|---|---|---|
| Prediction definition (event, horizon) | |||
| Held-out performance, metric named | |||
| Post-deploy monitoring method | |||
| Failure/degradation behavior | |||
| Data prerequisites in writing |
A failure worth checking
The aggregate-lift failure: accepting 'customers using our predictions see 23% higher retention' as model evidence. That number blends selection effects (who adopts predictive features), program effects (they also run more campaigns) and the model. Only held-out model metrics measure the model; only your own holdout measures the impact on you.
Common questions
Is asking for AUC too technical for a marketing evaluation?
No — you don't need to interpret the metric deeply; you need to observe whether the vendor has one. The existence and provenance of the number is the checklist item.
Can we validate a vendor's model ourselves?
Yes, and it's the gold standard: run their predictions against a holdout of your own for one cycle before wiring them to actions. A vendor confident in the model will help you design exactly this test.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.