Implementation worksheet · 6 min read
A Marketing Agent Incident Escalation Runbook
Have one stop control that halts all agent sending immediately, reachable by anyone on call without a deploy — that is the whole first minute. Then work the ladder: contain (stop sending, freeze the agent, do not start fixing), assess (how many received it, what exactly did they receive, is it still going), decide (who is accountable for the customer-facing response, which is a business decision not an engineering one), communicate, and only then diagnose. The most expensive mistake in these incidents is an engineer deciding alone whether to send an apology, because that decision is about the relationship and the liability rather than the bug.
Marketing agent incidents differ from service incidents in one important way: the damage has already left the building. A down service is repaired and forgotten; a wrong message is in the inbox of everyone who received it, screenshotted, and possibly shared. Speed matters for containment, but the actions after containment are relationship decisions and usually belong to people who are not on the engineering rota.
Put it into practice
1. Build the stop control before you need it, and test it quarterly
One action that halts all agent-initiated sending, available to anyone on call, no deploy and no vendor console required. Test it on a schedule like any other emergency control. A stop control that has never been exercised is a plan, not a capability, and you find out which during the incident.
2. Contain first and resist fixing
Stop sending, freeze the agent, preserve state. The instinct to fix the bug immediately is strong and wrong — fixing before containing has restarted sends more than once. Nothing gets diagnosed until sending is verifiably stopped.
3. Establish the facts as numbers, quickly
How many messages went out, to whom, containing exactly what, and is anything still queued. Approximate answers are fine in the first ten minutes; a wrong confident answer is not. The audit trail is what makes this a query rather than an investigation.
4. Classify severity by what the recipient experienced
Wrong send time is one thing; a wrong price, another customer's data, or a message to someone who opted out are categorically different. Data exposure and consent violations escalate immediately and carry obligations that no internal severity scale overrides.
5. Hand the customer-facing decision to a named business owner
Whether to apologise, to whom, in what words, and whether to offer anything. On-call engineering should not make this call alone — it concerns the relationship and potentially the liability. Name the owner and the deputy in the runbook, in advance, so the escalation is a phone call rather than a search.
6. Diagnose last, and change the permission matrix as part of the fix
After containment and communication. The fix is rarely only the bug — the action that caused the damage was granted a level in the permission matrix, and the review should ask whether that level was right. An incident that ends with a code fix and no governance change usually recurs in a different shape.
The escalation ladder
Copy this structure into your review document and record your observed result for each row.
| Phase | Action | Owner | Target |
|---|---|---|---|
| 0-1 min | hit the stop control | anyone on call | immediate |
| 1-5 min | freeze agent, preserve state | on call | 5 min |
| 5-15 min | counts, recipients, content, still queued? | on call + audit trail | 15 min |
| 15 min | classify severity by recipient impact | on call + business owner | |
| Immediate if data or consent | escalate on the separate path | named owner | no delay |
| 30 min | decide customer-facing response | named business owner | |
| Post-response | communicate to affected customers | business owner | |
| After | diagnose root cause | engineering | |
| After | review the permission matrix level | governance owner | within a week |
| Quarterly | test the stop control | engineering | scheduled |
A failure worth checking
Discovering during the incident that stopping requires a deploy. The wrong campaign is sending, the fix is understood in four minutes, and shipping it takes twenty-five — during which another eleven thousand messages go out. The stop control is the single highest-value thing in this runbook and the one most likely to be missing, because until the first incident it looks like a feature nobody needs.
Common questions
Should we tell customers about a wrong send?
Usually yes for anything visible or consequential, and that is a business decision with a named owner rather than an engineering one. Silence is only defensible when the error was genuinely unnoticeable, and teams overestimate how often that applies.
Does this belong in the normal on-call rota?
The containment steps, yes — speed matters and engineering has the access. The communication and apology decisions need a business owner who may not be on the rota, which is exactly why they are named in advance rather than found at 2am.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.