Stabilize first, explain second
A production incident is a time-constrained reliability problem. The first objective is to reduce customer impact and stop the situation worsening; root-cause certainty can come later. A repeatable response separates command, technical investigation, communications, and evidence so the team does not debug by crowd consensus.
Incident roles reduce coordination cost
- Declare severity using customer/business impact, not engineer anxiety.
- Assign an incident lead and one communication channel.
- Prefer reversible containment: rollback, disable feature, shed load, fail over, or pause a worker.
- Record a timeline of observed facts, changes, and decisions while the incident is active.
- After recovery, identify contributing system conditions and durable actions rather than blame.
Detect, contain, recover, learn
Detection turns into a declared response, containment reduces impact, recovery restores service, verification confirms stability, and follow-up converts evidence into preventive work.
Incident response lifecycle
Detection turns into a declared response, containment reduces impact, recovery restores service, verification confirms stability, and follow-up converts evidence into preventive work.
1
DetectStart
2
Triage + declareStep
3
ContainStep
4
RecoverStep
5
Verify stabilityStep
6
CommunicateStep
7
Post-incident actionsOutcome
A failed deployment under customer traffic
Incident habits that increase damage
Operational response checklist
- Declare owner, severity, impact, and communication channel.
- Freeze unrelated changes.
- Choose the safest reversible containment.
- Keep a timestamped timeline and evidence.
- Verify recovery through business signals, then create owned follow-up actions.