Skip to content
Documentation

Incident Command: Live Response and Postmortems

FieldValue
TypeSkill Resource
Source~/.copilot/skills/platform/references/incident-command.md
DescriptionNot specified

Source Content

Incident Command: Live Response and Postmortems

Incidents become outages when comms break down, not when systems break down. Impose calm structure — roles, channels, cadence, decision log — then convert the pain afterward into permanent improvements.

Contents

Use me for / Don’t use me for

Use for: driving an active incident right now, writing the PIR for a just-resolved incident, setting up an on-call program before you need it (rotation, comp, drills, runbooks), auditing a quarter of incidents for systemic patterns.

Don’t use for: designing the SLOs that define “incident” (references/observability.md), authoring per-alert runbooks in depth (references/observability.md + service owner), security-breach legal/disclosure (the security skill), postmortem prose polished for a customer audience (the technical-writing skill).

“We had a security breach” → defer to the security skill for legal/disclosure; run the operational room in parallel. “Define the SLOs for this service” → defer to references/observability.md; SLOs anchor severity here, they aren’t designed here.

Running a live incident

In order:

  1. Declare. Pick severity. Open #inc-YYYYMMDD-<slug>. Page an IC if you aren’t one.
  2. Assign roles. Pin: IC, Comms, Scribe, SME (+ Customer Liaison on SEV1/2). Solo? You’re IC + SME.
  3. Stop the bleed. Mitigation before root cause. Try one thing with a 15-min timer; move on if no improvement.
  4. Comms cadence. Every 15 min (SEV1) or 30 min (SEV2), even “still investigating” — silence breeds escalation.
  5. Decision log. Scribe captures every non-trivial action with timestamp, owner, reason, result.
  6. Hand off if the incident runs over 2 hours. Tired commanders make bad calls.
  7. Resolve. Metrics back to baseline for 15 min, final status-page update, channel summary, archive after the PIR lands.

Writing the postmortem

Within 5 business days. Sections: Summary, Impact (with error-budget burn — pairs with references/observability.md), Timeline, Root cause(s), Five Whys ending at a process or system gap, Contributing factors, What went well, Action items (owner + due date + ticket).

Blameless means the language targets the system, never a person — a postmortem never names an individual as “root cause.”

On-call program design

  • Sustainable rotation with paid comp — being on-call should not ruin anyone’s life.
  • Runbook coverage for every page-worthy alert (the runbook itself lives with references/observability.md + the service owner).
  • A drill cadence so the first real page isn’t the first rehearsal.
  • Error budgets exist to protect humans, not just numbers — if the budget burns, feature work pauses; that’s the contract.

Validate

scripts/check_pir.sh PIR.md verifies a postmortem has:

  • a timestamped decision log,
  • a named Incident Commander (rejects placeholders like TBD/TODO/N/A),
  • at least one action item with both an owner and a due date (YYYY-MM-DD).

Exits non-zero listing which of the three checks failed.

Self-rubric

  • Severity declared and matches customer impact.
  • Roles assigned and pinned — IC is not also debugging.
  • Mitigation tried before root-cause hunting.
  • External comms cadence is being met.
  • Decision log has timestamps and owners.
  • PIR is blameless — language targets the system, never a person.
  • scripts/check_pir.sh exits 0.

References