Skip to content
Documentation

Incident Response

What this is and who it’s for

This page stands up an incident response practice for an engineering team: who carries the pager, how you classify what’s broken, who runs the response, and how you learn from it afterward. It’s for the engineer or tech lead who owns a user-facing service and needs a repeatable way to handle outages and security events without inventing the process mid-fire. It assumes no existing on-call program — you’ll leave with a rotation, a priority matrix (P1–P5), defined incident roles, a communication plan, and a blameless review ritual. It does not cover building the monitoring that tells you something is wrong; that’s a prerequisite, linked below.

Use P1–P5 in all incident communications. When in doubt, declare the higher level first, then downgrade only after the Incident Commander confirms the impact.

Prerequisites

  • You can detect that something is broken. Incident response starts after detection. Stand up monitoring and alerting first — see Observability. Without it, your “incident response” is a citizen calling the help desk.
  • You have a place to communicate. A chat channel, a bridge line, or both. Roles below assume a shared incident channel.
  • You have somewhere to track action items. Your existing ticket tracker is fine. Action items that live in a document nobody reopens do not get done.
  • You have, or can get, a way to page a human. A paging tool is ideal, but a documented phone-tree fallback is acceptable for low-volume services — see “Starting without paid tooling” below.

The practice

Adopt these in order. A team can run a credible incident response with just the priority matrix and a single on-call engineer; everything else hardens the practice as the service grows.

On-call: a rotation, not a hero

On-call is a scheduled responsibility shared across the team, not a permanent assignment to whoever knows the system best. Build a rotation where each engineer takes a defined shift — one week is a common cadence — and rotates out. The goal is that any person on the rotation can begin a response; if only one human can, you have a single point of failure wearing a lanyard.

Keep on-call humane, because a burned-out responder is a slow and error-prone responder:

  • Two tiers, not one human forever. A primary responder and a secondary (or escalation) who is paged if the primary doesn’t acknowledge within a set window — for example, 15 minutes [VERIFY against your service’s tolerance].
  • Compensate or comp the time. Carrying the pager is work. [FILL IN: Maryland state compensation / comp-time policy for on-call.]
  • Quiet the noise. Every page that isn’t actionable trains the responder to ignore the pager. Tune alerts in Observability so a page means “a human must act now.”
  • No solo permanent on-call. If your rotation is one name, your first action item is recruiting a second.

The escalation policy is written down, not improvised: who is paged first, how long until it escalates, who is the final backstop, and how to reach a manager when an incident needs a decision above the responder’s authority.

Priority levels: define them before you need them

A priority level sets who wakes up, how fast, and what communication path starts.

When in doubt, declare the higher level first, then downgrade if confirmed. In this standard, only the Incident Commander may downgrade P1 or P2.

P1 — Critical

  • Use when critical production services are down, sensitive data may be exposed, or a security incident is confirmed.
  • Expected response: assign an IC immediately, wake leadership, and run dedicated incident response until containment or restoration.

P2 — Major

  • Use when critical functionality is impaired for many users or there is urgent operational risk.
  • Expected response: assign an owner, notify leadership, and begin urgent remediation.

P3 — Moderate

  • Use when a subset of users is affected and a workaround exists.
  • Expected response: assign a responsible team and track through normal incident handling.

P4 — Minor

  • Use when impact is limited and operations remain stable.
  • Expected response: defer to regular maintenance or sprint backlog flow.

P5 — Informational

  • Use for advisories, planned maintenance notices, and non-actionable alerts.
  • Expected response: record and review on schedule.

Incident record fields

Every incident record needs the same minimum fields so reports can be compared across services and environments.

  • Impacted service and environment
  • Priority, type, and category
  • Status, owner, and Incident Commander
  • Timeline with timestamps and timezone
  • TLP classification when security or sensitive data is involved
  • Recovery action, follow-up owner, and due date

Stabilize with reversible action first

Whatever your deployment model, the first move is to stabilize with a reversible action and preserve a trail of what changed. Prefer rolling back a release, re-pinning to a known-good version, or flipping a feature flag off over hand-editing live infrastructure and leaving no record of it.

  1. Detect a signal from alerts, deploy failures, or user reports.
  2. Stabilize with the most reversible action available: roll back, re-pin, flag off, or a small reviewed fix.
  3. Confirm the service is healthy again before you stop.
  4. Run the blameless review after the service is healthy.
  5. Close only when mitigation, guardrails, owners, and due dates are assigned.

Retain the incident record, the recovery change, the stakeholder notification, the post-incident review, and the action-item tickets. These artifacts support auditability and controlled recovery without relying on screenshots or chat memory.

The incident-command model

For anything P2 and above, separate the work of fixing from the work of coordinating. One person trying to debug, update stakeholders, and direct the team does all three badly. Assign these roles explicitly at the start of an incident; on a small team one person may hold two, but never collapse all three into the responder.

Incident Commander (IC)

Owns the response, not the keyboard. The IC runs the incident: declares priority, assigns the other roles, drives the response toward resolution, and makes the calls. The IC does not fix the problem — they make sure the right people are fixing it and that nothing falls through. When decisions exceed the IC’s authority, the IC escalates per the on-call policy.

Communications Lead

Owns the message, internal and external. The comms lead posts internal updates on a cadence and drafts status-page and stakeholder updates so the IC and ops lead can stay heads-down. For citizen-facing services this role is not optional — see the next section.

Operations Lead

Owns the hands-on remediation. The ops lead and any subject-matter responders investigate, mitigate, and fix, reporting status to the IC. The ops lead does not field stakeholder questions directly; those route through the comms lead so the fix isn’t interrupted.

Communication during an incident

Run two channels in parallel: internal coordination and external updates. Conflating them is how a half-formed internal theory ends up on a public status page.

Internal communication is a single source of truth — one incident channel or bridge where the IC, comms lead, and ops lead coordinate. Post updates on a fixed cadence (for example, every 30 minutes for a P1 [VERIFY]) even when the update is “still investigating.” Silence reads as a stalled response.

External communication matters more for government services than almost anywhere else, because citizens often have no alternative to the service that’s down. The comms lead owns a status page and stakeholder updates that are honest, plain-language, and timely. Acknowledge the problem, state what’s affected and what isn’t, give a next-update time, and never promise a restore time you can’t keep. Coordinate any public message with [FILL IN: agency communications / public information office] before it goes out, per agency policy.

The post-incident review

After every P1 and P2, hold a blameless post-incident review (PIR). Blameless means the review investigates how the system and process let the incident happen, not which person made a mistake. People act reasonably given the information and tools they had; if the outcome was bad, the system made the bad outcome likely. The moment a review becomes about fault, people stop reporting incidents and you go blind.

Run the review within a few days while memory is fresh, and keep it focused on systems:

  • Build a timeline. What happened, when, and how you knew — drawn from your incident channel and telemetry.
  • Find contributing factors, not a root cause. Most incidents have several; a single “root cause” usually hides the interesting ones.
  • Write action items with named owners and due dates. An action item without an owner is a wish. Track them where you track other work and review them until closed.
  • Share the review. Publish it to the team so the next responder learns from it.

Security incident vs. availability incident

These are different incidents with different first moves. An availability incident — an outage or degradation — follows the path above: declare priority, assemble roles, mitigate, communicate, review. A security incident — suspected breach, unauthorized access, data exposure, malware — branches early and has obligations the availability path does not.

For a suspected security incident:

  • Notify the security path first, not just the on-call engineer. [FILL IN: agency CISO / security operations center / Maryland statewide security contact.]
  • Preserve, don’t clean up. Resist the instinct to “fix it and move on.” Preserve logs, snapshots, and affected systems as evidence before remediating, because cleanup can destroy what an investigation needs.
  • Limit who knows and how. Security incidents have disclosure constraints; route communication through the security path, not the public status page, until told otherwise.
  • Know the notification clock. A confirmed breach may trigger legal notification obligations to affected individuals and the state. [VERIFY: Maryland breach-notification statute and agency obligations — confirm timelines and who notifies whom before relying on this.]

When in doubt about whether an availability incident is also a security incident, treat it as security and engage the security path. It’s reversible; a destroyed evidence trail is not.

Reference implementation

Fork these four artifacts into your team’s incident repo or wiki and adapt the bracketed values. They’re plain text so they paste into a ticket, a channel topic, or a doc without reformatting.

Incident priority matrix:

PRIORITY MATRIX — [SERVICE NAME]


| Priority | Criteria | Response | Comms |
| --- | --- | --- | --- |
| P1 | Core service down/unusable for most users; confirmed data loss/exposure; statutory deadline at risk | Page now; full IC structure | External + status page; cadence 30m |
| P2 | Significant feature degraded; subset of users affected; workaround exists | Page (business hrs or now if trending to P1) | Internal; external if user-facing |
| P3 | Limited impact; workarounds available; no citizen-facing degradation | Next business day via ticketing | Internal only |
| P4 | Low-impact; cosmetic, documentation, non-prod issues | Routine maintenance / sprint planning | Internal only |
| P5 | Observation or advisory; no action required | Log and review as needed | None |


When in doubt, declare the HIGHER priority.

Incident roles cheat sheet:

INCIDENT ROLES — assign at the start of every P1/P2


### Incident Commander (IC)


- Runs the incident; does NOT touch the keyboard.
- Declares priority, assigns roles, drives to resolution, escalates decisions.


### Communications Lead


- Owns internal cadence updates + external/status-page/stakeholder messages.
- Coordinates public messages with [PUBLIC INFO OFFICE] before posting.


### Operations Lead


- Owns hands-on investigation and remediation; reports status to IC.
- Does NOT field stakeholder questions — routes them through Comms.


Small team: one person may hold two roles. NEVER all three.
Escalation: primary -> secondary after [15] min no-ack -> manager for decisions.

Communication update template:

STATUS UPDATE


| Field | Entry |
| --- | --- |
| Service | - |
| Priority | - |
| Timestamp (with timezone) | - |
| Status (Investigating / Identified / Monitoring / Resolved) | - |


## What's wrong


## Who's affected


## What works


## Workaround


## Next update


Do not promise a restore time you cannot keep.

Blameless post-incident review template:

POST-INCIDENT REVIEW


| Field | Entry |
| --- | --- |
| Service | - |
| Date | - |
| Priority | - |


## Summary


## Timeline


-


## Impact


## Contributing factors


## What went well


## Action items


- [ ]


This review is BLAMELESS. We examine systems and process, not people.

Common pitfalls

  • No defined priorities, so everything is a fire drill or nothing is. If your team debates priority during the incident, you don’t have a priority scheme — you have an argument. Use the P1-P5 standard above and make the call mechanical.
  • A single hero on-call who burns out. If one name carries every page, you have a staffing risk masquerading as a strong engineer. Build a rotation with a backup tier and tune the noise so the pager means something.
  • Blameful post-mortems that make people hide incidents. When the review hunts for who to blame, the next person quietly doesn’t report. Keep it blameless and about systems, or you’ll lose the data the practice depends on.
  • Action items with no owner. “We should add a check” in a review doc that nobody reopens is a wish, not a fix. Every action item gets a named owner, a due date, and a ticket you track to closure.
  • No external comms, so citizens and stakeholders are left guessing. Silence during a citizen-facing outage erodes trust and floods the help desk. Run a status page and a stakeholder cadence; an honest “still investigating, next update in 30 minutes” beats nothing.
  • Confusing a security incident with an outage. Treating a breach like a normal outage means you “clean up” and destroy evidence, and miss notification obligations. If a security incident is possible, engage the security path and preserve before you remediate.

Starting without paid tooling

You don’t need a paging SaaS or a hosted status page to begin. A documented phone-tree fallback covers paging for a low-volume service, and a simple static page you control covers external status while you decide whether a richer tool is worth it. Start the practice with what you have; add tooling once the rotation and the priority matrix are real and the manual process is the bottleneck.