Blameless Postmortems and After-Action Reports

This guide explains how to run a structured incident review that improves your systems instead of punishing your people, and how to meet government reporting requirements without undermining psychological safety. Engineers and team leads at government agencies who are responding to incidents, building review processes, or trying to reduce repeat outages will find it most useful.

---

TL;DR

• A blameless postmortem asks “what failed in the system?” not “who made a mistake?”
• Run one after every P1, P2, near-miss, or recurring incident.
• Use the five whys to find root causes — don’t stop at “human error.”
• Every action item needs an owner and a due date, or it will never happen.
• Government after-action reports may be required by contract, policy, or law — treat them as the formal version of the same process.

---

Blameless Postmortem Fundamentals

A blameless postmortem is a structured review that happens after an incident. The goal is to understand what went wrong and fix it — not to find out who to blame.

The core belief behind blameless postmortems is that engineers make good decisions with the information they have at the time. When a decision leads to an outage, the system let that decision cause harm. Fix the system.

This does not mean no accountability. Engineers are still responsible for their work. The difference is that accountability focuses on improving future behavior, not punishing past behavior.

---

Psychological Safety and Incident Learning

When engineers fear punishment for mistakes, they stop reporting near-misses. A near-miss is an incident that almost happened but didn’t — or happened in a small way before it could grow. Near-misses are your best warning system.

If your team hides near-misses, you lose the chance to prevent the real incident. By the time the outage happens, the warning signs are gone.

Blameless postmortems create the conditions where engineers feel safe to say “this almost happened, here’s why.” That information makes your systems more reliable.

---

Blameless vs. Accountable

These two ideas work together, not against each other.

Concept	What it means	What it is not
Blameless	The review focuses on system failures, not personal failures	Letting harmful behavior go unaddressed
Accountable	Individuals own their decisions and their action items	Punishment, public shaming, or career consequences for honest mistakes

An engineer who repeatedly ignores runbooks, skips checklists, or causes incidents through recklessness is a different situation — that is a management conversation, not a postmortem conversation.

---

When to Run a Postmortem

Run a postmortem after any of these:

• P1 or P2 incident — any outage that affected real users or violated an SLA
• Near-miss — something almost went wrong, or went wrong at a small scale before it could cascade
• Recurring incident — the same thing broke twice, even if the first instance seemed minor

Reserve the process for incidents that have real learning value — you do not need a postmortem for every P3 or P4 ticket. Run the postmortem within 72 hours of resolution; memory fades fast and the details matter.

---

Government After-Action Reports

Government agencies often use the term “after-action report” (AAR) instead of postmortem. The content is similar, but the context is different.

An AAR in a government setting may be:

• Required by contract — some contracts specify that a formal AAR must be delivered to the client within a set number of business days after an incident
• Required by agency policy — many agencies have IT incident response policies that mandate formal documentation
• Required by law or regulation — some incidents trigger mandatory reporting to oversight bodies

The key difference is that AARs often go to leadership, contracting officers, or compliance teams. They use formal language, avoid internal jargon, and are subject to public records requests in most government contexts. Write your internal postmortem first and use it as the source of truth when drafting the formal AAR.

---

The Postmortem Structure

Every postmortem should cover these sections, in this order:

1. Incident summary — one paragraph describing what happened, when, and the impact
2. Timeline — a chronological sequence of key events (see diagram below)
3. Contributing factors — what conditions made the incident possible or worse
4. Impact summary — who was affected, for how long, and what was the measurable effect
5. What went well — actions taken during the incident that helped limit damage or speed recovery
6. What could have gone better — process gaps, tool failures, or communication breakdowns
7. Action items — specific improvements, each with an owner and a due date

The action items section is the most important part — a postmortem with no action items is an exercise in storytelling, not improvement.

Figure 1 (placeholder) — A timeline diagram for an incident postmortem. Show a horizontal time axis from T-0 to T+4 hours. Mark key events: T+0 First alert fires, T+15 On-call engineer paged, T+22 Incident declared, T+45 Root cause identified (replica promotion disabled), T+60 Fix deployed, T+75 Service restored, T+4hr Postmortem meeting scheduled. Below the timeline show ‘Impact window’ as a red bar between T+0 and T+75. Use annotations to show decisions made at each point.

---

The Five Whys Technique

The five whys is a simple root cause analysis method. You start with the symptom and ask “why” five times in a row. Each answer becomes the next question. Stop when you reach a systemic cause you can actually fix.

Worked example

Round	Question	Answer
Symptom	The database failover took 47 minutes instead of 5.	The replica was never promoted automatically.
Why 1	Why was automatic promotion not working?	The promotion script was disabled on this replica.
Why 2	Why was the script disabled?	An engineer disabled it during maintenance 6 weeks ago to avoid conflicts.
Why 3	Why was it not re-enabled after maintenance?	There was no checklist step to re-enable it.
Why 4	Why was there no checklist step?	The runbook was written before automatic promotion existed as a feature.
Why 5	Why was the runbook never updated?	There is no process for reviewing runbooks after infrastructure changes.

Root cause: no process for updating runbooks when infrastructure changes. Fix: add a runbook review step to the infrastructure change process, assign an owner, and set a review date.

Notice that the root cause is not “the engineer disabled the script.” That is a contributing factor. The root cause is the system that allowed the script to stay disabled for six weeks without anyone noticing.

---

Government-Specific Considerations

Mandatory Reporting Obligations

Some incidents require notification to external bodies. Know your reporting thresholds before an incident happens, not during one. Common bodies include:

• CISA — significant cybersecurity incidents affecting federal systems (Cyber Incident Reporting for Critical Infrastructure Act, CIRCIA)
• OMB — major incidents affecting federal information systems (OMB M-20-04)
• State equivalents — many states have similar requirements for systems handling state data
• Grant-funding agencies — some grants require incident reporting to the program office

Public Records Exposure

Incident documentation, postmortems, and after-action reports are records. In most jurisdictions they are subject to public records requests (FOIA, state open-records laws). Write your postmortems assuming they may be read by journalists, oversight bodies, or the public. This does not mean writing defensively — it means being factual and professional.

Change Management Integration

Many government agencies require formal change management for production changes. The action items from a postmortem often become change requests. Know your agency’s process and route action items through it.

Contract Reporting Timelines

If your system is operated under a government contract, check the incident response clauses. Many contracts require notification within 1 hour for certain incidents, and a written report within 72 hours.

---

Running the Postmortem Meeting

Who Attends

• The on-call engineer who responded
• The service owner or team lead
• A neutral facilitator (not the incident responder)
• A note-taker (not the facilitator)
• Anyone who made key decisions during the incident
• Optional: a representative from the affected user group

Keep the meeting to 90 minutes maximum — set an agenda and hold to it. The facilitator keeps the discussion focused on systems and processes, not people: they interrupt blame language, guide the group through the five whys, and do not need to be a technical expert. Share the rules of engagement below at the start of every meeting.

Rules of engagement

1. We are here to improve the system, not assign fault.
2. We assume every engineer made the best decision they could with the information they had.
3. No interrupting. One speaker at a time.
4. Action items get owners and due dates before the meeting ends.

After the meeting, the note-taker publishes the draft within 24 hours, owners review and confirm their action items, and the postmortem goes into your team’s shared knowledge base.

---

Common Mistakes

Stopping at Human Error

“Human error” is never the root cause — it is a symptom of a system that made error easy or recovery hard. Keep asking why until you reach the systemic condition you can actually change.

Action Items Without Named Owners

An action item with no named owner will not happen — every item needs one person’s name, not “the team.”

Action Items Without Due Dates

An action item with no deadline drifts into backlog oblivion — set a due date and put it in your project tracker.

Postmortems Without Follow-Through

Follow up on action items at your next team meeting. Review completion rates quarterly. If action items are consistently closed without being done, your postmortem culture is broken.

Skipping What Went Well

This section is not filler — it documents behaviors you want to reinforce and keeps the meeting from becoming purely negative.

Failing to Share the Postmortem

Other teams in your agency may have the same systemic risk — share your postmortems (at an appropriate classification level) so others can learn without experiencing the same incident.

---

Postmortem Template

Copy this template for every postmortem. Fill in every section and do not skip action items.

# Postmortem

| Field | Entry |
|---|---|
| Incident title | - |
| Date of incident | - |
| Severity | - |
| Duration | - |
| Postmortem author | - |
| Review date | - |
| Status | - |

---

## Incident Summary

---

## Timeline

| Time (UTC) | Event |
|---|---|
| Field | Field |

---

## Contributing Factors

-

---

## Impact Summary

| Dimension | Detail |
|---|---|
| Users affected | - |
| Services affected | - |
| Duration | - |
| SLA impact | - |
| Data loss | - |

---

## What Went Well

-

---

## What Could Have Gone Better

-

---

## Five Whys Root Cause Analysis

| Round | Why? | Answer |
|---|---|---|
| Symptom | - | - |
| Why 1 | - | - |
| Why 2 | - | - |
| Why 3 | - | - |
| Why 4 | - | - |
| Why 5 | - | - |

Root cause:

---

## Action Items

| # | Action | Owner | Due date | Status |
|---|---|---|---|---|
| Field | Field | Field | Field | Field |

---

## Lessons Learned

---

## Distribution

- [ ] Team shared knowledge base
- [ ] Service owner
- [ ] Agency IT leadership (if required by policy)
- [ ] Contracting officer (if required by contract)
- [ ] CISA / oversight body (if required by regulation)

---

Completed Example Postmortem

This is a real-format example — use it to calibrate what “done” looks like.

# Postmortem: Database Failover — 47-Minute Recovery Instead of 5

**Date of incident:** 2026-04-14
**Priority:** P1
**Duration:** 1 hour 15 minutes
**Postmortem author:** T. Reyes
**Review date:** 2026-04-16
**Status:** Final

---

## Incident Summary

On April 14, 2026 at 03:12 UTC, the primary database node for the citizen portal
failed due to disk exhaustion. Automatic failover to the replica did not occur.
The on-call engineer manually promoted the replica at 04:27 UTC. The service was
fully restored at 04:27 UTC. Approximately 2,400 users received errors or timeouts
during the impact window.

---

## Timeline

| Time (UTC) | Event |
|---|---|
| 03:12 | Primary database disk hits 100%. Write operations begin failing. |
| 03:12 | Disk alert fires. PagerDuty pages on-call engineer. |
| 03:15 | On-call engineer acknowledges page. |
| 03:22 | Incident declared (P1). Service owner notified. |
| 03:45 | Root cause identified: replica promotion script disabled on this host. |
| 03:50 | Decision made to manually promote replica rather than re-enable and debug the script. |
| 04:00 | Manual promotion complete. Application reconnects to new primary. |
| 04:15 | Service fully restored. Monitoring green. |
| 04:17 | Incident resolved. Postmortem meeting scheduled. |

---

## Contributing Factors

- The automatic replica promotion script was disabled on this replica 6 weeks prior.
- There was no alert or monitoring for "promotion script disabled."
- The maintenance runbook had no step to re-enable the script after maintenance.
- A previous incident with similar characteristics (8 months ago) produced no lasting action items.

---

## Impact Summary

| Dimension | Detail |
|---|---|
| Users affected | ~2,400 authenticated users |
| Services affected | Citizen portal (read and write operations) |
| Duration | 1 hour 15 minutes |
| SLA impact | Yes — SLA requires 99.5% monthly uptime; this incident consumed ~0.17% |
| Data loss | None |

---

## What Went Well

- On-call engineer responded within 3 minutes of the initial page.
- The runbook for manual replica promotion existed and was accurate.
- Service owner communicated a status update to agency stakeholders within 20 minutes.
- Post-incident disk cleanup was completed without a second outage.

---

## What Could Have Gone Better

- No monitoring existed for the disabled promotion script.
- The maintenance runbook had no verification step after completion.
- The previous postmortem (August 2025) identified the same risk but produced no completed action items.

---

## Five Whys Root Cause Analysis

| Round | Why? | Answer |
|---|---|---|
| Symptom | Failover took 47 minutes, not 5. | Replica was never promoted automatically. |
| Why 1 | Why was automatic promotion not working? | The promotion script was disabled on this replica. |
| Why 2 | Why was the script disabled? | An engineer disabled it during a maintenance window 6 weeks ago. |
| Why 3 | Why was it not re-enabled? | The maintenance runbook had no re-enable step. |
| Why 4 | Why was there no re-enable step? | The runbook predates the promotion script and was never updated. |
| Why 5 | Why was the runbook never updated? | No process exists to review runbooks when infrastructure changes. |

Root cause: no process for reviewing and updating runbooks when infrastructure components change.

---

## Action Items

| # | Action | Owner | Due date | Status |
|---|---|---|---|---|
| 1 | Add monitoring alert: "replica promotion script disabled" | K. Osei | 2026-04-21 | Open |
| 2 | Update maintenance runbook: add verification checklist at end | T. Reyes | 2026-04-21 | Open |
| 3 | Add disk-exhaustion runbook for primary and replica nodes | T. Reyes | 2026-04-28 | Open |
| 4 | Create process: runbooks must be reviewed when infrastructure changes | Service owner | 2026-05-05 | Open |
| 5 | Review all open action items from August 2025 postmortem | Team lead | 2026-04-21 | Open |

---

## Lessons Learned

Disabled automation with no monitoring is invisible risk. Any time a safety mechanism
is turned off — even temporarily — an alert should fire if it is not re-enabled within
a defined window. Runbooks are living documents. Treat them like code: review them
when the system they describe changes.

---

## Distribution

- [x] Team shared knowledge base
- [x] Service owner
- [x] Agency IT leadership
- [ ] Contracting officer (not required for this incident)
- [ ] CISA (threshold not met)

---

Next Steps

• Runbooks 101 — Learn how to write runbooks that stay accurate over time, including the review process that prevents the main anti-pattern in this guide.
• On-Call Basics — How to structure an on-call rotation, write escalation policies, and avoid burning out the engineers who respond to incidents.

---

Guide 7 of the Engineering Discovery series. Last verified: 2026-05-28.