Why Incidents Happen

Why this matters

Incidents happen when an assumption survives long enough to reach production. The useful lesson is not whether one person was careful enough. It is which product, engineering, release, or operational guardrail failed to catch the assumption earlier. Use this page after an incident, near miss, or surprising production defect to turn “what broke” into prevention work.

Mental Model

Think of an incident as an escaped assumption. Someone assumed a state could not happen, a dependency would stay healthy, or a migration would stay fast. Someone assumed staging was representative, or infrastructure metrics would reveal user pain.

The post-incident job is to identify the assumption, find the missing guardrail, and create proof that the guardrail now exists. If the follow-up does not create proof, the team is relying on memory, and memory is not a control.

Incident Learning Loop

flowchart TD
    IncidentLead(("Incident Lead")) --> Assumption["Name the escaped assumption"]
    Assumption --> Failure["Production failure or near miss"]
    Failure --> Guardrail["Identify the missing guardrail"]
    Guardrail --> Artifact["Create prevention artifact"]
    Artifact --> Verify["Verify it catches the failure"]
    Verify --> Release["Release with stronger control"]
    Release --> Observe["Monitor the same failure class"]
    Observe --> Guardrail

Notice the loop does not stop at restoring service. The durable work is the prevention artifact plus verification that it catches the same class of failure. That proof is what makes the lesson reusable.

What to Capture

Capture enough detail to change the system, not just explain the outage. These questions turn incident notes into delivery work that can be reviewed and verified.

Which user journey or operational path failed?
Which assumption allowed it through review, testing, staging, or monitoring?
Which system guardrail should have failed the release earlier?
What artifact proves the guardrail is now in place?
Who owns the prevention action, and when is it validated?

Barry’s Incident

Incidents are culture tests. The same failure can teach a team to hide risk, or it can teach the team to make risk visible sooner.

The punitive version

Imagine Barry ships a small case-routing rule on Thursday afternoon. The change passes review, staging, and unit tests, but production has old locked-account records that staging never represented. On Friday morning, users can update their cases, but some records never move to review.

Barry’s manager treats the incident as a personal failure. Barry is criticized in the incident review and written up for not catching the edge case. Barry never makes that exact mistake again, but the organization learns the wrong lesson.

The next month, engineers stop saying “I am not sure what happens here” in review. They delay releases, avoid owning risky work, and escalate small decisions because being visible now feels dangerous. Development slows because fear turns every change into a career risk.

The damage does not leave when Barry’s manager leaves the organization. A scared culture takes time to unlearn, and people protect themselves by hiding mistakes early. The organization loses because institutional knowledge gets covered up instead of shared.

The learning version

Now replay the same incident with a systems response. Barry still owns the change, but the review starts with a better question: “What assumption survived long enough to reach production?”

The team finds three gaps. Product never specified locked-account behavior, staging did not include old locked records, and monitoring watched server health instead of case progression. Barry did not fail alone; the delivery system let an assumption pass through every checkpoint.

The follow-up changes the system. Product adds acceptance criteria for locked and stale records, and engineering adds an API contract test. Storybook gets an edge-state example, staging gets a messy fixture, and the team alerts when case progression stalls. The next engineer gets caught by the guardrail before users do.

The outcome to create

The goal is not to make Barry afraid of mistakes. Barry is allowed to make mistakes because everyone grows from what the team can see. The goal is to make the next mistake smaller, earlier, and easier to catch.

A healthy incident review makes truth safe and vague follow-up uncomfortable. Engineers should leave more willing to name uncertainty, not less. Managers should leave with stronger controls to fund, review, and verify.

Better Follow-Up

After the incident, restore service and close the gap that made the failure possible. A one-off patch restores behavior; preventive work changes the system so the same failure class is significantly less likely to recur. Good follow-up work has a named owner, due date, and verification step.

Vague items like “be more careful with migrations” do not reduce risk. Specific items do: “Add a migration smoke test for legacy null fields and fail CI if the rollback path is missing.”

Weak follow-up	Better follow-up
Be more careful with migrations.	Add a migration smoke check for legacy null fields and fail CI when the rollback path is missing.
Improve monitoring.	Alert when intake completion rate drops below the agreed SLI for more than one measurement window.
Add more tests.	Add an integration test that sends duplicate webhooks, delayed retries, and legacy records through the full workflow.
Document the edge case.	Add acceptance criteria, API contract tests, and a Storybook example for the edge state.
Talk to the vendor.	Add vendor timeout budgets, escalation owner, data export path, and fallback behavior to the runbook.

Follow-Up That Does Not Reduce Risk

Do not stop at the immediate code fix.
Do not write action items that depend on memory or heroics.
Do not treat the person closest to the failure as the single point of failure.
Do not call the incident resolved while the prevention item is ownerless.
Do not treat post-incident notes as a substitute for delivery work.

Takeaway

An incident is resolved when service is restored. It is learned when the failed assumption has been turned into a guardrail that future work can run against.