Skip to content
Documentation

Runbooks 101

This guide explains what runbooks are, why they are a required operations artifact for government engineering teams, and how to write one that actually works during an incident. It covers the anatomy of a good runbook, how to avoid the most common runbook failures, government-specific requirements for audit and compliance, and a full worked example you can adapt for your own team. On-call engineers, tech leads, and operations staff at government agencies preparing for their first on-call rotation or an upcoming ATO review will find it most useful.

TL;DR

A runbook is a written, step-by-step procedure for responding to a specific operational situation. It exists so that any engineer on your team — not just the one who built the system — can resolve a known failure without having to figure it out from scratch at 3 a.m.

  • Every runbook must have a trigger condition, numbered steps, success criteria, and an escalation path.
  • Runbooks are evidence in audits and ATO reviews. Keep them current, dated, and version-controlled.
  • Government systems that handle PII must include a “stop and notify legal” step in any runbook where data exposure is possible.
  • A runbook that has not been tested is a runbook that will fail when you need it. Test them during a low-pressure window.
  • Write your first runbook from the last incident you had to figure out manually.

Runbook Anatomy

A runbook is a step-by-step written guide for responding to a specific, known operational situation. The situation could be a system failure, a routine maintenance procedure, a deployment rollback, or any other event that requires a predictable, repeatable response.

The defining characteristic of a runbook is specificity. “Check if the service is running” is not a runbook step. “Run kubectl get pods -n production -l app=api-service and look for pods in a state other than Running” is a runbook step. Specificity is what makes a runbook usable by someone who did not write it.

A runbook is not a design document — it does not explain why the system was built a certain way, it explains what to do right now in response to a specific situation. The reader during an incident is under pressure and needs to act, not learn.

Runbooks vs. Playbooks

These terms are often confused because they cover different levels of a response.

A runbook is tactical and procedure-level. It covers one specific scenario: “The API returns 502 errors after a deploy.” “The database disk is full.” “The nightly batch job fails to complete.” A runbook has a defined entry condition, numbered steps, and a defined exit condition (either resolved or escalated).

A playbook is strategic and incident-wide. It covers a class of incidents from start to finish: “How we respond to a data breach.” “How we handle a DDoS attack.” “How we respond to a ransomware event.” A playbook orchestrates multiple runbooks, defines roles and communication strategies, and handles coordination across teams. A playbook tells you which runbooks to run and in what order.

Both are necessary — build runbooks first, since they are the building blocks that make playbooks workable.

Why Teams Need Runbooks

Knowledge Transfer

In most teams, one or two engineers know the system deeply enough to handle a 3 a.m. incident without documentation. When those engineers are on vacation, sick, or have left the team, everyone else is guessing. A runbook converts that individual knowledge into organizational capability.

Faster Incident Resolution

An engineer following a well-tested runbook will resolve a known incident faster than an engineer figuring it out from scratch. Every minute of reduced MTTR is a minute of service restored to citizens or staff.

Compliance and Audit Evidence

Auditors reviewing an agency’s operational security posture will ask: “How do you respond to known failure modes?” A runbook library is the answer. An ATO review, a SOC 2 audit, or a FedRAMP continuous monitoring review may all ask for evidence of documented incident response procedures. “We know what to do” is not acceptable evidence. A dated, version-controlled runbook is.

Sustainable On-Call Rotations

An on-call rotation where every engineer has to figure things out from scratch is not a rotation — it is organized heroism. Runbooks are what make it possible to put junior engineers on-call without requiring senior engineers to be on standby every night.

Required Runbook Sections

A runbook that is missing any of these sections is incomplete. During an incident, a gap in the documentation is time lost.

Title and Scope

A plain-language title that describes the situation the runbook addresses. The scope clarifies which system or service it covers and what it does not cover. If the runbook only applies to the production environment, say so.

Trigger Condition

When should an engineer reach for this runbook? This can be the name of a specific monitoring alert, a user-reported symptom, or a combination of both. The trigger condition should be precise enough that the engineer does not have to guess whether this runbook applies to their situation.

Preconditions

What must be true before the engineer starts? This includes access requirements (which systems, which permissions, which tools), information to gather first (which deployment just ran, which user is affected), and any safety checks (is it safe to restart this service right now?). An engineer who discovers midway through a runbook that they do not have the required access is an engineer who wastes time and delays resolution.

Steps

Every action in sequence, with the exact command, URL, or UI path, the expected output, and what to do if the actual output differs. Do not write steps like “fix the database.” Write steps like “Run pg_top and look for queries with a lock wait time over 10 seconds. Note the query PID.”

Success Criteria

How does the engineer know the incident is resolved? Define a specific, measurable condition — not “it seems to be working again.” For example: “The P95 response time on the /api/applications endpoint drops below 800ms for five consecutive minutes.”

Escalation Path

If the runbook does not resolve the issue within a defined time window, who does the engineer call and what information do they bring? Specify the Tier 2 contact and list what to include in the escalation: the alert URL, the step where the runbook stopped working, any error output found.

Review and Test History

A runbook with no date is a runbook of unknown reliability. Include the date it was last reviewed and the date it was last actually used or rehearsed. Assign an owner responsible for keeping it current.

Common Runbook Failures

Vague Steps

This is the most common runbook failure. A step that requires the reader to already know how to do the thing the runbook is trying to teach them is not a step — it is a placeholder. Assume the reader knows almost nothing about this specific system’s internals.

Assumed Institutional Knowledge

“Check the usual dashboard” does not tell an engineer who started last month where to go. “Open Datadog, navigate to Service: api-service, and look at the P95 Latency graph over the last 30 minutes” is a usable step.

Outdated Content

A runbook that references a service that was renamed six months ago, or a Kubernetes namespace that no longer exists, or a Slack channel that was archived, is a runbook that breaks at a critical moment. Assign ownership and review after every system change.

Happy-Path-Only Coverage

If step 3 says “this should succeed” and it does not, what does the engineer do? Every decision point in a runbook needs a branch: “If you see X, go to step 4. If you see Y instead, go to step 8.”

Cognitive Overload

If the engineer has to remember the output of step 2 while doing step 9, extract that output into a clearly labeled intermediate state. Use checklists and sub-steps to make the procedure easier to follow under pressure.

Undocumented Procedures

If the procedure exists only as an undocumented practice that the senior engineer follows from memory, it is not a runbook — it is a dependency on one person.

Government-specific considerations

Runbooks as audit evidence

In a government context, runbooks are not just operational convenience — they are documentary evidence. An auditor reviewing your security posture will ask how your team responds to known failure modes. A runbook library with dated reviews and documented testing is the correct answer. “We know what to do” is not.

This matters for ATO reviews, FedRAMP continuous monitoring, SOC 2 audits, and state-level security reviews. Keep your runbooks in version control, not in a wiki with no history. Use meaningful commit messages when you update them. That history is evidence of an active, maintained operational practice.

Change control in government environments

Many government agency systems operate under change control processes that require approval from a change advisory board (CAB) before any configuration change is made to production — even during an incident.

This creates a real problem if your runbooks require you to make configuration changes to resolve an incident at 3 a.m. on a Saturday. The solution is not to skip change control — it is to establish a pre-approved emergency change procedure with your IT governance office before you need it.

A pre-approved emergency change procedure defines the categories of change that can be made without advance CAB approval, the documentation required after the fact (change record created within 24 hours), and who must be notified (CAB chair, IT director). Work this out in advance and include a step in affected runbooks that says: “This step requires an emergency change record. Create one at [URL] before proceeding.”

PII exposure check — a mandatory runbook step

Any runbook for a system that handles PII, PHI, or other sensitive data must include a PII exposure check step. This step appears whenever there is any possibility that the failure mode exposed data to unauthorized parties.

The step should read approximately: “If at any point in this process you believe citizen or patient data may have been accessed or exposed without authorization, stop troubleshooting. Contact [Privacy Officer name and phone number] and [Legal counsel name and phone number] immediately. Do not continue without their guidance. Preserve logs and do not restart or redeploy the affected service until forensics has reviewed it.”

This step is not optional. Breach notification clocks start at discovery. An engineer who spends two hours trying to “figure out for sure” before notifying legal has potentially cost the agency a compliance violation.

Runbooks for emergency changes involving classified or restricted systems

For systems with higher classification levels, runbooks may themselves require classification marking, restricted distribution, or storage in systems approved for that classification level. Consult your agency’s information security office for guidance before storing runbooks in any system not cleared for the sensitivity of the information they contain.

Writing Your First Runbook

Pick the incident your team has resolved manually at least twice. That repetition is what makes something a runbook candidate — if you have done the same thing twice, you will do it again, and the third time should be faster and less stressful.

Describe the Trigger

In one sentence, describe what went wrong. “The application returned 502 errors after a deployment.” “The nightly batch job did not complete by 6 a.m.” “A user reported they could not log in.”

Document What You Actually Did

Document what you actually did — not what you should have done. This is the raw material for the runbook steps.

Add the Reference Information

Every time you opened a browser tab, asked a colleague, or dug through logs to find a value — that is a step that belongs in the runbook.

Add Decision Branches

Where did the resolution depend on what you found? If you checked logs and they showed error A, you did X. If they showed error B, you would have done Y. Document both.

Define Success Criteria

What did “working again” look like? How did you know the incident was over?

Add the Escalation Path

If the steps had not worked, who would you have called?

Test the Runbook

Walk through the runbook with a colleague who was not involved in the original incident. Can they follow it without asking you questions? If not, the runbook is not done. The whole process takes 30 to 60 minutes for a routine incident, and a team that does this consistently after every incident builds a useful runbook library within a few months.

Runbook Storage Formats

Markdown in the Repository

Storing runbooks as Markdown files in the same repository as the code they cover means they go through code review, are version-controlled with a meaningful history, and are available to any engineer with repository access. Add a /runbooks or /docs/runbooks directory and name files descriptively: 502-after-deploy.md, database-disk-full.md.

Internal Wiki

Accessible to non-engineers and easy to link from monitoring alerts. The downside is that wikis often lack meaningful version history and runbooks drift out of date without the friction of a pull request review. If you use a wiki, link each runbook from the relevant monitoring alert so engineers land on the right procedure when the alert fires.

Alert-Linked Runbooks

Both PagerDuty and OpsGenie support attaching a runbook URL to an alert or service. This is the highest-friction-reduction option: when the alert fires, the on-call engineer sees the runbook link in the same notification that woke them up. Use this regardless of where the runbook is actually stored.

Avoid email, Slack messages, or personal notes — they are not runbooks, and they will not be found by the engineer who needs them at 3 a.m.

Keeping Runbooks Alive

A runbook that is not maintained becomes a liability — an engineer who follows an outdated runbook wastes time and may make the incident worse.

Post-Incident Review

When a runbook was used in a real incident, the engineer who used it updates it before closing the incident ticket. What step was wrong? What was missing? What would have made it faster?

Quarterly Audit

Once per quarter, assign one engineer per runbook to read through it and verify it against the current state of the system. Update service names, namespace changes, URL changes, and contact information. Log the review date.

Named Ownership

Every runbook has a named owner responsible for keeping it current. When the owner leaves the team, transfer ownership explicitly. A runbook with no owner is a runbook that will go stale.

Last Tested Date, Not Just Last Reviewed Date

Reading a runbook and verifying it is different from following it. Ideally, teams rehearse runbooks during a scheduled low-pressure window — a “game day” or quarterly drill. Runbooks that have been actually followed produce fewer surprises during real incidents.

Archive Runbooks for Decommissioned Systems

A library full of outdated runbooks is harder to navigate than a smaller, accurate one — when you decommission a service, archive or delete its runbooks.

Worked example: Application returns 502 Bad Gateway after deploy

The following is a complete runbook using the required structure. It covers a common scenario that most teams encounter when running containerized applications.

Runbook ID: RB-OPS-001

Title: Application Returns 502 Bad Gateway After Deploy

Owner: Platform Engineering team

Last reviewed: 2026-05-01 by Jane Smith

Last tested: 2026-04-15 during quarterly runbook rehearsal

Scope

This runbook applies to the api-service production deployment when the load balancer returns HTTP 502 errors within 30 minutes of a deployment. It covers rollback, log analysis, and upstream dependency checks. It does not cover 502 errors caused by infrastructure issues outside the application layer (see RB-INFRA-007 for network-level issues).

Trigger condition

  • Monitoring alert “api-service — 502 rate exceeds 1% over 5 minutes” fires in PagerDuty.
  • Users or help desk report “the system is down” or “getting an error” within 30 minutes of a deployment.

Preconditions

Before starting, confirm you have:

  • Read access to the production Kubernetes cluster (kubectl get pods -n production)
  • Access to the deployment history in the CI/CD system (CircleCI / GitHub Actions)
  • Access to the Datadog service dashboard for api-service
  • PagerDuty incident created and shared in #incidents Slack channel

PII exposure check

Before proceeding: If this incident involves any indication of unauthorized data access (unexpected external traffic, data in access logs that should not be there, an alert from the WAF about data exfiltration), stop. Do not continue troubleshooting. Contact the Privacy Officer at [phone] and Legal at [phone]. Preserve logs. Do not restart or redeploy the service until forensics has reviewed the environment.

Steps

  1. Open the Datadog dashboard for api-service and note the time the 502 rate started rising. Write it in the incident channel.

  2. Check whether a deployment was made in the last 30 minutes. Open the CI/CD system and look at the deployment history for api-service.

    • If a deployment was made in the window: go to step 3.
    • If no deployment was made: go to step 9.

  3. Check the current pod status: kubectl get pods -n production -l app=api-service

    • Expected: All pods show Running with 1/1 ready.
    • If pods show CrashLoopBackOff or ImagePullBackOff: the new container image is broken. Go to step 4.
    • If pods show Running but 502s are still happening: go to step 6.

  4. Roll back the deployment: kubectl rollout undo deployment/api-service -n production

  5. Wait 90 seconds. Run kubectl get pods -n production -l app=api-service again.

    • If pods are Running and the 502 rate in Datadog is falling: go to “Success criteria.”
    • If pods are still failing: go to step 12 (escalate).

  6. Check application logs for the new pods: kubectl logs -n production -l app=api-service --tail=100 --since=10m

    • Look for: connection refused to a database or downstream service, configuration errors, out-of-memory kills.
    • If you see “connection refused” or “dial tcp” errors to the database: go to step 7.
    • If you see an application startup error or missing environment variable: go to step 8.
    • If logs look clean and healthy: go to step 9.

  7. Check the database status in Datadog (service: postgres-primary). Look at CPU, connection count, and query latency.

    • If the database shows elevated CPU or connection count: this is a database capacity issue, not a deployment issue. Notify Tier 2 immediately. Do not roll back the deployment — it will not help.
    • If the database looks healthy: the connection string or credentials may have changed in the deployment. Check the deployment diff in the CI/CD system for environment variable changes.

  8. If you see a missing environment variable or configuration error in the logs: check the deployment manifest for missing or changed config map or secret references. Do not attempt to patch the secret in production directly — roll back the deployment (kubectl rollout undo deployment/api-service -n production) and fix the configuration in the deployment pipeline before redeploying.

  9. If no recent deployment: check upstream dependencies.

    • Open the Datadog service map and look for red indicators on services that api-service depends on (identity service, payment gateway, federal API).
    • Check vendor status pages: [link to vendor status page list in team wiki].
    • If a dependency is down: open a P2 incident, notify the contract officer for that vendor (see vendor contact list), and set up monitoring for recovery. Do not continue troubleshooting api-service — the root cause is upstream.

  10. If all dependencies are healthy and logs are clean: this may be a transient traffic spike or a load balancer health check misconfiguration. Check the load balancer target group health in AWS Console or equivalent. Look for targets being marked unhealthy.

  11. If you cannot identify a root cause after 20 minutes: go to step 12.

  12. Escalation. Page Tier 2 on-call (Tech Lead) via PagerDuty. Include in the escalation:

    • The incident Slack channel URL
    • The deployment SHA that was in progress (if any)
    • The pod status output from step 3
    • The most relevant log lines from step 6
    • The step number where this runbook stopped being useful

Success criteria

The incident is resolved when all of the following are true for five consecutive minutes:

  • kubectl get pods -n production -l app=api-service shows all pods as Running with 1/1 ready.
  • Datadog shows the 502 error rate below 0.1%.
  • The P95 response time on api-service is below 1000ms.

When success criteria are met: close the PagerDuty incident, post a brief summary in #incidents, and create a postmortem ticket if the incident lasted more than 30 minutes or involved a rollback.

Known issues and notes

  • The rollback in step 4 takes approximately 90 seconds to complete. The 502 rate may not drop immediately — wait for all new pods to terminate before declaring success.
  • The api-service load balancer health check has a 30-second timeout. Even after pods are Running, the load balancer may continue routing to old pods for up to 30 seconds.
  • If this runbook was reached from the “nightly batch job failed” alert instead of the “502 rate” alert, the batch job is a separate process — check RB-OPS-004 for batch job failures.

Figure 1 (placeholder) — A flowchart showing the runbook decision tree for the ‘502 Bad Gateway after deploy’ incident described in the worked example. Show: Alert fires → Check for recent deploy → Yes branch: check pod status → CrashLoopBackOff: roll back → pods recover: success, or still failing: escalate. Running pods branch: check logs → database connection error: check database → database issue: notify Tier 2, or connection string issue: roll back and fix config. Clean logs: check upstream dependencies → dependency down: notify contract officer, or all clear after 20 minutes: escalate. No recent deploy branch connects to the upstream dependencies check. Label each decision diamond with the question and each branch with the answer.

The runbook lifecycle

Runbooks do not write themselves and they do not maintain themselves — they need to go through a deliberate cycle to stay useful.

Write

The best time to write a runbook is immediately after an incident, while the resolution steps are fresh. The second-best time is after a near-miss — a situation where you almost had to debug something from scratch, but caught it in time. Write from what actually happened, not from what you planned would happen.

Test

Schedule a low-risk window — a weekday morning when traffic is low — and walk a junior engineer through the runbook step by step. Have them do the commands, follow the branches, and ask questions. Where they get confused, the runbook needs more detail.

Use

During a real incident, an engineer follows the runbook as written. If the runbook is wrong, they note where it diverged from reality and continue with their best judgment. They do not stop to update the runbook in the middle of an incident.

Review

After the incident is resolved, the engineer who used the runbook updates it: fixes the incorrect step, adds the missing branch, updates the success criteria. They set the new “last tested” date and push the change through code review.

Figure 2 (placeholder) — A runbook lifecycle diagram showing four stages arranged in a cycle: Write (after a new incident type is identified or a near-miss occurs), Test (engineer rehearses the runbook during a low-risk window or game day), Use (on-call engineer follows it during a real incident), Review (after the incident, the runbook is updated with what actually worked and the date is updated). Connect the four stages with arrows in a clockwise cycle. Add a separate ‘Quarterly Audit’ node connected to the Review stage with a dashed arrow, labeled ‘Even if no incident occurred — verify steps are still accurate.’ Add a note that runbooks with no review in 12 months should be archived or deleted.

Runbook template

Copy the template below, replace every bracketed value with your team’s specifics, and store it in version control.

# Runbook


| Field | Entry |
|---|---|
| Title | - |
| Runbook ID | - |
| Owner | - |
| Last reviewed | - |
| Last tested | - |


---


## Scope


## Trigger condition


## Preconditions


Before starting, confirm you have:


- [ ]


## PII / PHI exposure check


If at any point you believe citizen or patient data may have been accessed or exposed
without authorization: STOP. Do not continue troubleshooting. Call the Privacy Officer
and Legal contact. Preserve logs. Do not restart or redeploy the service until they have
assessed the situation.


## Steps


1.
   - Expected result:
   - If different:


## Success criteria


The incident is resolved when all of the following are true:


-


After resolution:


## Escalation


If this runbook has not resolved the issue within the escalation window, page the Tier 2
contact. Include in the escalation:


- The incident channel URL
- The step number where the runbook stopped being useful
- The exact error message or unexpected output you found
-


## Notes and known issues


---


*Review this runbook after every incident where it was used.
Update the "Last reviewed" date and push the change to version control.*

Next steps

With your first runbook written, the most valuable next investments are:

  • On-Call and Escalation Basics — How to structure the rotation and escalation ladder that determines who uses the runbooks and when. Runbooks without an on-call process are instructions with no one to follow them.
  • Blameless Postmortems — How to turn every incident into a structured learning opportunity. The postmortem is the primary input for improving runbooks after the fact.
  • Monitoring and Alerting basics — How to make sure the alerts that trigger your runbooks are accurate, specific, and waking up the right people for the right reasons.

External References