On-Call and Escalation Basics
This guide explains what on-call engineering is, why teams that skip it face serious reliability and compliance problems, and how to set up a basic on-call rotation and escalation ladder for a government agency engineering team. It covers priority levels, who should be on the rotation, what an on-call engineer actually needs to do their job, and the specific legal and compliance obligations that government teams must account for during incidents. Engineering leads, program managers, and individual engineers at government agencies who are establishing a first on-call process or reviewing an existing one before it is tested by a real incident will find it most useful.
TL;DR
On-call means someone is responsible for being reachable and ready to respond to production incidents during off-hours. Without it, outages go undetected, incidents drag on, and one person ends up doing all the emergency work indefinitely.
- Define P1–P4 priority levels with explicit response time targets before your first incident.
- Have at least two people in the rotation: a primary and a backup.
- Government teams have legal escalation obligations for security incidents and PII breaches that go beyond engineering judgment.
- Write runbooks before you need them. Runbooks are what make an on-call rotation sustainable.
- Protect the on-call engineer: limit overnight pages to P1 and P2, provide time-in-lieu for weekend pages, and hold a retrospective after every significant incident.
On-Call Defined
On-call is a formal arrangement where one or more engineers agree to be reachable and able to respond to production incidents within a defined time window, even outside of normal business hours.
Being on-call does not mean sitting at a desk waiting. It means carrying a phone or pager, monitoring an alert channel, and being able to assess an alert and begin troubleshooting within the response time defined for that alert’s priority.
On-call is not a punishment. It is a professional engineering practice. Every service that is expected to run reliably outside of business hours requires someone who is responsible for it during those hours. Pretending otherwise does not make the service more reliable — it just makes the response more chaotic when something breaks.
The Cost of No On-Call Process
When no on-call process exists, several things happen predictably:
Undetected Downtime
Alerts fire into an inbox nobody monitors at night. Monitoring dashboards go unread. Citizens or staff discover the outage and report it hours later. By the time engineering knows about it, data may already be in an inconsistent state.
Extended Incident Duration
Without a defined escalation path, whoever discovers the incident has to find the right person, who has to find the right access, who has to find the right documentation — all under pressure, at 2 a.m. Mean time to resolution (MTTR) grows to hours instead of minutes.
Heroism Substituting for Process
One or two engineers end up as informal emergency contacts because they know the system best. They get called at all hours. They burn out. They leave. When they leave, the institutional knowledge goes with them and the situation gets worse.
Compliance Exposure
Most government agencies have legal obligations to report security incidents and data breaches within a defined window — often 24 to 72 hours from discovery. If nobody detects the incident for six hours because there is no on-call process, that window is already a third spent before engineering even begins to respond.
The cost of setting up an on-call process is a few hours of planning and a small ongoing time commitment per engineer. The cost of not having one is measured in outages, compliance failures, and engineer attrition.
Priority levels
Define your priority levels before you have an incident. Defining them during an incident leads to arguments, inconsistent responses, and delayed escalation. MDDS uses P1-P5 per the Incident Response runbook.
The table below summarizes the four active response levels. Adapt the examples to your specific services, but keep the response time targets close to these baselines.
| Priority | Name | Definition | Response time target | Escalation trigger | Example |
|---|---|---|---|---|---|
| P1 | Critical | System is completely down, data breach is confirmed or suspected, or a security control has failed | Acknowledge within 15 minutes. Incident declared immediately. | Escalate to Tier 2 if not resolved in 30 minutes. Notify agency IT manager immediately. | Benefits portal returns 500 errors for all users. Login.gov integration returning auth failures for 100% of requests. Suspected unauthorized access to production database. |
| P2 | Major | A major feature or user-facing function is broken, but the system is partially available. No confirmed data exposure. | Acknowledge within 30 minutes. | Escalate to Tier 2 if not resolved in 1 hour. | Document upload fails for all users. Payment processing is unavailable. Staff portal works but citizen portal is down. |
| P3 | Degraded | A function is impaired, but a workaround exists. The majority of users are unaffected. | Acknowledge within 2 hours during business hours. Non-urgent during off-hours. | Escalate if workaround fails or issue becomes P2. | PDF generation is slow but eventually succeeds. A non-critical report is failing to generate. Search results are delayed. |
| P4 | Minor | A low-impact issue with no user-facing degradation, or a cosmetic defect. | Acknowledge and log next business day. | No escalation required unless priority changes. | A typo in a non-critical UI element. A dashboard widget shows stale data by a few minutes. A non-essential background job failed once and retried successfully. |
Important: P1s and confirmed or suspected security incidents must always wake up the on-call engineer, regardless of the time. P3s and P4s should never generate a phone call or page outside business hours. They go into a ticket queue for the next business day.
Who should be on-call
The most common on-call mistake is assigning one person to own it all. That person burns out. Their on-call knowledge never transfers. When they quit, the team has nothing.
Structure your rotation with at least three roles:
Primary On-Call Engineer
The first responder. They carry the pager for the week. They acknowledge alerts, assess priority, follow runbooks, and either resolve the incident or escalate it. Ideally, the primary rotates across all engineers on the team, not just senior members.
Secondary On-Call Engineer
The person who is called if the primary does not acknowledge an alert within the response time window, or if the primary needs a second set of hands on a complex incident. The backup also covers for the primary if they become unavailable mid-incident.
Escalation Contact
The person — usually a tech lead or engineering manager — who handles incidents that require decisions beyond the primary engineer’s authority, or situations that need to involve non-engineering stakeholders. The escalation contact is not on-call in the sense of being paged for every alert, but they must be reachable within 30 minutes for P1 incidents.
Keep the on-call rotation to a cycle of no fewer than four engineers. With four engineers, each person is on-call roughly one week per month. With two engineers, they are on-call every other week, which is unsustainable for more than a few months.
Do not make the on-call rotation opt-in for senior engineers and mandatory for junior engineers. That is unfair and ineffective. Juniors should be on-call with a buddy system: pair a junior primary with a senior backup until the junior has responded to five or more incidents independently.
The escalation ladder in a government context
A government agency engineering team has a longer escalation chain than a startup. This is not bureaucracy for its own sake. Each level on the chain has a real function.
| Level | Who | Conditions for escalation | Their role in the incident |
|---|---|---|---|
| Tier 1 | On-call engineer (primary) | First to respond to any alert | Assesses, follows runbook, resolves or escalates |
| Tier 2 | Tech lead or engineering manager | P1 not resolved in 30 minutes; any P1 at the start if the primary needs judgment calls | Makes technical decisions, coordinates resources, decides when to involve non-engineering stakeholders |
| Tier 3 | Agency IT Director / CIO | Confirmed or likely breach; major outage affecting many users; any incident requiring public communication | Makes agency-level decisions, authorizes emergency changes, notifies agency head |
| Tier 4 | Agency head / Legal / Privacy Officer / Communications | Confirmed data breach; incident involving media attention; regulatory reporting obligation triggered | Authorizes regulatory notifications, manages external communications, legal liability decisions |
In a typical commercial engineering team, the chain stops at Tier 2 or Tier 3. In a government agency, Tier 4 is real and can be required by law.
When a security incident is detected — unauthorized access, anomalous data exfiltration, failed authentication from unexpected IP ranges — the cybersecurity team or CISO must be notified immediately, in parallel with the engineering escalation chain, not after the engineers have “ruled things out.” Investigation is their job, not yours. Your job is to notify, preserve evidence (do not reboot servers before forensics has run), and follow their instructions.
When to escalate beyond engineering
Escalating beyond engineering is not a failure. It is the correct action when a situation exceeds what engineers can or should handle alone.
Security Incidents
Any confirmed or suspected unauthorized access to systems or data. Notify the cybersecurity team immediately. Do not attempt to clean up the incident before they have assessed it, because you may destroy forensic evidence. Do not discuss a suspected security incident in public Slack channels.
PII or PHI Data Breach
Any confirmed or likely exposure of personally identifiable or protected health information. Notify Legal and the Privacy Officer immediately. Most states have breach notification laws that require agencies to notify affected individuals within 30 to 72 hours of discovery. Federal systems may be subject to additional reporting requirements under the Federal Information Security Modernization Act (FISMA). These clocks start at discovery — the moment engineering confirms or reasonably suspects a breach — not at the moment the legal team is notified.
Vendor or Third-Party Outage
If a critical third-party service such as an identity provider, payment processor, or federal API is down and the outage is not in your control, notify the contract officer responsible for that vendor relationship. They have contact methods and contractual escalation rights that engineers typically do not. Do not spend engineering hours on hold with vendor support if a contract officer can escalate through procurement channels.
Media Inquiry
If a reporter calls or emails asking about an outage or incident, transfer them immediately to the agency communications office. Engineers do not speak to the press about incidents. This is standard government policy and protects both the agency and the engineer.
What an on-call engineer needs
An on-call rotation fails when engineers are expected to respond to incidents but are not given the tools to do so. Before your first rotation goes live, confirm that every on-call engineer has:
- Runbooks. Written, tested step-by-step procedures for the most common failure scenarios. Without runbooks, on-call is guesswork under pressure.
- Monitoring access. Read access to the dashboards, logs, and metrics for every service they are responsible for. Access provisioned at 3 a.m. during an incident is access that arrives too late.
- A way to declare an incident. Whether it is a Slack command, a PagerDuty incident, a ServiceNow ticket, or a phone tree — the process must be written down and practiced before it is needed.
- A communication channel. A dedicated incident channel (Slack, Teams, or equivalent) where the incident timeline is documented in real time. This serves as the incident log.
- Vendor contact information. A list of emergency support contacts for every third-party system in the production environment. Include account numbers, SLA tier, and the after-hours phone number where applicable.
- Emergency change authorization. In many government environments, system changes require change advisory board (CAB) approval. This is incompatible with a 3 a.m. incident response. Establish a pre-approved emergency change process with your IT governance office before you need it.
- Knowledge of what they do and do not have authority to do. Can the on-call engineer roll back a deployment alone? Can they restart a database? Can they take a system offline? Write this down explicitly. Ambiguity about authority leads to delayed action during incidents.
On-Call Health and Sustainability
An on-call rotation that is not sustainable will collapse. Engineers will find ways to avoid the rotation, leave the team, or — worse — stop responding to alerts because the volume is too high and most alerts are noise.
Limit Off-Hours Pages to P1 and P2
If engineers are being woken up for P3 alerts, fix the alert threshold, not the engineer’s willingness to respond. Alert fatigue is a real phenomenon that degrades incident response quality.
Minimum Four-Person Rotation
Below four people, the rotation is unsustainable. Below two people, it is not a rotation at all.
Time-in-Lieu for Overnight and Weekend Pages
An engineer who is paged at 2 a.m. on a Saturday and spends three hours on an incident should not be expected to work a full day on Monday. This is both fair and practical — a fatigued engineer is more likely to make mistakes during the next incident.
Blameless Retrospective After Every Significant Incident
Run it blameless: the goal is to understand what failed and improve it, not to identify who is at fault. Document what happened, what the resolution was, what the runbook got wrong, and what changes are needed to prevent a recurrence. Retrospective notes become the primary input for improving runbooks.
Monthly Alert Volume Review
Track how many pages fire in off-hours versus business hours, how many are actionable versus noise, and what the average resolution time is. Increasing page volume with no corresponding increase in resolved incidents means alert thresholds need adjustment.
Government-specific considerations
Breach notification timelines
Most U.S. states have data breach notification laws that require agencies to notify affected individuals and in some cases state regulators within a defined window after discovery of a breach. This window is typically 30 to 72 hours at the state level. Some states (California, New York, Virginia) have stricter or more specific requirements.
Federal systems have additional requirements. FISMA requires agencies to report information security incidents to US-CERT within one hour for the highest priority levels. The specific reporting requirements depend on the impact level (Low, Moderate, High) of the affected system.
Do not assume that “we haven’t confirmed it was a breach” stops the notification clock. Most notification laws trigger on discovery of a suspected breach, not a confirmed one.
Your agency’s Privacy Officer and Legal team will know your specific obligations. The on-call engineer’s job is to notify them immediately so those obligations can be met — not to make the legal determination themselves.
FISMA incident reporting
Systems authorized under FISMA are required to report incidents to the agency’s information security program office and to US-CERT (operated by CISA) within defined timelines. The NIST SP 800-61 Computer Security Incident Handling Guide provides the framework most federal agencies follow.
Key categories from NIST 800-61 that trigger mandatory reporting: unauthorized access, denial of service affecting mission-critical systems, malicious code on a government system, and improper usage involving sensitive data. Include a step in your P1 and security incident runbooks that explicitly calls out FISMA reporting obligations: “If this incident may be reportable under FISMA, notify [agency security officer] immediately with the incident details.”
Chain-of-command notification for P1/P2 incidents
Most government agencies have a required notification chain for high-priority incidents. This is separate from the engineering escalation chain. An outage affecting hundreds of thousands of citizens is a policy event, not just a technical event.
Document the chain of command for your agency’s notification requirements before your first P1 incident. Who needs to know? In what order? Within what time window? What format does the notification need to take? A phone tree that gets figured out during a live P1 incident is a phone tree that will fail when you need it most.
Escalation ladder template
Copy and adapt this template for your team. Fill in real names and contact information, and store it somewhere every on-call engineer can access in under 30 seconds.
| Level | Role | Who (name) | Contact method | Response time target |
|---|---|---|---|---|
| Tier 1 — Primary | On-call engineer | [Rotating — see schedule] | PagerDuty / phone | Acknowledge within 15 min (P1), 30 min (P2) |
| Tier 1 — Secondary | Backup on-call | [Rotating — see schedule] | PagerDuty / phone | Acknowledge if primary does not respond in 10 min |
| Tier 2 | Tech Lead / Eng. Manager | [Name] — [phone] | Phone call | Available within 30 min for P1 escalation |
| Tier 2 — Security branch | Cybersecurity / CISO | [Name] — [phone] | Phone call | Notify immediately for any security incident |
| Tier 3 | Agency IT Director | [Name] — [phone] | Phone call | Available within 1 hour for P1 breaches or major outages |
| Tier 4 — Legal | Privacy Officer / Legal | [Name] — [phone] | Phone call | Notify immediately if PII/PHI breach is confirmed or suspected |
| Tier 4 — Communications | Agency Communications | [Name] — [phone] | Phone call | Notify before any public statement about the incident |
On-call vs. a help desk
On-call and a help desk are different functions. Confusing them creates gaps in both directions.
A help desk handles user-facing requests during business hours: password resets, access requests, “my account isn’t working” tickets. Help desk staff follow scripts. They escalate to engineering when a ticket is beyond their scope. Help desk tickets are low-urgency by design.
An on-call engineer handles production system failures at any hour. They investigate and fix infrastructure and application problems. They follow runbooks but also apply technical judgment. On-call issues are often urgent by definition.
The confusion arises when agencies create a “24/7 help desk” and assume it covers after-hours system outages. It does not. A help desk agent who cannot access server logs or deploy a rollback cannot resolve a P1 incident, regardless of what hours they work.
Both functions are necessary. They need separate processes, separate escalation paths, and separate tooling.
Figure 1 (placeholder) — An escalation ladder diagram for a government agency engineering team. Show three tiers arranged vertically: Tier 1 (On-Call Engineer — responds first, owns the incident), Tier 2 (Tech Lead / Engineering Manager — escalated when Tier 1 cannot resolve in 30 minutes or for any P1), Tier 3 (Agency IT Director plus Legal plus Communications — escalated for breaches, major outages, or media-sensitive events). Use upward vertical arrows between tiers labeled with escalation conditions. Add a separate horizontal branch from Tier 2 to a Cybersecurity / CISO box, labeled ‘Any security incident — immediate parallel notification.’ Show a secondary backup at Tier 1 as a sibling box to the primary on-call engineer.
Next steps
With a working on-call rotation and escalation ladder in place, the most impactful improvements are:
- Runbooks 101 — How to write step-by-step operational procedures that make your on-call rotation sustainable and effective. A rotation without runbooks depends entirely on tribal knowledge.
- Blameless Postmortems — How to run a retrospective after an incident that improves the system rather than assigning blame. The output of a good postmortem is improved runbooks and reduced future incident frequency.
- Alert hygiene — How to review and right-size your monitoring alerts so that the signal-to-noise ratio is high enough to trust.