Monitoring 101 for Delivery Teams

This guide explains the basics of monitoring a government web service — what to measure, how to alert well, and how to build from nothing to a solid baseline in 90 days. Junior and mid-level engineers on delivery teams that are new to monitoring, or teams inheriting a system that currently has no monitoring in place, will find it most useful.

TL;DR — Minimum Monitoring Checklist

If your service passes every row in this table, you are in decent shape. If not, this guide explains how to get there.

Done	Check	Why it matters
- [ ]	HTTP 5xx error rate is tracked	Tells you when users are hitting broken pages
- [ ]	Response time (p95) is measured	Catches slowdowns before users call the help desk
- [ ]	Server CPU and memory have alerts	Prevents crashes from resource exhaustion
- [ ]	At least one uptime check exists	Tells you when the service is completely unreachable
- [ ]	Alerts reach a real person or team, not a shared inbox	Ensures someone actually sees and responds
- [ ]	Alert thresholds are documented	Prevents alert fatigue and mystery numbers
- [ ]	External dependency health is tracked	Catches upstream failures that are not your code

Monitoring, Logging, and Observability

Monitoring is the practice of collecting data about your system while it runs and alerting someone when something looks wrong. It is not the same as logging — logs are records of what happened, an event journal, while monitoring watches those records (and other numeric signals) for patterns that suggest a problem. You need both, but they serve different purposes.

Monitoring is also not the same as observability. Monitoring tells you when something is broken. Observability helps you figure out why it broke. Think of monitoring as a smoke alarm and observability as the fire investigation afterward. Both matter, but you start with the alarm. See Observability 101 for the next step up.

External Resources

The Cost of No Monitoring

Without monitoring, you learn about problems in one of two ways: a user calls your help desk, or the service falls over completely. Both are bad outcomes.

Government services often have constituents who cannot easily switch to another option. A broken benefits portal, a failed permit filing system, or an unavailable court date scheduler affects real people with real deadlines. These are not inconveniences — they can mean missed legal deadlines, delayed benefits, or loss of income.

Silent failures are the most dangerous kind. This is when the system appears up but is returning errors to a subset of users, silently dropping form submissions, or producing incorrect output. No alert fires and no one knows to fix anything — users give up or call a help desk that has no context on what is wrong.

Teams without monitoring also cannot answer the basic questions that arise during and after an incident: When did this start? How many users were affected? Is it getting better or worse after our fix? Was it caused by our last deploy? These questions are required for post-incident reports, which are often mandatory in government contracts, ATO processes, and inspector general reviews.

The Four Golden Signals

Google’s Site Reliability Engineering book defines four signals that together tell you whether a system is healthy. They apply equally well to a simple government web form as to a complex distributed microservice architecture.

Figure 1 (placeholder) — A simple monitoring architecture showing a web application emitting metrics to a collection layer (Prometheus scraping metrics endpoint, or CloudWatch agent pushing metrics), storing data in a time-series database, feeding a dashboard (Grafana or CloudWatch Dashboards), and routing alerts through an alerting engine (Alertmanager or CloudWatch Alarms) to a notification target (PagerDuty, email, or Slack). Show data flow arrows between each component and label every box.

Latency

Latency is how long it takes your service to respond to a request. Slow responses harm users directly. They also signal deeper problems — a slow database query, an overloaded server, or a dependency that is struggling under load.

Measure latency as a distribution, not just an average. The p95 value (the response time that 95% of requests fall under) tells you more than the mean, because averages hide the worst experiences. Track both successful request latency and failed request latency separately — a service that fails instantly is a different problem from one that fails after a 30-second timeout.

Government example: a veterans benefits portal that takes 8 seconds to load a form is effectively broken for users on slower rural internet connections. Setting a p95 alert at 3 seconds tells you before a constituent complains or gives up.

Traffic

Traffic is the volume of requests your service is handling — requests per second, page views per minute, jobs processed per hour. It is the baseline context for every other signal. An error rate of 5 errors per minute means something very different when your traffic is 50 requests per minute versus 50,000.

Track traffic over time to understand your normal patterns. Most government services have predictable traffic cycles tied to business calendars — fiscal year deadlines, application windows, benefit payment schedules.

Government example: a tax filing portal sees 10x normal traffic in the week before the filing deadline. If you know your baseline, you can plan for the spike, scale ahead of it, and know whether a traffic drop means a network outage or just that the deadline passed.

Errors

Errors are requests that fail. This includes HTTP 5xx responses, failed background jobs, database connection failures, third-party API call failures, and any other category of “this operation did not complete successfully.” Express error rate as a percentage of total requests so it scales with traffic.

Track errors by type and by endpoint. A 2% aggregate error rate is actionable only if you know which endpoint is failing and what kind of error it is producing.

Government example: a 2% error rate on a service processing 50,000 permit applications per day means 1,000 failed applications, silently. Applicants may not know their submission failed. Set alerts on error rate so that number never creeps up unnoticed.

Saturation

Saturation is how close to full your system is. This includes CPU percentage, memory usage, disk space, database connection pool utilization, message queue depth, and thread pool exhaustion. Saturation is a leading indicator — it predicts future failures before they happen.

Government example: a database connection pool running at 90% capacity is likely to drop connections during normal load spikes. Tracking saturation lets your team act before users are affected, not after. Set your alert threshold at 80%, not 100%. At 100%, you have already failed.

What to Monitor First

When starting from nothing, prioritize strictly in this order. Do not try to monitor everything at once — you will drown in noise before you learn what normal looks like.

Uptime check — Is the service reachable at all? An external HTTP health check every 60 seconds is your most basic safety net. Configure it to check a URL that exercises a real dependency (not just a static file) and alert immediately when it fails.
HTTP 5xx error rate — Are users hitting server errors? Alert when the 5xx rate exceeds 1% of requests over a 5-minute rolling window.
Response time (p95) — Is the service slow? Agree on an acceptable baseline for your use case (2 seconds is a common starting point for web UIs) and alert when you sustain above it for 5 minutes.
Server CPU and memory — Are resources being exhausted? Alert at 80% sustained usage over 5 consecutive minutes.
External dependency health — Does your app call another API, an identity provider, a payment processor, or an agency mainframe? Watch for failures on those calls, tracked separately from your own error rate.
Database health — Connection pool utilization, slow queries (anything over 5 seconds), and replication lag if you run read replicas.

Once you have run these for two weeks and calibrated the thresholds against real traffic patterns, you are ready to add more detail.

Types of Monitors

Uptime Checks

An external service pings your URL on a schedule and alerts you when it gets no response or an error response. Start here. Every government web service needs at least one uptime check configured before it accepts production traffic.

Synthetic Tests

Synthetic tests simulate a real user completing a workflow — logging in, submitting a form, completing a payment, downloading a document. They run on a fixed schedule and alert you when the workflow breaks, even before any real user has tried it. Synthetic tests are especially useful for critical government flows where a failure has legal or regulatory consequences.

Real-User Monitoring

Real-user monitoring (RUM) collects performance data from actual users’ browsers, reporting what real users experience — including users on slow devices or slower network connections. Important caveat: RUM typically requires a JavaScript snippet that sends browser telemetry to a third-party service. Review your data classification and PII posture before enabling it on any system that processes sensitive data — see the Government-specific considerations section below.

Log-Based Alerts

Log-based alerts trigger when a pattern appears in your logs — for example, when the phrase “database connection refused” appears more than three times in 60 seconds. These are flexible but require a working log pipeline and careful pattern design to avoid false positives.

Metric-Based Alerts

Metric-based alerts trigger when a numeric threshold is crossed — CPU above 80%, response time above 2 seconds, error rate above 1%. These are simpler to set up and less prone to false positives from log formatting changes. Start with metric-based alerts.

What a Useful Alert Looks Like

Alert fatigue is the condition where a team receives so many alerts, or so many alerts that do not require action, that they start ignoring all alerts. This is the most common failure mode in monitoring programs. Teams tune out noise and miss the real fire.

A useful alert answers three questions without requiring the on-call engineer to look anything up: What is broken? What is affected? What should I do first?

Every alert should have a runbook link — even a three-bullet runbook is better than nothing. A runbook explains what the alert means, what to check first, and who to escalate to if the first check does not resolve it.

Category	Bad alert example	Good alert example
Title	`High CPU`	`Permit portal API — CPU sustained above 80% for 5 min (prod)`
Context provided	None	Dashboard link, recent deploy list, runbook link
Severity calibration	Every alert is CRITICAL	Error rate → CRITICAL; saturation warning → WARNING
Threshold design	Alert fires whenever CPU > 50%	Alert fires only when sustained > 80% for 5 consecutive minutes
Actionability	`Check the server`	`Step 1: Check /metrics dashboard. Step 2: Identify which process is consuming CPU. Runbook: [link]`
After-hours paging	Wakes someone at 3am for 0.5% error rate	Pages on-call only for alerts that require immediate human action to prevent data loss or outage
Re-alerting behavior	Fires every 2 minutes while the condition holds	Fires once, then again only if it worsens or has not resolved after 30 minutes

Review your alert configuration every sprint. If an alert fires and the on-call engineer always marks it as a false positive, either fix the threshold or delete the alert. A deleted unhelpful alert is better than one that trains your team to ignore the channel.

Government-Specific Considerations

PII in Logs and Metrics

Government services frequently process personally identifiable information — names, Social Security Numbers, dates of birth, addresses, case numbers, medical information, immigration status. This data must not flow into your monitoring tools unless those tools are specifically approved to handle it at the appropriate sensitivity level.

Review your application’s log output for PII before enabling any monitoring pipeline. Common sources of accidental PII in telemetry include:

Full request URLs containing user identifiers, case numbers, or session tokens as query parameters
Error messages that echo back user-submitted form values
Database query logs that include literal WHERE clause values
Stack traces that include in-memory object representations with user data

Use log scrubbing middleware to redact sensitive fields before logs leave your application process. Define a list of field names that must always be redacted (ssn, dob, address, case_id if it maps to a person) and apply that list consistently across all services.

What Telemetry Is Safe to Send to SaaS Vendors

Before sending any telemetry data to a SaaS monitoring vendor, your agency security team must review:

What data categories are being transmitted (pure numeric metrics vs. log content vs. traces with request context)
Whether any PII or system-sensitive data is included in that telemetry
Whether the vendor holds a FedRAMP authorization at the appropriate impact level for your system

Pure numeric metrics — CPU percentage, request count, error count — carry lower risk than log content or traces, which may include user context. If you are unsure, send only numeric aggregates to cloud vendors and keep raw log content on-premises or in a FedRAMP-authorized log store.

Resource: FedRAMP Marketplace — search by product name to find current authorization status.

FedRAMP Considerations for Monitoring Tools

FedRAMP authorization does not automatically mean a tool is approved for your specific use case. Before selecting a monitoring tool, confirm:

The tool’s authorization level matches your system’s impact level (Low, Moderate, or High)
You have reviewed the tool’s Customer Responsibility Matrix (CRM) and your team can meet all customer-side controls
The tool is listed in your system’s ATO documentation as an approved external service or inheritance chain

For agencies operating in AWS GovCloud, AWS CloudWatch in GovCloud is the lowest-friction path. For Azure Government tenants, Azure Monitor is the equivalent. Both are well-understood in federal ATO packages.

Records Retention for Log Data

Federal records retention requirements may apply to your log data. Raw logs that capture user activity on a government system can be considered federal records under 44 U.S.C. Chapter 31 and NARA guidance.

Many teams configure log retention of 30 days for cost reasons without realizing they may be legally required to retain records for 3, 7, or even longer based on the program. Work with your agency records officer to determine the correct retention schedule before you configure any automatic deletion policy.

Resources

Tool Options by Maturity Level

Maturity level	Tool options	Notes
Starter — free tier, self-hosted	Prometheus + Grafana + Alertmanager	Maximum control, no vendor risk. Requires operational effort to run and maintain. Good for learning.
Starter — managed	Uptime Robot (uptime checks only), Grafana Cloud free tier	Fast to set up. Check FedRAMP status before sending log content.
Intermediate	Datadog, New Relic, Dynatrace	Powerful, easy to set up, broad ecosystem. Significant cost at scale. Not all have FedRAMP authorization.
FedRAMP Moderate	AWS CloudWatch (GovCloud), Azure Monitor (Azure Government), Splunk Cloud (FedRAMP Moderate authorized)	Required if your system is at Moderate or High impact level and sends log content or traces to the vendor.
Open-source full stack	Prometheus + Grafana + Loki + Tempo + Alertmanager	Maximum flexibility and no vendor lock-in. High operational burden. Best suited to mature platform teams.

30 / 60 / 90 Day Rollout Plan

Timeframe	Goal	Actions
Week 1	Know when you are down	Configure uptime checks for every public endpoint. Confirm alerts reach a real person (not a shared inbox). Verify the alert fires by temporarily taking the service down in a non-prod environment.
Week 2	Know when users hit errors	Add HTTP 5xx rate monitoring. Set initial alert threshold at 1% over a 5-minute window. Write a first runbook for the 5xx alert.
Week 3–4	Understand your baseline	Run the uptime and error rate monitors without tuning thresholds. Observe what normal looks like. Document your typical traffic patterns and error floor.
Days 30–45	Know when you are slow	Add p95 response time monitoring. Use your baseline data to set a realistic threshold.
Days 45–60	Know when you are close to full	Add server CPU, memory, and database connection pool monitoring. Alert at 80% sustained saturation.
Days 61–75	Watch your dependencies	Add monitoring for every external service your app calls. Alert when upstream error rates exceed your baseline.
Days 75–90	Build a shared dashboard	Create one dashboard showing all four golden signals. Share with the full team. Make it the first thing the on-call engineer opens.
Day 90+	Add synthetic tests	Instrument your most critical user flows with synthetic checks. Review and tune all alert thresholds based on 90 days of real data.

Next Steps

Observability 101 — Once basic monitoring is in place, this guide covers structured logging, distributed tracing, and debugging problems you did not predict.
On-Call Basics — How to set up a sustainable on-call rotation, write effective runbooks, and run post-incident reviews without burning out the team.
NIST SP 800-137 — The federal standard for information security continuous monitoring programs. Required reading for teams building toward an ATO.
Google SRE Book — Chapter 6 — The original source for the four golden signals, free online.
CISA Zero Trust Maturity Model — Monitoring and telemetry visibility are core pillars of the zero trust posture required for modern federal systems.