Observability 101 — Beyond Monitoring

This guide explains the three pillars of observability — metrics, logs, and traces — and how to use them together to debug problems that basic monitoring cannot explain. It also covers structured logging, correlation IDs, and what government teams must do before sending observability data to any external tool. Engineers who already have basic monitoring in place, or who have read Monitoring 101, and want to build the deeper visibility needed to debug complex or hard-to-reproduce problems will find it most useful.

TL;DR — Observability Quick Reference

Concept	One-sentence summary
Monitoring	Tells you when something is broken
Observability	Helps you understand why something is broken
Metrics	Numbers that change over time (CPU %, error count, response time)
Logs	Structured records of discrete events that happened inside your system
Traces	Records of a single request’s journey across multiple services
Structured logging	JSON log output that machines can query and filter, not just humans can read
Correlation ID	A unique ID attached to a request that flows through every service it touches
OpenTelemetry	The open standard for collecting and exporting all three pillars

Monitoring vs. Observability

Monitoring and observability are complementary but distinct. Monitoring tells you when something breaks; observability helps you understand why it broke.

Monitoring

Monitoring watches for known problems. You define a threshold — error rate above 1%, CPU above 80% — and an alert fires when that threshold is crossed. This works well for known failure modes: a server going down, a disk filling up, a third-party API returning errors.

Observability

Observability is the property of a system that lets you understand its internal state by examining its outputs—without needing to know in advance what questions you will ask. It is designed for unknown unknowns: the bug that only happens for a specific combination of user data and request timing, the performance regression that only affects one tenant, the cascading failure that started with a timeout in a service you did not think was in the critical path.

In Practice

A simple analogy: monitoring is like checking the gauges on your car’s dashboard. If the oil light comes on, you know the oil is low. Observability is like having a mechanic who can diagnose the engine by examining any signal from the car—not just the pre-wired warning lights—including signals that the car manufacturer never anticipated would matter.

Government systems are increasingly distributed: many agencies now run services that call identity providers, shared platforms, vendor APIs, and legacy mainframes as part of a single user transaction. In that environment, basic monitoring tells you something went wrong. Observability tells you which of those eight services caused the problem and why.

External Resources

The Three Pillars

Figure 1 (placeholder) — The three observability pillars shown as three parallel vertical lanes, all flowing into a unified observability platform at the bottom. Lane 1 (Metrics): time-series data points (CPU %, request rate, error count) flowing from application and infrastructure into a metrics store labeled ‘Prometheus / CloudWatch Metrics’. Lane 2 (Logs): structured JSON log events flowing from services into a log aggregator labeled ‘Loki / CloudWatch Logs / Splunk’. Lane 3 (Traces): request spans (start time, duration, service name, trace ID, parent span ID) flowing from each service into a trace store labeled ‘Jaeger / AWS X-Ray / Tempo’. At the bottom, all three lanes converge into a unified observability platform box (Grafana / Honeycomb) where an engineer performs correlation and root-cause analysis.

Metrics

Metrics are numbers that change over time. They answer questions like: How many requests per second is this service handling right now? What percentage of those requests returned an error? How much memory is the process using?

Metrics are efficient — you can store millions of data points over time without massive storage costs — and they are fast to query. They are ideal for dashboards and alerts. The tradeoff is that metrics lose detail. A metric tells you the error rate went up; it does not tell you which specific request failed or what error message it produced.

Common government metric examples

http_requests_total{method="POST", status="500", endpoint="/api/applications"} — count of failed application submissions
db_query_duration_seconds{query="fetch_case_status"} — time spent on a specific database query type
external_api_calls_total{vendor="login_gov", outcome="success"} — count of successful identity verification calls

Logs

Logs are structured records of discrete events that happened inside your system. A log event says: at this exact timestamp, this specific thing occurred, with these specific details.

Logs provide the rich context that metrics cannot. When you know from your metrics that the error rate spiked at 2:14pm, you go to your logs to find out what specific errors occurred, which users were affected, and what request data was involved. Logs are the evidence; metrics are the alarm. See the Structured Logging section below for what that means and why it matters in practice.

Traces

Traces are records of a single request’s complete journey through your system. In a monolith — a single application handling everything — a trace is straightforward: request in, response out, duration measured. In a distributed system where a single user action triggers calls to five different services, a trace connects all of those calls into one picture.

A trace is made up of spans. Each span represents one unit of work: processing a request in your API, executing a database query, calling an external service. Each span records its start time, duration, any errors, and contextual metadata. The spans are linked together using a trace ID — a unique identifier that flows through every service call triggered by the original request.

When a request fails in a multi-service architecture, the trace shows you exactly which span failed, which service it was in, how long each step took, and what the call chain looked like. Without traces, you are correlating logs across multiple services by hand, matching timestamps and hoping they align.

When to Use Observability

Basic monitoring is sufficient for simple, single-service systems where failures are predictable and straightforward to diagnose. You add metrics and uptime checks and you are done.

Observability becomes essential in these situations:

Distributed systems and microservices. When a user request touches four services and one of them misbehaves, monitoring tells you the user got an error. Only traces tell you which service caused it.
Hard-to-reproduce bugs. When a bug only manifests for a specific user, only on Tuesdays, only when the database is under load — rich logs and traces let you reconstruct the conditions that led to the failure without being able to trigger it intentionally.
Performance regressions that affect only some users. Aggregate metrics smooth out individual experiences. Observability lets you answer: are my p95 response times slow because of all users, or because of a specific tenant, endpoint, or geographic region?
Post-incident investigation. Regulatory and contractual obligations in government often require detailed incident reports. Observability data is the evidence base for those reports. “We don’t know what happened” is not an acceptable root cause analysis in a federal environment.
Security and audit events. FISMA, FedRAMP, and many agency-specific regulations require detailed audit trails of who accessed what and when. Structured logs and traces are the technical foundation of those audit trails.

Structured Logging

Structured logging means writing log output as key-value pairs that machines can parse, filter, and query — rather than freeform text that only humans can read.

The standard format is JSON. Every field that matters has a defined key name. Every event has a consistent schema.

Unstructured log (hard to use)

[2026-05-28 14:22:31] ERROR Failed to process application for user john.doe@example.gov after 3 retries. DB connection timed out.

To search this log, you must use text search. You cannot easily extract all errors by user, count errors by type, or filter by time range without writing fragile regex patterns.

Structured log (queryable)

{
  "timestamp": "2026-05-28T14:22:31.042Z",
  "level": "error",
  "message": "Application processing failed",
  "service": "application-service",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "user_id": "redacted",
  "retry_count": 3,
  "error_type": "DatabaseConnectionTimeout",
  "duration_ms": 30041,
  "environment": "production"
}

Notice that user_id is redacted. In government systems, PII must be removed or replaced before log events leave the application process. See the Government-specific considerations section below.

With structured logs, you can query: give me all DatabaseConnectionTimeout errors in the last hour, grouped by service. You can set alerts on specific error types. You can join log events to traces using the trace_id field. None of this is practical with unstructured text logs.

Log Levels

Log levels indicate the severity and intent of a log event. Using them consistently is important — if everything is logged at ERROR, you cannot distinguish a real problem from a diagnostic message.

Level	When to use it	Example
DEBUG	Detailed diagnostic information useful during development. Should be disabled in production by default — DEBUG volume is extremely high.	`{"level":"debug","message":"Cache miss for key","key":"user_session_abc123"}`
INFO	Normal, expected system behavior. Major lifecycle events. Should be on in production at moderate volume.	`{"level":"info","message":"Application submitted successfully","application_id":"APP-2026-00441"}`
WARN	Something unexpected happened but the system recovered or continued. Should be investigated but is not an immediate incident.	`{"level":"warn","message":"Retry succeeded after 2 failures","service":"payment-gateway","retry_count":2}`
ERROR	An operation failed. A user-visible error occurred. Immediate attention is warranted.	`{"level":"error","message":"Database connection refused","error_type":"ConnectionRefused","duration_ms":30000}`
FATAL / CRITICAL	The service cannot continue. It is shutting down or is in an unrecoverable state. This should be extremely rare.	`{"level":"fatal","message":"Cannot bind to required port 8080. Exiting."}`

A common mistake is to log at ERROR for every caught exception, including expected ones like “user not found” or “session expired.” Reserve ERROR for conditions that require investigation. Expected business logic outcomes are INFO events.

Distributed Tracing

Distributed tracing works by attaching a trace ID to every incoming request and passing that ID along to every downstream service call, database query, and external API call that the request triggers.

Each service adds its own span to the trace — a record of the work it did, how long it took, whether it succeeded, and any relevant metadata. When the request completes, all the spans are collected and stored together, indexed by the shared trace ID.

To examine a failed request, you look up its trace ID and see the complete call graph: Service A called Service B, which called the database (4ms), then called the identity provider (1,200ms — there is your problem), then returned to Service A, which assembled the response and returned it to the user.

Tracing requires instrumentation — each service must be modified to extract the trace ID from incoming request headers, create a span for the work it does, inject the trace ID into outgoing request headers, and report the completed span to a trace collector. OpenTelemetry (see below) handles most of this automatically through auto-instrumentation libraries.

Correlation IDs

A correlation ID is a unique identifier attached to a user request that flows through every service and log event associated with that request. It serves the same linking purpose as a trace ID, but is simpler and does not require distributed tracing infrastructure.

You generate a correlation ID at the edge of your system — in your load balancer or API gateway — and include it in every log event produced while processing that request. When a user reports a problem and gives you a timestamp and their user ID, you look up log events with matching correlation IDs and reconstruct exactly what happened.

Correlation IDs are the minimum viable version of distributed tracing. They work in monolithic systems, require very little infrastructure, and have a low operational cost. If you cannot implement full distributed tracing yet, start with correlation IDs.

Implementation: generate a UUID at request entry. Store it in request context, or a thread-local store. Include it in every log event: "correlation_id": "7f3d9a20-1c4b-4e3f-8f01-29cd0b1a3e87". Pass it downstream as an HTTP header (X-Correlation-ID). This lets you search your log aggregator for all events associated with any single request.

Government-Specific Considerations

PII Scrubbing Before Logging

Government services handle sensitive personal data. PII — names, Social Security Numbers, dates of birth, addresses, case numbers, immigration status, medical information — must be removed or redacted from log events before those events leave your application process.

This is not optional. Unredacted PII in logs that flow to a log aggregator — especially a cloud-based one — is a data spillage event with serious legal and regulatory consequences.

Implement PII scrubbing as middleware or a log filter that runs on every log event before it is written. Define a denylist of field names that must always be redacted: ssn, dob, full_name, address, phone_number, email (in some contexts), and any domain-specific identifiers that link to a person. Replace values with a redaction marker: "ssn": "[REDACTED]".

Be especially careful with:

Query parameters in logged URLs (/search?name=Jane+Doe&ssn=123456789)
Error messages that echo user input
Request body logging in debug mode
Database query logs that include literal values

Retention Policies for Log Data

Observability data can pile up quickly. A high-traffic service in production can generate gigabytes of log data per day. Teams often configure short retention windows (7 to 30 days) for cost reasons.

Before configuring any retention or deletion policy, confirm the applicable federal records schedule with your agency records officer. Logs that capture user activity on a government system may be federal records with mandatory retention periods. Deleting them prematurely can violate 44 U.S.C. Chapter 31 and NARA requirements.

A practical approach: separate application performance logs (which have shorter retention needs) from audit and access logs (which are more likely to be federal records) and configure different retention schedules for each category.

What You Can Send to Cloud Observability Vendors Under FedRAMP

The general rule: numeric metrics are low-risk. Log content with system context is higher risk. Log content with PII is not permitted unless the vendor is authorized at the correct impact level and your agency has reviewed the data handling agreement.

Before connecting any cloud observability tool, confirm:

The vendor has a current FedRAMP authorization at the applicable impact level — check the FedRAMP Marketplace
The data sharing agreement (DSA) or terms of service are compatible with your agency’s data handling requirements
You have verified that PII scrubbing is in place and effective before data leaves your environment

Separation of Duties for Log Access

Not every engineer should have unrestricted access to production logs. Log data in government systems can contain sensitive system configuration details, partial PII that survived scrubbing, and security-sensitive audit events.

Define access tiers for your observability platform and document them in your system security plan (SSP):

Engineers: read access to application performance data (metrics, traces, application-level logs)
Senior engineers / SREs: read access to infrastructure and security logs
Auditors: read-only access to audit log exports through a formal process
Security team: access to security event logs with full audit trail of their own access

Tool Options

Tool	Category	FedRAMP status	Notes
OpenTelemetry	SDK / standard (not a vendor)	N/A — open standard	The vendor-neutral standard for instrumenting applications. Use it to avoid lock-in. Works with any backend.
Prometheus + Grafana	Metrics + dashboards	Self-hosted (no FedRAMP authorization needed)	Widely adopted open-source stack. Requires operational effort.
Loki + Grafana	Logs + dashboards	Self-hosted	Prometheus-compatible log aggregation. Lower cost than Elasticsearch.
Jaeger	Distributed tracing	Self-hosted	CNCF-graduated tracing project. Good OpenTelemetry compatibility.
AWS X-Ray (GovCloud)	Tracing + service map	FedRAMP High (GovCloud)	Best choice for teams already in AWS GovCloud. Deep integration with Lambda, ECS, EC2.
AWS CloudWatch Logs (GovCloud)	Logs + metrics	FedRAMP High (GovCloud)	Mature, integrated with the AWS ecosystem. Cost can grow quickly at high log volumes.
Splunk Cloud (FedRAMP)	Logs + SIEM	FedRAMP Moderate authorized	Common in large federal agencies. Powerful query language (SPL). High cost.
Honeycomb	Full observability	Not FedRAMP authorized (as of 2026)	Excellent product for observability engineering. Not suitable for Moderate/High federal systems without a separate authorization path.
Datadog	Full observability	FedRAMP Moderate authorized (GovCloud)	Broad feature set, good DX. Confirm current authorization status on FedRAMP Marketplace before use.

What Good Logs Look Like vs. Bad Logs

Category	Bad log example	Good log example
Format	`ERROR user 12345 failed login`	`{"level":"error","message":"Login failed","user_id":"[REDACTED]","reason":"invalid_credentials","timestamp":"2026-05-28T14:22:31Z"}`
PII handling	Logs full SSN in URL query params	Redacts SSN before logging; logs only a masked identifier
Log level	Every log event is `ERROR`	Log levels match actual severity; INFO for normal events, ERROR for genuine failures
Context	`Database error`	`{"error_type":"ConnectionTimeout","service":"case-lookup","db_host":"db-primary","duration_ms":30041,"trace_id":"abc123"}`
Consistency	Different services use different field names for the same concept	All services use the same field names (`trace_id`, `user_id`, `service`, `duration_ms`) defined in a shared log schema
Verbosity in prod	DEBUG logging enabled in production, generating 50GB/day	DEBUG disabled in production; INFO and above only
Queryability	Log lines require regex to parse	Every field is a JSON key that a log aggregator can index and filter

OpenTelemetry SDK in 5 Minutes (Node.js)

OpenTelemetry is the open-source, vendor-neutral standard for collecting metrics, logs, and traces from your applications. Instrumenting your Node.js service with OpenTelemetry gives you traces automatically for HTTP requests, database calls, and outbound API calls — without changing your business logic code.

Install the required packages:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

Create a tracing.js file that you load before your application starts:

// tracing.js — load this file before any other application code
// Example: node --require ./tracing.js server.js

const { NodeSDK } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");

const sdk = new NodeSDK({
  // Replace with your collector endpoint — Jaeger, AWS X-Ray OTLP, or Grafana Tempo
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://localhost:4318/v1/traces",
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // Disable filesystem instrumentation — too verbose, low value for most apps
      "@opentelemetry/instrumentation-fs": { enabled: false },
    }),
  ],
  serviceName: process.env.OTEL_SERVICE_NAME || "my-government-service",
});

sdk.start();

// Graceful shutdown — flush remaining spans before process exits
process.on("SIGTERM", () => {
  sdk.shutdown().then(() => process.exit(0));
});

What this gives you automatically, with no additional code changes:

A trace for every incoming HTTP request, including method, URL, status code, and duration
Child spans for every outbound HTTP call your service makes
Child spans for database queries (if using a supported driver like pg, mysql2, or mongoose)
A trace_id that propagates through outbound calls to downstream services that also use OpenTelemetry

To add a custom span for a specific operation — for example, a complex business logic function — use the tracer API:

const { trace } = require("@opentelemetry/api");

const tracer = trace.getTracer("case-processing");

async function processApplication(applicationId) {
  return tracer.startActiveSpan("processApplication", async (span) => {
    span.setAttribute("application.id", applicationId);
    try {
      const result = await doTheWork(applicationId);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Important for government systems: do not attach PII to span attributes. Span attributes are stored in your trace backend and visible to anyone with trace access. Use opaque identifiers such as application_id and case_number rather than names, SSNs, or other personal data.

Next Steps

Monitoring 101 — If you have not yet set up basic monitoring, start there. Observability is most valuable once you have a working alerts baseline.
Architecture Diagrams 101 — Understanding your system’s architecture is prerequisite to designing an effective observability strategy. If you do not know your trust boundaries and data flows, you do not know where to instrument.
OpenTelemetry Getting Started — Official documentation with guides for Node.js, Go, Python, Java, and more.
NIST SP 800-92 — Guide to Computer Security Log Management — The federal standard for log management, covering collection, storage, analysis, and protection of log data.
CISA — Logging Made Easy — CISA’s free log collection and analysis toolset for agencies that lack a commercial SIEM.