Skip to content
Documentation

Observability

What this is and who it’s for

This page defines what good observability looks like for a production service: the signals you emit, how you alert on them, and how you tell at a glance whether a system is healthy. It’s for the engineer who already has something deployed and now needs to know whether it’s working in production — not just whether the tests passed. It covers the practice, not the tooling: emit the signals against an open standard and the platform that ingests them becomes a swappable detail.

Prerequisites

  • You have a service deployed to a real environment via a pipeline. Observability has nothing to observe without a running deployment.
  • You can add dependencies and environment configuration to the service. Instrumentation is code; it ships through the same pipeline as everything else.
  • You have a backend that ingests logs, metrics, and traces. The practice here stays portable across whichever backend you run.

The practice

Observability rests on three signal types — logs, metrics, and traces — plus the alerting layer that turns those signals into action. Emit all three from day one. A system with only logs can tell you what happened to one request but not whether the fleet is degrading; a system with only metrics can tell you the error rate is up but not why.

Logs

Logs are the timestamped record of discrete events: a request handled, a job failed, a config reloaded. They answer “what exactly happened to this one request” when you already know something is wrong. Emit them as structured JSON, not free text, so the backend can index and query fields instead of grepping strings. See Structured logging below for the field contract and the PII rule.

Metrics

Metrics are numeric measurements aggregated over time: request count, error count, latency percentiles, queue depth. They answer “is the whole system healthy right now and how does that compare to an hour ago.” They’re cheap to store and fast to query, which makes them the foundation for dashboards and alerts. Track the Golden Signals first — latency, traffic, errors, and saturation — then use RED and USE as drill-down views when you need more detail.

Traces

Traces follow a single request across every service it touches, recording the timing of each hop as a tree of spans. They answer “where did the latency go” in a system where one user action fans out across several services. Without traces, a slow request in a multi-service architecture is a guessing game across disconnected logs.

Alerting

Alerting is the layer that watches metrics and pages a human when a user-facing symptom crosses a threshold. It’s the only signal type that interrupts someone’s sleep, so it has the highest bar: every alert must be actionable. Alert on symptoms, not causes — see “SLIs, SLOs, and alerting on symptoms” below.

Structured logging

Emit every log line as a single JSON object with consistent field names. At minimum include a timestamp, a severity level, a message, a request or correlation ID, and the service name. Consistent fields let you filter level=error AND service=payments across the whole fleet instead of reading lines one at a time. Put the correlation ID on every line so you can reconstruct one request’s full path; it’s also the join key between your logs and your traces.

Never log PII. Maryland agency systems routinely handle data covered by state and federal sensitivity rules — names, addresses, SSNs, case numbers, health and benefits data — and a log aggregator is a copy of that data in a system with different access controls and a long retention window. Treat logs as a place where PII leaks become breaches. Redact or hash identifiers before they reach the logger, and review log output for sensitive fields the same way you review code. [VERIFY: confirm the agency’s specific data-classification policy and retention limits with its security or privacy officer before enabling log shipping.]

Metrics with the Golden Signals

The Golden Signals are the default metrics every user-facing service should track. They are the fastest way to tell whether the app is healthy without turning the first dashboard into a diagnosis exercise. RED and USE still matter, but they are supporting lenses rather than the headline model.

The Golden Signals

Track these four signals on every app or service that users depend on:

  • Latency — how long requests take to complete.
  • Traffic — how much demand the service is handling.
  • Errors — how many requests fail.
  • Saturation — how close constrained resources are to capacity.

Together, these answer the first operational question: is the service healthy right now, or is it drifting toward trouble. If you only have room for one top-level health dashboard, make it this set plus the SLO status.

Where RED and USE fit

RED and USE are still useful once you need to debug. RED is a request-path view: Rate, Errors, and Duration. USE is a resource view: Utilization, Saturation, and Errors. They help explain why the Golden Signals moved, but they should not replace them as the baseline.

Distributed tracing

Instrument requests to propagate a trace context across service boundaries so the backend can stitch the spans into one trace. The reason is concrete: in a multi-service request, latency and errors hide in the gaps between services, and disconnected per-service logs can’t show you those gaps. A trace shows the whole tree — which span was slow, which call failed, how the time was actually spent. Once you adopt OpenTelemetry for metrics, context propagation for traces comes from the same instrumentation.

SLIs, SLOs, and alerting on symptoms

A Service Level Indicator (SLI) is a metric that reflects user experience — request success rate, or the percentage of requests served under a latency threshold. A Service Level Objective (SLO) is the target for that indicator over a window, for example “99.5% of requests succeed over 30 days.” The SLO is what defines “bad”: without it, a metric is just a number on a chart and nobody can say whether the current value is fine or an emergency. Set an SLO for every user-facing service.

Alert on symptoms, not causes. A symptom is something the user feels — error rate above the SLO, latency past the threshold. A cause is an internal condition like high CPU or a full disk, which may or may not affect users. Alerting on every cause floods the on-call with pages for conditions the system is handling fine, and the noise trains people to ignore the pager — so the one page that mattered gets missed too. Page on the symptom; surface causes on a dashboard for diagnosis once you’re already looking.

Actionable alerts versus noise

Every alert that pages a human must demand a human action right now. If the response to an alert is “acknowledge and ignore,” it isn’t an alert — it’s a metric that should live on a dashboard. Tie alerts to SLO burn, route the rest to dashboards, and review your alert volume regularly: a steadily rising page count is the early signal of alert fatigue. The on-call who can trust the pager is the one who responds fast when it matters; protect that trust. This practice feeds directly into Incident Response, where these same signals drive detection and diagnosis.

Dashboards that answer “is it healthy” at a glance

Build one top-level dashboard per service that answers a single question in five seconds: is this healthy right now. Lead with the Golden Signals and the SLO status; put RED or USE drill-downs lower for when someone is diagnosing rather than checking. A dashboard nobody can read at a glance is a dashboard nobody reads. The same Golden Signals and SLO status are also what a progressive-delivery system uses to judge a canary healthy or not — a canary is promoted or rolled back by exactly these indicators.

OpenTelemetry as the instrumentation standard

Instrument with OpenTelemetry (OTel), the vendor-neutral CNCF standard for emitting logs, metrics, and traces. The payoff is portability: you instrument the code once against the OTel API and can point the output at any compliant backend, so changing vendors is a configuration change, not a re-instrumentation project. This matters for a state agency, where procurement cycles and contract changes are facts of life — proprietary agents that lock your telemetry to one vendor turn a routine vendor switch into a rewrite. Use the OTel SDK for your language and the OTel Collector to receive, process, and route the signals.

Reference implementation

Fork these snippets. They use OpenTelemetry and Prometheus-style configuration; adapt field names and thresholds to your service.

OpenTelemetry instrumentation in a Node service, exporting traces over OTLP to a collector:

import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";


const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "benefits-intake-api",
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT, // e.g. [Otel Collector](http://otel-collector:4318/v1/traces)
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});


sdk.start();

A structured log line with the field contract from the practice section. Note the correlation ID and the absence of PII — the applicant is identified by an opaque case ID, never a name or SSN:

{
  "timestamp": "2026-05-31T14:02:11.482Z",
  "level": "error",
  "service": "benefits-intake-api",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "message": "downstream eligibility check failed",
  "case_id": "MD-2026-0098431",
  "downstream": "eligibility-service",
  "status_code": 503
}

A Prometheus alerting rule that pages on a user-facing symptom (error-rate SLO burn) rather than a cause. It fires when more than 1% of requests fail over five minutes, sustained for ten:

groups:
  - name: benefits-intake-slo
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{service="benefits-intake-api",status=~"5.."}[5m]))
            /
          sum(rate(http_requests_total{service="benefits-intake-api"}[5m]))
            > 0.01
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Benefits Intake API error rate above 1% SLO"
          description: "5xx rate has exceeded the 99% success SLO for 10 minutes. See the service dashboard and runbook."

Common pitfalls

  • PII in the logs. If a name, SSN, address, or case detail appears in a log line, you’ve copied sensitive data into a system with weaker access controls and long retention. Redact or hash identifiers before they reach the logger and review log output for sensitive fields. For Maryland agency data this is a breach, not a bug.
  • Alerting on causes, so the on-call drowns. If your pager fires on high CPU, full disks, and every internal blip, the on-call learns to ignore it and misses the page that mattered. Alert on user-facing symptoms tied to SLOs; move cause-level metrics to a dashboard.
  • Dashboards nobody reads. If a dashboard takes more than a few seconds to answer “is this healthy,” it won’t get opened during an incident. Build one top-level health dashboard per service, Golden Signals and SLO first, details below.
  • Metrics with no SLO. If you track latency and error rate but never set a target, nobody can say whether the current value is fine or an emergency. Set an SLO for every user-facing service so “bad” is defined before the incident, not during it.
  • Vendor lock-in from proprietary agents. If your telemetry is emitted by a vendor’s proprietary agent, switching backends means re-instrumenting every service. Instrument with OpenTelemetry so a vendor change is a config change.