Skip to content
Documentation

Observability: SLOs, Alerts, Dashboards

FieldValue
TypeSkill Resource
Source~/.copilot/skills/platform/references/observability.md
DescriptionNot specified

Source Content

Observability: SLOs, Alerts, Dashboards

Design observability that answers questions you didn’t know you’d ask. SLOs anchor the alerts; alerts hand off to runbooks; dashboards tell the story without 47 panels of noise.

Contents

Use me for / Don’t use me for

Use for: defining SLIs/SLOs and burn-rate alerts, rebuilding a noisy alerting setup, designing a Grafana dashboard set, instrumenting metrics/logs/traces with OpenTelemetry, auditing observability spend (cardinality, retention, sampling).

Don’t use for: running the live incident (references/incident-command.md), building the operator that emits the metrics (references/kubernetes-operator.md), CI-pipeline observability (references/github-actions.md), threat detection / SIEM rules (the security skill).

“Production is down” → defer to references/incident-command.md; the SLO and burn-rate output is what sets severity there. “Add an events metric label per user_id” → refuse on cardinality grounds; propose a sampled high-cardinality event store instead.

Naming the user journeys

Pick 3–5 critical flows. SLOs attach to journeys, not to CPUs. Start simple — the first SLOs won’t be perfect, and that’s fine; iterate on what you learn.

Choosing SLIs

The four golden signals: latency, traffic, errors, saturation. A good SLI is measurable, meaningful to users, controllable, and proportional — if the SLI changes, users notice.

Don’t use CPU/memory/disk as a primary SLI (symptom, not user impact); don’t use raw counts instead of rates (“error rate,” not “number of errors”); don’t use internal metrics users never feel (queue depth, cache hit rate) unless they provably gate user experience.

Setting SLO targets and error budgets

Error Budget = (1 - SLO) × Time Window

A 99.9% availability SLO over 30 days gives (1 - 0.999) × 30 days = 43.8 minutes of budget.

Service criticalityAvailabilityLatency P95Error rate
Critical (revenue)99.95–99.99%100–200ms< 0.1%
High priority99.9–99.95%200–500ms< 0.5%
Standard99.5–99.9%500ms–1s< 1%

Error budget policy example (balanced): >75% consumed → increase reliability focus; >90% consumed → pause feature work.

Burn-rate alerting

Multi-window (fast + slow) burn-rate alerts catch both outages and slow drift, and page only on user-visible symptoms — never a raw threshold.

# Fast burn: 2% of monthly budget in 1 hour
- alert: ErrorBudgetFastBurn
  expr: (
    error_rate_5m > (14.4 * error_budget_slo)
    and
    error_rate_1h > (14.4 * error_budget_slo)
  )
  for: 2m
  labels: {severity: critical}


# Slow burn: 10% of monthly budget in 3 days
- alert: ErrorBudgetSlowBurn
  expr: (
    error_rate_6h > (1.0 * error_budget_slo)
    and
    error_rate_3d > (1.0 * error_budget_slo)
  )
  for: 15m
  labels: {severity: warning}

Alert design patterns

  • Symptoms, not causes. Alert on user-visible latency/error rate, not raw CPU.
  • Every alert is actionable, novel, and user-visible impact — or it shouldn’t page.
  • Hysteresis. Different fire/resolve thresholds prevent flapping.
  • Grouping and suppression. group_wait: 30s, group_interval: 2m, repeat_interval: 1h; inhibit downstream alerts when the upstream cause is already firing.
  • Every alert carries a runbook link and dashboard/panel reference in its annotations — no orphan alerts.

Dashboard pyramid

Overview (golden signals) → service (RED + USE) → instance (debug). Max 7±2 panels per screen. Consistent color semantics: green = healthy, yellow = degraded, red = critical, blue = informational.

DashboardAudienceRefresh
OverviewCross-team / exec5–15 min
SRE operationalOn-call15–30 s
Developer debugDev teams30 s–2 min

Use recording rules for expensive queries rather than raw PromQL in every panel; keep resolution reasonable (don’t chart 3600 raw points where 240 tells the same story).

Tracing and cardinality

Wire OpenTelemetry context propagation across every service boundary — a trace that stops propagating is just expensive logging. Sample heads, retain tails on errors. Run a deliberate cost pass: drop labels that explode cardinality (never a user ID as a label), tier metric retention, sample logs.

Validate

scripts/lint_slo_doc.sh [slo-doc.md|slo-doc.yaml] confirms every SLO section has an explicit SLI + numeric target + error-budget window, and every alert mention has a nearby runbook link and dashboard/panel mention. Pure Python stdlib regex/line scanning — no required external tool; yamllint runs as a bonus check on .yaml/.yml targets if installed.

Generative helpers (optional)

Three Python scripts generate starting-point artifacts from a service description — not linters, run them by hand when scaffolding a new service’s observability:

  • scripts/slo_designer.py --service-type api --criticality high --user-facing true — generates a full SLI/SLO/error-budget/burn-rate-alert framework as JSON.
  • scripts/dashboard_generator.py --service-type api --name "Payment Service" — generates a role-optimized (sre/developer/executive/ops) dashboard spec, optionally as Grafana-importable JSON (--format grafana).
  • scripts/alert_optimizer.py --input alerts.json --analyze-only — audits an existing alert config for noise, coverage gaps, duplicates, and alert-fatigue risk.

Self-rubric

  • Every alert maps to an SLO or a runbook entry; no orphan alerts.
  • Burn-rate alerts (multi-window) used instead of raw thresholds.
  • Dashboards tell a story top-to-bottom; no 30-panel walls.
  • Cardinality budget considered; no user IDs as labels.
  • Traces propagate across every service boundary in scope.
  • Runbook link on every alert.
  • scripts/lint_slo_doc.sh exits 0.

References