Observability — Prometheus + OpenTelemetry

Field	Value
Type	Agent Reference
Source	~/.copilot/agents/_refs/go-backend-engineer/observability.md
Description	Not specified

Source Content

Observability — Prometheus + OpenTelemetry

Three signals, three pipelines:

• Metrics — Prometheus client → /metrics endpoint → scraped → Grafana
• Traces — OTel SDK → OTLP/gRPC → Tempo
• Logs — Zap JSON → stdout → Promtail → Loki (correlated via trace_id)

Prometheus metrics (internal/telemetry/metrics.go)

package telemetry

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

type Metrics struct {
    HTTPRequestsTotal   *prometheus.CounterVec
    HTTPRequestDuration *prometheus.HistogramVec
    HTTPRequestSize     *prometheus.HistogramVec
    HTTPResponseSize    *prometheus.HistogramVec
    DBQueryDuration     *prometheus.HistogramVec
    ActiveConnections   prometheus.Gauge
}

var metrics *Metrics

func InitMetrics(serviceName string) *Metrics {
    metrics = &Metrics{
        HTTPRequestsTotal: promauto.NewCounterVec(prometheus.CounterOpts{
            Name: serviceName + "_http_requests_total",
            Help: "Total number of HTTP requests",
        }, []string{"method", "path", "status"}),
        HTTPRequestDuration: promauto.NewHistogramVec(prometheus.HistogramOpts{
            Name:    serviceName + "_http_request_duration_seconds",
            Help:    "HTTP request duration in seconds",
            Buckets: prometheus.DefBuckets,
        }, []string{"method", "path"}),
        HTTPRequestSize: promauto.NewHistogramVec(prometheus.HistogramOpts{
            Name:    serviceName + "_http_request_size_bytes",
            Buckets: prometheus.ExponentialBuckets(100, 10, 8),
        }, []string{"method", "path"}),
        HTTPResponseSize: promauto.NewHistogramVec(prometheus.HistogramOpts{
            Name:    serviceName + "_http_response_size_bytes",
            Buckets: prometheus.ExponentialBuckets(100, 10, 8),
        }, []string{"method", "path"}),
        DBQueryDuration: promauto.NewHistogramVec(prometheus.HistogramOpts{
            Name:    serviceName + "_db_query_duration_seconds",
            Buckets: prometheus.DefBuckets,
        }, []string{"operation", "table"}),
        ActiveConnections: promauto.NewGauge(prometheus.GaugeOpts{
            Name: serviceName + "_active_connections",
        }),
    }
    return metrics
}

func GetMetrics() *Metrics { return metrics }

Cardinality rules

• path label must be the route template (/users/:id), never the raw URL — otherwise IDs explode cardinality.
• status = full code (200, 404); never bucket into 2xx.
• Avoid labels on user-supplied data (tenant slug, email, search query). Use exemplars/traces for that.
• Targeted limit: < 100 unique series per metric in steady state.

Fiber logging+metrics middleware

internal/api/middleware/logging.go (excerpt):

func Logging() fiber.Handler {
    return func(c *fiber.Ctx) error {
        start := time.Now()
        requestID := c.Get("X-Request-ID")
        if requestID == "" {
            requestID = uuid.New().String()
            c.Set("X-Request-ID", requestID)
        }
        ctx, span := telemetry.StartSpan(c.UserContext(), c.Method()+" "+c.Path(),
            attribute.String("http.method", c.Method()),
            attribute.String("http.url", c.OriginalURL()),
            attribute.String("request.id", requestID),
        )
        defer span.End()

        ctx = telemetry.WithRequestContext(ctx, requestID, requestID, c.Cookies("session_id", ""), nil, nil, nil)
        c.SetUserContext(ctx)
        logger := telemetry.FromContext(ctx)

        m := telemetry.GetMetrics()
        m.ActiveConnections.Inc()
        defer m.ActiveConnections.Dec()

        err := c.Next()
        dur := time.Since(start)
        status := c.Response().StatusCode()
        m.HTTPRequestsTotal.WithLabelValues(c.Method(), c.Route().Path, strconv.Itoa(status)).Inc()
        m.HTTPRequestDuration.WithLabelValues(c.Method(), c.Route().Path).Observe(dur.Seconds())

        span.SetAttributes(attribute.Int("http.status_code", status))
        if err != nil {
            telemetry.RecordError(ctx, err)
            logger.Error("request_completed", zap.Int("status", status), zap.Duration("duration", dur), zap.Error(err))
        } else {
            logger.Info("request_completed", zap.Int("status", status), zap.Duration("duration", dur))
        }
        return err
    }
}

Endpoints

Path	Purpose	Port
/metrics	Prometheus scrape	main API port
/healthz	Liveness	main API port
/readyz	Readiness (DB ping, etc.)	main API port
/debug/pprof/*	Go runtime profiling	separate internal port

pprof on a separate, internal-only port avoids accidentally exposing it to ingress.

SLOs (suggested defaults)

SLI	Objective
Availability (5xx_rate)	99.9% over 30d
Latency p99	< 500ms over 30d
DB query p99	< 100ms over 30d

Generate Prometheus alerting rules from SLO YAML using Sloth. Alert on error budget burn rate, not raw thresholds.

Trace sampling

• Dev/staging: AlwaysSample (visibility > cost)
• Production: parent-based + ratio (e.g. 0.05) + tail sampling for errors via OTel collector