Performance

Field	Value
Type	Skill Resource
Source	~/.copilot/skills/backend/references/performance.md
Description	Not specified

Source Content

Performance

Performance work is profile-first, one variable at a time, and measured — never a guess-and-change loop. Optimizing code without a profile pointing at it is superstition; it burns engineering time and frequently makes the wrong line faster while the real bottleneck sits untouched.

The workflow

1. Establish a baseline. go test -bench=. -benchmem against the package under suspicion, before touching any code.
2. Profile to find the actual hot path. -cpuprofile/-memprofile plus go tool pprof, not intuition about where the time “probably” goes.
3. Change exactly one thing. A single allocation removed, a single algorithm swapped, a single buffer reused — never a bundle of changes measured together.
4. Re-measure with benchstat, comparing the before/after .txt output. A change with no statistically significant delta is not a win, no matter how it reads.
5. Paste the benchstat delta into the commit or PR body as evidence. “Faster” is not a claim this skill accepts without the numbers attached.

# 1. baseline
go test -bench=BenchmarkInvoiceService_Finalize -benchmem -count=10 ./internal/service/... > before.txt

# 2. profile the hot path
go test -bench=BenchmarkInvoiceService_Finalize -cpuprofile=cpu.prof -memprofile=mem.prof ./internal/service/...
go tool pprof cpu.prof

# 3. make exactly one change, then re-run the same benchmark
go test -bench=BenchmarkInvoiceService_Finalize -benchmem -count=10 ./internal/service/... > after.txt

# 4. compare
benchstat before.txt after.txt

references/observability.md already wires a pprof endpoint on a gated internal port for live production debugging — this workflow profiles the same binary’s hot path locally or in CI, before the code ships, using go tool pprof against a benchmark run rather than a running service.

GOMEMLIMIT and GOMAXPROCS in a container

Set GOMEMLIMIT to 80-90% of the container’s memory limit so the Go garbage collector proactively collects before the kernel’s cgroup OOM killer ends the pod. This matters directly for a service deployed to k8s — a Go process that only knows its own heap size, not the container’s ceiling, will happily grow until it gets killed.

# k8s Deployment
resources:
  limits:
    memory: 2Gi
env:
  - name: GOMEMLIMIT
    value: "1800MiB" # ~90% of the 2Gi container limit

Go 1.25+ reads cgroup v2 CPU quotas automatically and sets GOMAXPROCS to match, so a container limited to 2 CPUs correctly gets GOMAXPROCS=2 instead of the host’s full core count. Clusters still running cgroup v1 do not get this for free — import go.uber.org/automaxprocs for its side effect:

import _ "go.uber.org/automaxprocs" // cgroup v1: sets GOMAXPROCS from the container's CPU quota at init

Without one of these, a pod on cgroup v1 sees the node’s full CPU count, over-schedules goroutines relative to its actual quota, and gets throttled by the kernel instead of scheduling correctly in the first place.

CI benchmark gates compare to a baseline, never an absolute threshold

Cloud CI runners have 5-10% run-to-run noise from shared hardware, noisy neighbors, and thermal variance. An absolute gate — “fail if over 200µs/op” — either never trips because the number drifts under it by luck, or flakes constantly because normal noise crosses the line. Compare the PR branch to the base branch instead, and fail only on a statistically significant regression:

.github/workflows/benchmark.yml

- name: Benchmark base branch
  run: |
    git fetch origin main
    git worktree add /tmp/base origin/main
    (cd /tmp/base && go test -bench=. -benchmem -count=10 ./internal/service/...) > base.txt

- name: Benchmark PR branch
  run: go test -bench=. -benchmem -count=10 ./internal/service/... > pr.txt

- name: Compare with benchstat
  run: |
    go install golang.org/x/perf/cmd/benchstat@latest
    benchstat base.txt pr.txt | tee benchstat.txt
    if grep -qE '~\s*\(p=' benchstat.txt; then
      echo "no significant change"
    elif grep -qE '\+[0-9]' benchstat.txt; then
      echo "::error::statistically significant regression detected — see benchstat.txt"
      exit 1
    fi

benchstat marks a delta with ~ when it is not statistically significant given the sample’s variance — that is the noise floor being filtered out, not a comparison being skipped.

Where to profile a running service

This file covers the benchmark-driven workflow during development: baseline, profile, one change, re-measure, evidence in the PR. For profiling a live incident against a service already running in production, see references/observability.md’s pprof section — same tool, same profile types, but reached through the gated internal port instead of a local go test run.