Rollback Playbook Template

Field	Value
Type	Agent Reference
Source	`~/.copilot/agents/_refs/platform-sre-kubernetes/rollback.md`
Description	Not specified

Source Content

Rollback Playbook Template

Every change ships with a rollback plan. If you can’t write one in under five minutes, the change is too big — split it.

Trigger criteria — when to roll back, not investigate

Roll back immediately (no hand-wringing) if any of these are true within 30 minutes of rollout:

HTTP 5xx rate > 2× the 24h baseline for > 5 minutes.
p99 latency > 1.5× baseline for > 5 minutes.
Pod restart loop on > 25% of replicas.
A user-facing path returning the wrong data (correctness regressions don’t get a “let’s wait and see”).
An on-call engineer says “I don’t understand what’s happening” — that’s a rollback signal, not a debugging signal.

Investigate-first is acceptable only for: increased non-error latency at low percentiles, log-volume changes, dashboards looking weird without an SLO breach.

The rollback options

A. ArgoCD revert (preferred)

# 1. Find the last known-good revision
argocd app history <app>

# 2. Roll back to that revision
argocd app rollback <app> <revision-id>

# 3. Verify
argocd app wait <app> --health --timeout 300
kubectl -n <ns> rollout status deploy/<svc>

Records the revert as a sync event. The Git state is unchanged — open a PR to revert the actual commits afterwards so the next sync doesn’t re-deploy the bad version.

B. Image tag revert via PR

If the change was a values-file bump only:

git revert <commit-sha>
git push origin main      # ArgoCD auto-syncs in non-prod
# For prod: open the revert PR, get one approval, merge.

Prefer this when the team needs a paper trail (most production changes).

C. Imperative rollout undo (last resort)

kubectl -n <ns> rollout undo deploy/<svc>
kubectl -n <ns> rollout undo deploy/<svc> --to-revision=<n>   # if specific revision needed

Drift from Git — ArgoCD will try to re-sync. Use only if A and B aren’t fast enough (e.g. live SEV1 with bleeding users) and immediately follow with a Git revert PR to reconcile.

Verification after rollback

Re-run the post-deploy verification suite from validation-and-rollout.md:

Error rate back to baseline within 5 minutes.
Latency back to baseline within 5 minutes.
No pod restarts since rollback completion.
Smoke-test Job (PostSync hook) passes.

If any signal doesn’t return to baseline after rollback, the problem is upstream (a dependency, the cluster itself, a misconfigured ArgoCD project) — escalate to the incident commander, don’t chase it alone.

Special cases

Database migrations — not all migrations are reversible. The app must remain compatible with the previous schema for one release (expand-then-contract). If you can’t roll back the app because the migration broke compatibility, you wrote a backwards-incompatible migration; that’s a separate, larger problem (see database-designer skill).
Stateful workloads (CloudNativePG, MinIO) — image revert is safe; data revert is not. Restore from pgBackRest / Velero only with the data-owning team in the loop.
CRDs — never roll back a CRD schema change while CRs of the new shape exist; conversion webhooks fail. Drain to the old shape first.

Rollback section template (paste into PR description)

## Rollback
- **Trigger**: 5xx rate > 2× baseline for > 5min, OR p99 > 1.5× baseline, OR pod restart loop > 25% replicas.
- **Action**: `argocd app rollback <app> <prev-rev>` (revision: `abc1234`).
- **Verification**: error rate + p99 latency back to baseline within 5min; smoke job passes.
- **Reverse-migration**: none required / migration is expand-only and safe to leave applied.
- **Owner during window**: @<on-call>.