Validation & Rollout

Field	Value
Type	Agent Reference
Source	~/.copilot/agents/_refs/platform-sre-kubernetes/validation-and-rollout.md
Description	Not specified

Source Content

Validation & Rollout

Three phases: pre-deploy validation, controlled rollout, post-deploy verification. No phase is skipped, even for “trivial” changes.

Pre-deploy validation

Run locally and in CI before merge.

Check	Command	Pass criterion
Helm syntax	helm lint .p3/helm	exit 0, no warnings
Template render	helm template <r> .p3/helm -f .p3/helm/values/values.<env>.yaml	no errors, output matches expected diff
Schema validation	helm template ... | kubeconform -strict -summary	all resources known & valid
Server-side dry-run	kubectl apply --dry-run=server -f - (piped from template)	no admission errors
Policy check	kyverno apply <policy-dir> --resource <(helm template ...)	all policies pass
Image signature	cosign verify --certificate-identity ... <image>	signed by expected workflow
Diff vs cluster	argocd app diff <app>	only the intended drift

CI runs all of these as required checks; merge to main is blocked otherwise.

Rollout — environments in order

test  →  staging  →  prod

Promotion = a PR that bumps image.tag (or targetRevision) in the next env’s values file. Skip-promotion is a process violation; use a hotfix-<env> branch if a fix needs to land out-of-band.

Sync mode

Env	ArgoCD sync	Notes
test	auto, self-heal, prune	ephemeral, can be torn down
staging	auto, self-heal	~prod parity, soak before prod
prod	manual or auto pinned to tag	sync gated by release PR

Rollout watch

Once synced:

kubectl -n <ns> rollout status deploy/<svc> --timeout=5m
kubectl -n <ns> get pods -l app.kubernetes.io/name=<svc> -w
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -30

Healthy = all pods Ready 1/1, no CrashLoopBackOff, no OOMKilled in the last 10 minutes.

Post-deploy verification (≥ 15 minutes)

Watch four signals against the SLO baseline:

Signal	Expected	Where
HTTP error rate	within 1σ of pre-deploy	Grafana service dashboard
p99 latency	within 1σ of pre-deploy	same dashboard
Pod restarts	0 since rollout completion	kubectl get pods
Saturation (CPU/mem)	< 70% of limits	Grafana / kubectl top

If any signal regresses meaningfully, roll back first, investigate after.

Smoke test

A Job template alongside the chart runs a minimal end-to-end check post-sync:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "<chart>.fullname" . }}-smoke
  annotations:
    "argocd.argoproj.io/hook": PostSync
    "argocd.argoproj.io/hook-delete-policy": HookSucceeded
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: curlimages/curl:8.10.1
          command: ["sh","-c"]
          args: ["curl -fsS http://{{ include \"<chart>.fullname\" . }}/healthz && curl -fsS http://{{ include \"<chart>.fullname\" . }}/readyz"]

ArgoCD reports the Application as Progressing until the Job succeeds — a failure surfaces in the UI without manual log-spelunking.

Required output for every change

The agent always produces:

1. Plan — scope, risk, blast radius, prerequisites.
2. Changes — manifests / templates / values / docs touched.
3. Validation — commands above, with expected pass criteria.
4. Rollout procedure — step-by-step, including who approves each gate.
5. Rollback procedure — see rollback.md.
6. Observability checks — dashboard links, alert IDs, log queries.