Roll Forward Recovery

Recovery is a forward Git change that moves the declared system state to a safer state. The forward action may be a targeted fix, feature flag change, revert commit, values-file re-pin, or reviewed data repair.

How recovery choices relate

When an incident hits production, pick the fastest safe recovery path. Each option routes through different verification steps because not all paths carry the same runtime risk.

flowchart TD
    Incident["Production problem"] --> Choice["Choose safest recovery path"]
    Choice --> RollForward["Roll forward with fix"]
    Choice --> Revert["Revert with new commit"]
    Choice --> Rollback["Rollback to prior version"]
    RollForward --> Verify["Verify service and data"]
    Revert --> Verify
    Rollback --> Risk["Check schema, data, and compatibility risk"]
    Risk --> Verify

Rollback adds an extra verification gate: confirm the older version is still compatible with the current runtime before it runs. Roll-forward and revert skip that gate because they deploy code shaped by the runtime changes already in Git.

The terms

Term	What it means	Best-case window	Worst-case risk
Roll forward	Fix the problem in a new commit and deploy it	5-15 minutes for a simple one-line fix	Fix is complex, the fix has a bug, or incident pressure masks the real problem
Revert	Create a new commit that undoes a recent change	3-5 minutes with git and deploy automation	Reverts needed code alongside the bad code and breaks dependent systems
Rollback	Redeploy an older version from a prior state	2-3 minutes with an existing artifact	Old code is incompatible with current schema, messages, cache keys, or vendor APIs

Preference

We prefer roll-forward recovery, including revert commits, because it moves Git forward and avoids silent state collisions.

Rollback

Rollback seems fastest but introduces a hidden problem: you’re redeploying code that may be incompatible with the current runtime state. If any commit since the rollback target touched the database schema, cache key structure, message format, or external API contract, the old code may fail to read the new state or silently corrupt data. A two-minute rollback can mask a two-hour data-recovery incident.

Revert commits

Revert commits are safer than rollback because they respect the current data state, but a revert only works when the problem is isolated to a change that nothing else now depends on. If the revert removes a schema migration, API shape, or feature flag path that later commits require, it becomes its own incident.

Roll-forward fixes

Roll-forward fixes directly address the real problem: identify what is wrong, fix it, and deploy the fix. This is fastest when the fix is clear and small. The risk is incident pressure, so keep the change narrow and verify the service and data immediately after deploy.

Rules

If it changes runtime behavior, it belongs in Git.
Do not repair the cluster by hand and leave Git unchanged.
Emergency work can move faster, but it still uses PRs, review, checks, and audit history.
main must stay deployable; unfinished user-facing behavior belongs behind a flag.
Stabilization is not complete until the follow-up owner and due date are recorded.

Forward actions

Action	Use when	What changes
Fix forward	Cause is clear and the fix is small	New corrective commit
Feature flag off	Risky behavior is already behind a flag	Flag default or targeting
Revert commit	Bad change is isolated and safe to undo	New revert commit
Re-pin tag	Several commits need to be bypassed	Environment values file
Data repair	Data shape is part of the incident	Reviewed data or schema fix

Even git revert is a new commit. That is the point: recovery stays reviewed, visible, and auditable.

Why not classic rollback

Redeploying the previous artifact can be risky when the current database schema, queued messages, feature flags, cache shape, or external records have already changed. Use rollback only when roll-forward or revert would take unacceptable time and you have verified no schema, format, cache, queue, or contract changes since the prior version.

Emergency branch policy

Use an allowed branch prefix, usually fix/ or revert/, plus an incident reference in the PR.

Field	Example
Branch	`fix/prod-login-timeout`
PR title	`fix(auth): restore login timeout handling`
Incident ref	`INC-2026-0610`
Follow-up	Add regression coverage and incident action item

Do not use hotfix/ or emergency/ as branch prefixes.

Avoid

editing Kubernetes resources by hand
patching a pod or container directly
changing ArgoCD UI state without committing the same Git state
bypassing checks because the change is small
bundling cleanup into emergency recovery
leaving stale flags after stabilization

Incident follow-up

After service is healthy, identify why the issue reached users or blocked an environment. Common causes include missing tests, weak UAT, unclear requirements, unreliable infrastructure, incomplete monitoring, and migration risk. The incident is done only when the root prevention work has an owner, date, and tracked fix.

Automation

Automation may alert, link the incident to a release commit, and open draft revert, flag-default, or re-pin PRs. Humans still choose the recovery path when data, migrations, or external side effects are involved.

Evidence

Retain the recovery PR, incident timeline, ArgoCD sync record, release or values-file diff, and post-incident action items.