Disaster Recovery

Disaster recovery has two jobs: restore service safely and preserve the audit trail. On this platform, most system recovery is a forward Git change; data recovery is handled through encrypted backups and tested restores.

The safest recovery is the one that restores service without making the system harder to understand afterward. That is why recovery still runs through Git, PRs, ArgoCD, incident records, and restore-test evidence.

Recovery decision path

flowchart TD
  subgraph Triage
    Incident["Production problem"] --> Choice["Choose safest recovery path"]
  end
  subgraph Mitigation
    Flag["Disable flag or fix forward"]
    Revert["Revert with new commit"]
    Repin["Re-pin and check compatibility"]
    Restore["Restore data from backup"]
  end
  subgraph Resolution
    Verify["Verify service and data"]
    RCA["RCA and follow-up"]
  end
  Choice --> Flag
  Choice --> Revert
  Choice --> Repin
  Choice --> Restore
  Flag --> Verify
  Revert --> Verify
  Repin --> Verify
  Restore --> Verify
  Verify --> RCA
  style Verify fill:#dcfce7,stroke:#16a34a,color:#14532d
  style RCA fill:#dcfce7,stroke:#16a34a,color:#14532d

Start with the least invasive action that can restore service without risking data integrity. Stabilization is not complete until the follow-up owner, due date, and evidence are recorded.

First response

When production is unhealthy, do these before guessing at a fix:

Confirm the incident scope: which service, environment, users, and time window are affected.
Identify the most recent changes: PRs, release tags, feature flags, config values, migrations, and dependency updates.
Decide whether the fastest safe mitigation is flag off, fix forward, revert, re-pin, failover, or restore.
Record the decision in the incident timeline before or immediately after acting.
Verify service health and data integrity after the mitigation.

The pressure in an incident makes shortcuts attractive. The record protects the team because it shows what was known, who approved the action, and why the recovery path was chosen.

System recovery

Every cluster, Helm chart, policy, and per-environment values file is declared in Git and reconciled by ArgoCD. That means the system is reproducible: point ArgoCD at the repo, pull immutable images from ECR, and reconcile the declared state.

Use these actions first when they are safe:

Action	Use when	What changes
Disable a feature flag	Risky behavior is already behind a flag	Flag default or targeting
Fix forward	Cause is clear and the fix is small	New corrective commit
Revert commit	Bad change is isolated and safe to undo	New revert commit
Re-pin tag	Several commits need to be bypassed	Environment values file
Reconstruct from Git	Environment state is lost or corrupted	ArgoCD reconciles declared state

Even git revert is a new commit. That is the point: recovery stays reviewed, visible, and auditable.

Choosing the recovery action

Choice	Best when	Watch for
Disable a feature flag	The bad behavior is isolated behind a flag	The flag must actually cover the failing path
Fix forward	The cause is known and the fix is small	Incident pressure can make a rushed fix risky
Revert commit	One recent change caused the issue and nothing depends on it	Reverting may remove useful changes beside the bad one
Re-pin tag	Multiple commits need to be bypassed quickly	Older artifact may not match current data or config
Restore data	Data is lost or corrupted	Restore may overwrite newer valid data if scope is wrong
Reconstruct from Git	Environment state is missing or corrupted	Data still needs its own restore path

Prefer the action that fixes the smallest risky surface. If two choices look equally fast, choose the one with clearer verification.

Rollback risk

Classic rollback can look fastest, but it may redeploy code that no longer matches the current runtime state. Before re-pinning or redeploying an older artifact, confirm that no later change altered the database schema, message format, cache key, feature flag expectation, queue contract, or vendor API interaction.

If that compatibility check is unclear, prefer a narrow fix forward or revert commit. A two-minute rollback can become a data-recovery incident when old code reads new state incorrectly.

Recovery PRs

Emergency recovery PRs should be small and boring. The PR should contain only the mitigation needed to restore service, not cleanup, refactors, or long-term redesign.

Use this PR body shape:

## Incident

INC-2026-0610 — production login timeout failures

## Recovery action

Revert the timeout handler change from PR 421.

## Why this action

The failing behavior started after the timeout handler shipped.
The revert does not touch schema, cache keys, queues, or external API contracts.

## Verification

- Login smoke test passes in staging.
- Production error rate returned to baseline after ArgoCD sync.
- No failed login spike in monitoring after 15 minutes.

## Follow-up

- Add regression coverage for timeout behavior.
- Review alert threshold in the incident RCA.

The PR is part of the incident record. A future reader should understand the recovery without reading the whole chat transcript.

Data recovery

Git does not hold databases or uploaded assets. Those are protected separately:

databases run on Amazon Aurora or RDS and are encrypted at rest with AWS KMS
database snapshots are stored durably in S3
object storage uses S3 durability and encryption controls
block storage uses encrypted EBS volumes
transfers use TLS 1.2+ encrypted channels

A backup that has never been restored is not evidence. Restore tests prove that snapshots can actually bring data back before an incident depends on them.

Restore checks

Before restoring data, confirm:

which dataset is affected
when the bad state started
which snapshot or backup predates the bad state
whether a partial restore is safer than a full restore
who approved the restore
how data integrity will be checked afterward

After restoring data, verify with application smoke tests and direct spot checks. Do not declare recovery complete only because the restore job exited successfully.

Recovery objectives

When a disaster occurs, recovery runs through the Incident Response runbook.

Target	Time
Acknowledge	15 minutes or less
Mitigate	2 hours or less
Recover	4 hours or less
Root Cause Analysis	72 hours or less after restoration
Evidence retention	60 days or more, or per record policy

Response objectives prioritize restoring normal service quickly while preventing data loss or corruption. Recovery is never allowed to trade data integrity for speed.

Emergency branch policy

Emergency work uses a normal allowed prefix, usually fix/ or revert/, plus an incident reference in the PR.

Field	Example
Branch	`fix/prod-login-timeout`
PR title	`fix(auth): restore login timeout handling`
Incident ref	`INC-2026-0610`
Follow-up	Add regression coverage and incident action item

Do not use hotfix/ or emergency/ branch prefixes. The emergency path can move faster, but it still needs review, checks, audit history, and follow-up.

Stabilization and follow-up

After the service is healthy:

confirm the user-visible symptom is gone
confirm data integrity was not harmed
confirm monitoring has returned to baseline
link the recovery PR, release record, and ArgoCD sync in the incident
assign follow-up work for tests, alerts, documentation, or process gaps
complete the RCA within the target window

An incident is not done when the page is quiet. It is done when the team knows why the issue reached users and what will prevent the same failure mode from returning.

Avoid

editing Kubernetes resources by hand
patching a pod or container directly
changing ArgoCD UI state without committing the same Git state
bypassing checks because the change is small
bundling cleanup into emergency recovery
leaving stale flags after stabilization

ATO evidence

Disaster-recovery evidence should prove that recovery was controlled, backups were configured, restores were tested, and follow-up work was assigned.

Evidence	Supports
Backup policy and encryption evidence	`CP-9`
Quarterly restore-test report	`CP-10`
Recovery PR, release diff, or values-file re-pin	`CM-4`, `IR-4`
Incident timeline and post-incident action items	`IR-4`, `IR-5`
ArgoCD sync record after recovery	`CM-4`, `IR-4`