Environments

Environments are promotion targets, not branches. They share the same Git-declared delivery model while staying separated by AWS account, runner group, IAM role, network boundary, data policy, and approval gate.

The point is narrow and important: environments should differ by configuration and access control, not by a separate code history. That gives developers one release path while giving ATO reviewers clear evidence for isolation.

Mental model

Think of the application as one artifact moving through several rooms. The artifact should stay the same; the room changes around it.

The image digest is the artifact. The environment values, network boundary, runner group, secrets references, and approval rules are the room. When a bug appears only in staging or production, start by comparing the room before assuming the artifact changed.

Tier model

Tier	Lower	Upper
Environments	Preview, Development, Test	Staging, Production
AWS account	`<lower-tier AWS account>`	`<upper-tier AWS account>`
Domain	`example.dev`	`example.gov`
Runners	`dev-runners`	`prod-runners`
Purpose	Development, integration testing, pre-release validation	Authoritative, public-facing layers

Lower environments give developers and security engineers a safe space to test code, configuration, and infrastructure changes before promotion. Upper environments are authoritative, public-facing layers and cannot reuse lower-tier credentials, network paths, or production data copies.

Lower and upper tier responsibilities

Responsibility	Lower tier	Upper tier
Purpose	Fast validation, integration, and pre-release confidence	Public-facing validation and production service
Data	Synthetic, anonymized, or test data only	Production or production-like data under stricter access
Access	Broader developer access, still least-privilege	Restricted operational access
Promotion	Automatic or lower-risk movement	Human-approved movement
Evidence	Preview, test, and values-diff records	Approval, release, sync, and monitoring records

What to notice: lower is not “anything goes.” It is safer because it is isolated, not because controls disappear.

Promotion flow

flowchart LR
  subgraph Lower["Lower tier"]
    PR["Preview from PR"]
    Test["Test from main"]
  end
  subgraph Upper["Upper tier"]
    Staging["Staging approval"]
    Prod["Production approval"]
  end
  PR -->|on merge| Test
  Test -->|on approval| Staging
  Staging -->|on approval| Prod
  style Prod fill:#dcfce7,stroke:#16a34a,color:#14532d

Environment	Input	Purpose
Preview	PR head SHA	Per-PR validation before merge
Development	`main` or active integration branch, when configured	Shared lower-tier integration
Test	`main` plus beta tag	Continuous integration signal
Staging	Release-candidate promotion	Production-like verification
Production	Clean release tag	Public traffic

test updates automatically from main. staging and production are manual because a person decides when to promote. ArgoCD can still auto-sync inside each environment; the manual part is the promotion input.

What happens at each step

Step	What the engineer does	What the system records
Preview	Open or update a PR	PR SHA, preview deployment, checks, comments
Test	Merge reviewed PR to `main`	Squash commit, workflow run, beta tag or image digest
Staging	Request or approve promotion	Approver, release candidate tag, ArgoCD sync
Production	Approve protected deployment	Approver, clean release tag, ArgoCD sync, monitoring signal

If you cannot find the record for a step, treat that as a documentation or evidence gap. The goal is not just to run the software; the goal is to know how it got there.

Runner boundary

flowchart TB
  subgraph Source["Git source"]
    GH["GitHub Enterprise"]
  end
  subgraph Lower["Lower AWS account — example.dev"]
    DevR["dev-runners"]
    LEnvs["Preview · Development · Test"]
  end
  subgraph Upper["Upper AWS account — example.gov"]
    ProdR["prod-runners"]
    UEnvs["Staging · Production"]
  end
  GH -->|triggers| DevR
  GH -->|triggers| ProdR
  DevR -->|deploys to| LEnvs
  ProdR -->|deploys to| UEnvs
  style GH fill:#f0fdf4,stroke:#16a34a,color:#14532d
  style DevR fill:#fef3c7,stroke:#b45309,color:#451a03
  style ProdR fill:#fef3c7,stroke:#b45309,color:#451a03

All deployments originate from GitHub Enterprise and are executed by self-hosted GitHub Actions runners that can reach only the appropriate EKS clusters:

dev-runners deploy exclusively to lower environments within the example.dev account.
prod-runners deploy exclusively to upper environments within the example.gov account.

Both groups operate under segregated IAM roles, network policies, and ArgoCD deployment permissions. Each runner uses IAM Roles for Service Accounts to assume a scoped role for its environment. No deployment workflow, token, or runner has cross-environment visibility or control.

What can differ

The image digest stays the same across promotion. Only environment-specific values may differ: replicas, limits, endpoint URLs, feature flag defaults, non-secret references, and target namespaces. If a staging problem cannot reproduce in test, assume the difference is configuration until proven otherwise.

Lower environments use synthetic or anonymized data only. Production data does not move back into preview, development, test, or staging.

Allowed environment differences

Difference	Allowed reason	Review cue
Replicas and resource limits	Different load and availability needs	Confirm the app still has enough headroom
Endpoint URLs	Each tier calls tier-appropriate services	Confirm no lower tier calls production-only dependencies
Feature flag defaults	Exposure is staged by environment	Confirm defaults are safe and documented
Secret references	Each environment reads its own secret value	Confirm Git stores references, not values
Domain names	Lower and upper tiers use different domains	Confirm production domain is not reused in lower tiers
Network policies	Upper tier is more restrictive	Confirm lower-tier policy does not create a production path

Do not allow differences that change the application artifact itself. Rebuilding an image per environment weakens the whole promotion model because staging no longer proves what production will run.

Access provisioning

Access is provisioned through central SSO. Requests follow the standard service-desk process and must be approved by a Technical Owner or Technical Admin. Grants are fulfilled by IAM and Cloud Services teams so every access grant is traceable, authorized, and least-privilege.

Access provisioned outside this process, including manually attached IAM policies or shared credentials, is not tracked, cannot be audited, and will be revoked when discovered.

When an environment acts different

Use this order before making a code change:

Check the image digest. If it differs, promotion did not use the same artifact.
Check the values diff. Look for replicas, flags, endpoints, limits, and secret references.
Check ArgoCD sync state. An out-of-sync app may not be running the Git-declared state.
Check runner and IAM scope. A failed deployment may be blocked by the wrong runner group or role.
Check data assumptions. Lower environments must not depend on copied production data.
Check monitoring and logs. Confirm whether the symptom is application behavior, network access, or capacity.

Enforcement

Account-level isolation — each tier is a distinct AWS account, enforcing IAM, network, and data boundaries at the highest level.
Runner isolation — runners are environment-scoped, IRSA-controlled, and ephemeral, eliminating cross-environment credential leakage.
Protected environment approval — staging and production promotion require the right human approval.
Zero cross-access — lower-environment apps, APIs, and infrastructure are never directly reachable from upper environments, eliminating lateral-movement risk.
Consistency via GitOps — Terraform, Helm, and ArgoCD guarantee parity while preserving separation.
Centralized auditability — every deployment and access event is logged in GitHub, AWS CloudTrail, and ArgoCD.

ATO evidence

Environment evidence needs to prove both promotion control and data separation.

Evidence	Supports
Environment reviewer settings	`AC-3`, `AC-6`
Runner group and IAM role mapping	`AC-3`, `AC-6`
Values diff between lower and upper environments	`CM-2`, `CM-3`
ArgoCD Application or ApplicationSet record	`CM-2`, `CM-3`
Synthetic-data confirmation for lower environments	`MP-2`, `SC-28`