Monitoring

Our approach

Monitoring and alerting are not afterthoughts here — they are woven directly into the GitOps workflow and embody the Zero Trust principle of continuous verification. Every log, metric, and event reinforces traceability; every action leaves an auditable trail; and every alert feeds back into the same loop the platform runs on: code, commit, correct.

The feedback loop

Everything begins and ends with GitHub — the single source of truth.

The diagram below shows how a critical alert travels from detection through remediation and back into the auditable Git record.

flowchart TD
  subgraph Automation[Automation]
    Detect["Incident detected"]
    Notify["Critical alert routed to Google Chat"]
  end
  subgraph Human["Human work"]
    Triage["Engineer triages alert"]
    Record["GitHub issue or PR opened"]
  end
  subgraph Audit_Trail["Audit trail"]
    Git["Change recorded in Git"]
  end
  Detect -->|"on threshold breach"| Notify
  Notify -->|"on receipt"| Triage
  Triage -->|"on remediation start"| Record
  Record -->|"on merge or close"| Git
  style Git fill:#dcfce7,stroke:#16a34a,color:#14532d

When an incident occurs, critical alerts are surfaced automatically to the appropriate Google Chat channel, where engineers can act immediately. All remediation — from triage to resolution — is documented in GitHub issues or PRs, so the entire lifecycle stays auditable. For how we treat the incidents themselves, see the Incident Response runbook.

Monitoring

Monitoring is multi-dimensional, covering infrastructure, applications, and user experience. Grafana is the central observability hub, integrating logs, metrics, and traces and visualizing health across:

• Infrastructure metrics — EKS clusters, nodes, pods, network throughput.
• Application performance — Drupal response time, database latency, CloudFront cache ratios.
• Resource utilization — CPU, memory, storage thresholds.
• Incident tracking — alert trends, SLA compliance, uptime reports.

Grafana pulls event data from AWS CloudWatch, CloudTrail, and Security Hub, and integrates with GuardDuty and Wiz to flag anomalous behavior or posture deviations. Real-time alerting rules ensure no silent failures occur.

Alerting

Effective alerting is about context, not noise — alerts are tuned to actionable conditions so engineers are notified only when intervention is needed. We meet developers where they already collaborate: Google Chat. Critical notifications route to specific spaces via webhooks and always link back to the originating GitHub issue or PR, so the thread carries immediate traceability and the team can collaborate or @-mention the right people in place.

Alerting rules

Rules balance sensitivity and clarity — highlighting genuine risk without overwhelming the team:

Rule	Triggers when
Pod Health	Critical pods crash, restart frequently, or fail readiness probes
Cluster Capacity	CPU or memory exceeds 80% sustained for more than 5 minutes
Ingress/Egress Anomalies	CloudWatch + Firewall Manager logs show unusual traffic or blocked ranges
Application Errors	High 5xx rates, elevated latency, or failed database connections
Security Findings	GuardDuty, Wiz, or Security Hub critical findings (routed to security)
Drift Detection	Terraform or ArgoCD detects drift between declared and live state

Each rule maps to one or more Google Chat channels — the table below shows which channel receives each category of notification.

Alerting channels

Channel	Carries
releases	New production releases, deployment outcomes, post-deploy verifications
security	Critical vulnerabilities, compliance findings, GuardDuty alerts
outages	Real-time disruptions, degraded performance, availability incidents
cluster-resources	Utilization warnings, scaling events, performance thresholds

Security findings live in GitHub

Security visibility is centralized in GitHub Advanced Security using SARIF (Static Analysis Results Interchange Format). Every scanner — Trivy, Dependabot alerts, Kyverno, and Wiz — exports its results as SARIF directly into GitHub Advanced Security where supported. Renovate opens the dependency update PRs that turn those findings into reviewed remediation work. In a GitOps workflow this matters because:

• Git is the single source of truth, so all findings are traceable to code, not to infrastructure drift.
• Developers fix, discuss, or dismiss findings through auditable pull requests — the same flow as any other change.
• Enterprise administrators get consolidated visibility across every repository.

False positives are not discarded; they are annotated and recorded in GitHub, preserving institutional knowledge and the audit trail.

Audit retention

After every deployment, the trail is preserved for evidence and review: ArgoCD retains commit and image metadata for traceability, GitHub logs are retained for one year, Wiz continuously scans for compliance drift and vulnerabilities, and Grafana/CloudWatch track performance and anomalies. Results are tied to POA&M items for risk tracking and evidence review. These specific sources are retained because together they cover the three evidence categories required for ATO maintenance — change provenance (ArgoCD + GitHub), posture state (Wiz), and operational health (Grafana/CloudWatch) — so any auditor inquiry can be answered from a single coherent record.

Audit events must include actor, action, target resource, environment, result, timestamp, and source system. Security-relevant events include authentication, authorization failure, privilege change, deployment, production promotion, secret access, policy violation, vulnerability finding, incident creation, and administrative override.

Where it’s enforced

• Monitoring and alerting configs are deployed via GitOps. Grafana dashboards, alert rules, and routing are declared in Git and reconciled by ArgoCD — so “what do we monitor and when do we page?” is answerable from the repo and changes go through the same review as any other code.
• AWS-native telemetry (CloudWatch, CloudTrail, Security Hub, GuardDuty) and Wiz feed findings into the loop continuously, with no silent-failure gaps.
• Routing back to Git. Every critical alert links to a GitHub issue or PR and ties to POA&M items, so detection, remediation, and audit retention stay inside the same auditable system. Critical alerts are acknowledged within 15 minutes and triaged into an incident, finding, or false-positive record.

ATO evidence

Monitoring evidence should prove that audit events are generated, retained, reviewed, and routed to accountable remediation. Retain the audit event taxonomy, log retention policy, alert routing table, monthly monitoring review, and POA&M-linked finding records.

Evidence	Supports
Audit event taxonomy and log samples	AU-2, AU-3, AU-12
Retention policy and review record	AU-6, AU-11
POA&M-linked finding records	CA-7, SI-4

See also

• Incident Response runbook — how alerts become incidents, RCAs, and reports.
• 06 — Compliance — how Renovate turns dependency findings into reviewed PRs.
• 02 — Networking — the flow logs and findings feeding Security Hub.
• 01 — Authorization — which monitoring tools Platform Engineering manages versus inherits.
• Sourced from the platform’s security & ATO documentation.