FRD: Make Visual Regression Gate Blocking

Field	Value
ID	FRD-046
Owner	David Holmes
Status	Draft
Last Updated	2026-05-26
Target Release	v2.1.0
Type	Infra
Complexity	M

Document Summary

This FRD defines the work to promote the visual regression step in .github/workflows/validate.yml from advisory (continue-on-error: true at line 95) to blocking. A path filter will be added so that documentation-only PRs bypass the visual gate, keeping CI fast for non-visual changes while enforcing visual correctness for all code and style changes.

Introduction

Overview

The CI pipeline runs visual regression tests via Storybook snapshot comparison. Currently this step is advisory: it runs but cannot fail the build due to continue-on-error: true. This means visual regressions can merge unnoticed. Promoting the step to blocking ensures that unreviewed visual changes cannot reach main.

Goals

Make visual regression failures block PR merge.
Add a path filter so documentation-only PRs (changes limited to docs/, *.md, *.mdx) skip the visual regression step entirely and remain fast.
Ensure a clear workflow for intentional visual changes (updating baseline snapshots).

Non-Goals

Changing the visual regression tooling or test infrastructure.
Adding new visual test coverage (covered by FRD-047/undocumented stories).
Modifying other CI steps.

Scope

In Scope

Item	Description
`.github/workflows/validate.yml`	Remove `continue-on-error: true` from the visual regression step.
Path filter	Add `paths-ignore` or a conditional to skip visual regression when only doc files changed.
Baseline update workflow	Document the process for updating visual snapshots when intentional visual changes are made.
CI status check	Ensure the GitHub branch protection rule includes the visual regression check.

Out of Scope

Item	Reason
Visual regression tooling changes	Tooling is stable; this FRD changes only the gate behavior.
New snapshot baselines	Covered by the undocumented stories FRD.
Flaky test investigation	Assumed addressed before making the gate blocking.

Users and Pain Points

User	Pain Point
Component developer	Visual regressions merge unnoticed because the gate is advisory, causing downstream surprises.
Design reviewer	Cannot rely on CI to catch unintended visual changes; must manually review screenshots.
Docs contributor	If visual regression is blocking on all PRs, docs-only PRs are slowed by unnecessary screenshot comparison.

Definitions

Term	Definition
Visual regression	An unintended change to the rendered appearance of a component, detected by pixel-level comparison of Storybook snapshots.
Baseline snapshot	The reference screenshot that represents the expected appearance of a component. Updated intentionally when a visual change is approved.
Advisory gate	A CI step that reports results but does not block merge on failure (`continue-on-error: true`).
Blocking gate	A CI step whose failure prevents PR merge.

Current State

.github/workflows/validate.yml line 95 has continue-on-error: true on the visual regression step. This means:

The step runs and reports visual differences.
The overall workflow succeeds regardless of visual regression results.
PRs with visual regressions can merge without review of the visual diff.
The advisory status was originally set because the baseline snapshots were incomplete and flaky tests caused false positives.

Proposed Solution

Remove `continue-on-error`

Delete continue-on-error: true from the visual regression step in validate.yml. The step will now fail the workflow on any snapshot mismatch.

Add path filter

Add a conditional to the visual regression step (or the job containing it) that skips when the PR only modifies documentation files:

if: |
  !contains(github.event.pull_request.title, '[docs-only]') &&
  (
    github.event_name != 'pull_request' ||
    contains(needs.changes.outputs.files, 'src/') ||
    contains(needs.changes.outputs.files, '.css') ||
    contains(needs.changes.outputs.files, 'package.json')
  )

Alternatively, use the dorny/paths-filter action to detect whether any non-doc files were changed, and gate the visual step on that output.

Baseline update workflow

Add a CI comment or documentation explaining how to update baselines:

Run pnpm test:visual:update locally to regenerate changed snapshots.
Commit the updated baseline images.
Push and re-run CI.

Branch protection

After the change, ensure the GitHub branch protection rule for main includes the visual regression check as a required status check (when it runs).

Requirements

ID	Priority	Requirement
VR-01	P0	Visual regression step failures block PR merge.
VR-02	P0	Documentation-only PRs skip the visual regression step.
VR-03	P0	Intentional visual changes can be resolved by updating baseline snapshots.
VR-04	P1	CI output clearly indicates which snapshots differ and how to update them.
VR-05	P1	Branch protection rule includes the visual regression check.

Functional Requirements

When a PR changes files under src/, .css files, package.json, or Storybook configuration, the visual regression step runs and is blocking.
When a PR changes only files under docs/, *.md, or *.mdx, the visual regression step is skipped.
On snapshot mismatch, the CI step fails with a clear error message listing the differing snapshots.
The CI failure message includes instructions for updating baselines.
Updated baseline snapshots committed to the PR cause the visual regression step to pass on re-run.

Non-Functional Requirements

Category	Requirement
CI time	Documentation-only PRs must not wait for visual regression (~3-5 minutes saved).
Reliability	The visual regression step must have a false-positive rate below 1% before being made blocking. Flaky tests must be investigated and fixed first.
Developer experience	The baseline update workflow must be documented and take fewer than 5 minutes.

API/Interface Requirements

No API changes. The workflow file is the interface.

Accessibility Requirements

Not applicable. This is a CI configuration change.

Content and Documentation Requirements

Update CONTRIBUTING.md (or equivalent) with the baseline update workflow.
Add a comment in validate.yml explaining why the visual step is blocking and how to handle failures.
Optionally add a CI bot comment on PRs that fail visual regression, linking to the update instructions.

Dependencies

Dependency	Type	Notes
`.github/workflows/validate.yml`	Internal	The workflow file to modify.
`dorny/paths-filter` (optional)	External	GitHub Action for path-based conditional steps. Already used in some workflows.
Stable baseline snapshots	Internal	Baselines must be up-to-date before making the gate blocking.
GitHub branch protection	External	Must be updated to include the visual regression check.

Risks and Tradeoffs

Risk	Likelihood	Impact	Mitigation
Flaky visual tests block legitimate PRs.	Medium	High	Audit and fix flaky tests before enabling the blocking gate. Add a retry mechanism (1 retry) to the visual step.
Developers unfamiliar with snapshot updates create merge delays.	Medium	Medium	Document the update workflow. Add a CI bot comment with instructions on failure.
Path filter incorrectly skips visual regression for a PR with mixed changes.	Low	Medium	Use `dorny/paths-filter` with explicit include patterns rather than exclude patterns. Err on the side of running the visual step.

Open Questions

Should the visual regression step have a single automatic retry before failing, to absorb rare rendering flakes? Leaning toward yes, one retry.
Should we use dorny/paths-filter or GitHub’s native paths filter on the workflow trigger? dorny/paths-filter is more flexible for per-step conditionals.
Should baseline snapshot updates require a separate approval from a design reviewer? Leaning toward no for v1 (any team member can approve), revisit if visual quality regresses.

Acceptance Criteria

continue-on-error: true is removed from the visual regression step in validate.yml.
A PR that changes src/ files and has a snapshot mismatch fails CI.
A PR that changes only docs/ files skips the visual regression step and passes CI.
Updating baseline snapshots and pushing resolves the visual regression failure.
The baseline update workflow is documented.
GitHub branch protection includes the visual regression check.
pnpm typecheck passes.

LLM Handoff Instructions

When implementing this FRD:

Open .github/workflows/validate.yml and locate the visual regression step (around line 95).
Remove continue-on-error: true.
Add a path-detection step using dorny/paths-filter@v3 that sets an output has_visual_changes to true when any file outside docs/, *.md, *.mdx is modified.
Add if: needs.changes.outputs.has_visual_changes == 'true' to the visual regression step.
Add a comment in the YAML explaining the path filter logic.
Test by creating a docs-only PR (should skip visual step) and a code PR (should run and block on mismatch).
Update branch protection settings via GitHub UI or gh api to include the visual regression check.
Document the baseline update workflow in CONTRIBUTING.md or the project’s developer guide.

Decision Log

Date	Decision	Rationale
2026-05-26	Make visual regression blocking.	Advisory gates provide no enforcement; visual regressions have merged unnoticed.
2026-05-26	Skip for docs-only PRs.	Documentation changes cannot cause visual regressions. Running the step wastes 3-5 minutes.
2026-05-26	Use `dorny/paths-filter` for path detection.	More flexible than workflow-level `paths` filter; allows per-step conditionals.

Document History

Version	Date	Author	Changes
0.1	2026-05-26	David Holmes	Initial draft.