triaging-visual-review-runs

by posthog

Visual Review is PostHog's screenshot-regression product: CI captures storybook + playwright screenshots, diffs them against committed baseline hashes, and gates the PR until a human approves the visible changes. A PR with visual changes carries a visual-review GitHub status check that stays red until each diffed snapshot is approved or tolerated in the VR UI . npx skills add https://github.com/posthog/ai-plugin --skill triaging-visual-review-runs Download ZIPGitHub59

Triaging visual review runs

Visual Review is PostHog's screenshot-regression product: CI captures storybook + playwright screenshots, diffs them against committed baseline hashes, and gates the PR until a human approves the visible changes. A PR with visual changes carries a visual-review GitHub status check that stays red until each diffed snapshot is approved or tolerated in the VR UI.

This skill teaches an agent how to answer the questions a human reviewer would actually ask, by chaining the VR MCP tools — instead of reaching for gh pr view and tab-hopping to the VR web UI. The read tools cover status / scope / history / triage. Two are reversible DB-only triage marks (approve-create, tolerate-create); one ships the change (finalize-create) — it commits the baseline and greens the gate, and only that one needs explicit per-run human confirmation.

When this skill applies

Trigger this skill on any of:

• A PR number, branch name, or commit SHA paired with words like visual review , VR , snapshot , screenshot , storybook diff , playwright snapshot , baseline , approve , tolerated , quarantine .

• Questions about why a PR is blocked, what visually changed, or whether a diff is real.

• "Is my run done?" / "What's left to review?" / "Has this story flaked recently?"

• A failing visual-review GitHub check or a PR comment from the posthog-bot mentioning visual review.

When the user asks for the rendered diff image itself, the VR web UI is faster — direct them there. This skill is for everything around the diff: status, scope, history, triage.

Tools

Read tools (safe to call freely):

Tool Purpose posthog:visual-review-runs-list List runs, filter by pr_number / commit_sha / branch / review_state. Start here. posthog:visual-review-runs-retrieve Full detail for a single run (status, summary counts, supersession). posthog:visual-review-runs-snapshots-list Per-snapshot results inside a run: identifier, result, diff %, classification, baseline + current artifact URLs. Quarantined snapshots are excluded by default (see quarantined_count); pass include_quarantined=true to see them. posthog:visual-review-runs-snapshot-history-list A single story's last N runs across master/PRs — the flake check. posthog:visual-review-runs-counts-retrieve Aggregate counts for queue triage (how many runs in needs_review, etc.). posthog:visual-review-runs-tolerated-hashes-list Hashes the team has explicitly accepted as "known flake / acceptable variation". posthog:visual-review-repos-list Repos (one per GitHub repo) — usually only one matters; useful for filtering. posthog:visual-review-repos-retrieve Repo metadata: baseline file paths, PR-comment configuration.

Triage tools (reversible, DB-only — they record a review decision but do NOT change the baseline or the gate):

Tool Purpose posthog:visual-review-runs-approve-create Mark changed / new snapshots reviewed (approved) in the DB. Does NOT commit or green the gate — ship via finalize. posthog:visual-review-runs-tolerate-create Mark a single changed snapshot as a known tolerated alternate. Does NOT change the baseline — use for benign variants.

Ship tool (irreversible, outward-facing — requires explicit per-run human confirmation; see the gate ):

Tool Purpose posthog:visual-review-runs-finalize-create Commit the approved baseline to the PR branch and green the GitHub visual-review check. This ships the change.

Mark-reviewed call shape (approve-create):

• id (required) — the run UUID. It's the route parameter, so the call fails without it.

• snapshots: [{identifier, new_hash}] — new_hash is the content_hash of each snapshot's current_artifact. This only records the review in the DB; nothing is committed and the gate stays red until you finalize.

Toleration call shape — both fields are required:

• id (required) — the run UUID. It's the route parameter, so the call fails without it.

• snapshot_id (required) — the UUID of the individual snapshot to tolerate (from visual-review-runs-snapshots-list). This identifies which snapshot inside the run; it does not replace the run id.

Finalize call shape (finalize-create) — the all-or-nothing ship action:

• id (required) — the run UUID.

• approve_all: true — approve every still-pending changed/new snapshot before finalizing (tolerated ones are left alone). Use when you've verified every remaining diff is intended.

• Omit approve_all (default false) to finalize a run you've already reviewed snapshot-by-snapshot. Finalize is all-or-nothing: it fails with 409 not_fully_resolved (and lists what's left) unless every changed/new snapshot is approved, tolerated, or quarantined.

• It commits exactly the snapshots approved in the DB — tolerated snapshots keep their baseline and are never overwritten. When a baseline commit is pushed, its SHA comes back on the run's metadata.baseline_commit_sha. It's absent when nothing needed committing (everything resolved by toleration/quarantine — the gate still greens) or when the commit was skipped: no PR, or a 409 sha_mismatch because the PR has newer commits (that one leaves the gate red — re-run CI on the latest commit and finalize again).

If finalize fails with 409 stale_run, the run has been superseded — visual-review-runs-list { pr_number } and finalize the newest one. A successful finalize often kicks off a fresh CI run, which is normal.

The finalize gate

Finalize is the one irreversible, outward-facing action in this skill: it rewrites the baseline committed to the PR and greens the merge gate. Treat it like pushing to someone's branch — never automatic.

Before any finalize-create call, all of these must hold:

• You verified the diffs. You pulled the current (and, for changed, baseline) PNGs and looked at them, ran the flake check on anything suspect, and reached a per-snapshot verdict. Metadata alone is never enough.

• You presented the verdict and waited. Show the user, per snapshot, what changed and your recommendation, then stop.

• The user explicitly approved this run. A broad "get the gate green" / "fix the PR" is permission to investigate and recommend — NOT to finalize. When the task implies finalizing but the human hasn't said it for this specific run, ask.

approve-create and tolerate-create are reversible triage and don't need this gate — but they don't ship anything either. The moment you're about to finalize-create and can't point to a specific human "yes" for this run, stop and ask.

Vocabulary cheat sheet

These appear in tool output and matter for interpretation:

• Run review_state: needs_review (open, awaiting human), clean (zero diffs), processing (CI still uploading), stale (a newer run on the same PR has superseded this one — check superseded_by_id).

• Run run_type: storybook (component snapshots) or playwright (full-page e2e snapshots).

• Snapshot result: unchanged, changed (real diff), new (no baseline yet), removed.

• Snapshot classification_reason: tolerated_hash (matches a known-tolerated hash, no action needed), below_threshold (under the noise floor), exact (byte-identical), "" (real diff requiring review).

• Snapshot review_state: pending or approved.

• Run summary: total / changed / new / removed / unchanged / unresolved / tolerated_matched — unresolved is what's actually blocking review.

Workflows

"What's the VR status of this PR?"

The single most common job. Map a PR number to its run state in two calls.

• posthog:visual-review-runs-list { pr_number: <n>, limit: 5 } — sort by created_at desc, take the latest non-stale one.

• If the run has summary.changed > 0 or summary.unresolved > 0, drill in: posthog:visual-review-runs-snapshots-list { id: <run_id> } and report the changed snapshots.

Report back: PR number, run UUID, review_state, summary counts, and the _posthogUrl deep link so the user can click straight to the diff viewer.

"Is the diff real or unrelated?"

The most useful judgment a code-aware agent can add. Combine three signals: scope match, flake history, and the actual rendered images. The agent should look at the screenshots — not just describe metadata.

Scope check — git diff master...HEAD --stat (or against the PR's base branch) → list of touched paths. Cross-reference with posthog:visual-review-runs-snapshots-list { id } filtered to result: changed → story identifiers. Stories are namespaced like <area>-<scene>--<story>--<theme>; e.g. scenes-app-settings-user--settings-user-profile--dark maps to frontend/src/scenes/settings/user/.... Use this to translate story id → likely source path.

Visual inspection — for each changed snapshot, the tool result contains current_artifact.download_url and baseline_artifact.download_url. These are pre-signed S3 URLs to PNG files; pull them and look:

curl -s -o /tmp/vr-baseline.png " "
curl -s -o /tmp/vr-current.png " "

Then Read both files (the Read tool renders images visually) and compare. Things to call out:

• The actual visible delta (text changed, button moved, layout shift, color drift, missing element).

• Whether the change is consistent with the diff_pixel_count and diff_percentage in the metadata (e.g. 54% diff but the images look near-identical → screenshot framing changed, not the UI).

• Whether the baseline and current have different dimensions (width / height fields). Mismatched dimensions usually mean the story rendered to a different viewport or didn't fully render before screenshot — a flake signal, not a regression.

Flake history — run the flake check below for any story that looks suspect.

Verdict — combine all three:

• Scope plausible + visible regression matches the code change → real diff, recommend approval.

• Scope mismatch + dimensions mismatch + frequent prior changes → flake, recommend tolerating the hash.

• Scope plausible + visible regression looks unintended → push a fix; do not approve.

Always include a one-line description of what you saw in the images — the user uses this to decide whether to trust your verdict without opening the VR UI themselves.

Flake check: "Has this story been changing?"

Once you have a suspect snapshot identifier:

posthog:visual-review-runs-snapshot-history-list { id: <snapshot_id> } → returns prior outcomes for the same story.

Verdicts:

• Mostly unchanged and this run's diff is the outlier → likely a real regression caused by this PR.

• Frequent changed across unrelated branches/master → flaky story; recommend tolerating the hash via the UI.

• Recent removed or large-jump dimension change → baseline likely stale; recommend re-baselining on master.

Triaging the queue

When the user is doing housekeeping rather than asking about a specific PR:

• posthog:visual-review-runs-counts-retrieve → total queue size.

• posthog:visual-review-runs-list { review_state: needs_review, limit: 50 } (paginate if needed).

• Group by branch author or run_type to surface clusters (e.g., "12 PRs blocked on the same shared component change" usually means a single underlying root cause to address).

• Prefer surfacing runs whose summary.changed > 0 over runs that are only new — new means no baseline yet, which is usually trivial to approve; changed is the real review work.

Output expectations

For PR-status questions, lead with the verdict in one line, then 2-4 bullets of supporting context. Always include the _posthogUrl deep link to the run — humans need to see the rendered images to make the call, the agent can only describe the metadata.

For triage / aggregate questions, a short table beats prose. Group by what the user is going to act on.

What NOT to do

• Do not approve or tolerate without explicit user confirmation. The verdict is yours to recommend; the decision to ship belongs to the user. Once they say "approve those" / "tolerate that", call the tool.

• Do not assume the failing GitHub check on a PR is unrelated to VR — if a visual-review check is red on a PR you're working on, that's the trigger to run this skill.

• Do not declare a verdict from metadata alone when result: changed. Pull the baseline and current PNGs and look at them; metadata can only say "something changed", not whether the change is intended.