Visual and accessibility checks for every Storybook PR

Three weeks after I shipped Storybook previews for every pull request, I noticed I had stopped opening them.

The project was storybook-github-pages, a GitHub Action that gives every pull request its own Storybook preview link. It worked. But a preview link is like being handed the keys to a house and asked to find what the builders changed. You'd have to walk every room, open every cupboard and notice that one shelf moved 2 centimetres.

Nobody does that for hundreds of components. I don't either, and I built the thing.

What I wanted was a report waiting on the doormat: these rooms changed, this one broke, this door has no handle anymore. That grew into a new project: Storybook Swiss Knife.

What it does

On every pull request it builds Storybook, takes a screenshot of every story, compares each one with the main branch, and runs axe, an automated accessibility checker, on the same pages. Interaction tests written as play functions run in the same pass, so a broken interaction shows up next to the screenshots.

You see the results in three places:

• two required checks on the pull request, swiss-knife / visual and swiss-knife / accessibility;
• one comment with the preview link and both results;
• a Visual panel inside the preview, with the before and after screenshots side by side and the changed pixels highlighted.

If a visual change is on purpose, a reviewer adds the visual-approved label and the visual check turns green. Accessibility works differently: known problems are recorded in a baseline, and only new ones block. I wrote about that in its own post.

It runs entirely in GitHub Actions and publishes to GitHub Pages, so there's no external service to sign up for. It supports Storybook 8.6, 9 and 10, any framework, and npm, pnpm, yarn or bun.

Why trust a green check

Taking screenshots with Playwright is a solved problem. Trusting the green check that comes out of them is harder.

A pull request is code from someone else. If the job that runs that code is also the job that posts the green check, it's a student grading their own exam. The pull request can change the job, and the job can post whatever it likes.

So the work is split in two:

• The student sits the exam. visual.yml builds the pull request and takes the screenshots with a read-only token and no secrets. All it can do is hand in its answers: a bundle of results.
• The teacher marks it. visual-gate.yml runs on workflow_run, from the main branch. It never runs the pull request's code. It reads its settings from the main branch, decides, and posts the checks.

A workflow_run trigger can't live inside a reusable workflow, so the teacher needs a small file of its own in your repository:

# .github/workflows/visual-gate.yml
name: Visual Gate
on:
  workflow_run:
    workflows: ["Visual"]
    types: [completed]
permissions:
  contents: read
jobs:
  gate:
    if: github.event.workflow_run.event == 'pull_request'
    uses: Archetipo95/storybook-github-swiss-knife/.github/workflows/visual-gate.yml@v0.7.0
    permissions:
      actions: read
      checks: write
      contents: write
      issues: write
      pull-requests: write
    with:
      caller_workflow: .github/workflows/visual.yml
      protected_paths: .storybook/swiss-knife.json

caller_workflow and protected_paths stop the student from editing the exam itself. A pull request that changes the visual workflow or the capture settings fails the checks. Those changes go in a pull request of their own, where nobody can miss them.

The threat model in the docs is explicit that the checks protect against mistakes and unnoticed changes, not against a malicious author. A pull request can still change what its own stories show. That's what code review is for.

The approval label that approved too much

The bug I'm least proud of was in the label.

An approval is a signature on a page. It should only cover the page you read. The first version accepted visual-approved if it was added after the first run on the latest commit had started. That sounds fine until someone pushes while a reviewer is looking. The reviewer approves the screenshots of commit A. Meanwhile commit B is still being captured. The label is newer than B's run, so it approves B too, and nobody has seen B's screenshots.

Force-pushing back to an older commit was worse: it brought back an approval given to a different commit.

The fix in 0.6.0 made the rule strict. The label counts only if it was added after the run that captured these exact screenshots had finished, and after the last force-push. A new push, a force-push or a re-run of the capture withdraws it. You label again once the new results are in.

It's annoying in the moment, but it's the only version of the rule where a green check means a person looked at the pixels that will be merged.

Bugs I only found by running it

Some of my favourite bugs from the changelog:

• A story with closed or timeout in its name, like accordion--closed, could never show a reviewable change. Playwright's error message includes the screenshot name, and the code that keeps real timeouts and closed pages blocking was searching that message for the bare word. It's a spam filter that blocks every email containing "free", so your colleague Mr Freeman never gets any mail. Every diff on that story was reported as a crash no label could accept.
• Saved screenshots are cached per GitHub machine version, and during a rollout two versions run at once. A job could land on the other version, find nothing saved and fail. Jobs now fall back to the same commit's screenshots from the other version, with a warning. I wrote the whole story in a post about the cache key.
• With visual.threshold: 0.2, Playwright's default, a background changing from blue #1d4ed8 to purple #6d28d9 counts as unchanged. Playwright compares colours in YIQ, the colour space of old analog TV, where brightness carries most of the weight. Those two colours have almost the same brightness, so to the comparison they look like the same grey on a black-and-white TV. At 0.1 it's caught. For a design system, start at 0.1.

None of these are clever. You only find them by running the tool on real Storybooks, again and again. The end-to-end tests do that for three setups: Storybook 8 with Vue on npm, 9 with React on pnpm, and 10 with Vue on yarn.

Built with an agent, owned by me

Swiss Knife went from the first commit to v0.7.0 between October 4 and October 12: 112 commits and more than 500 tests. 106 of those commits are co-authored with Claude, and I'm not going to pretend otherwise.

The agent wrote a lot of the code. Think of a builder who lays bricks faster than anyone you've hired, but has never read the building code. What it couldn't do was decide what "trustworthy" means here: which job may post a check, when a label stops counting, what happens to a pull request from a fork (it gets checks, nothing is published). Those rules came first. Every change had to keep them, and there are tests that check job permissions and that every action is pinned to an exact commit, so a "quick fix" can't loosen them unnoticed.

That's how I work with agents at my day job too: they make the change, the checks catch the obvious regressions, and I still read the diff and own the merge.

Try it

From the root of a repository with Storybook:

npx --yes https://github.com/Archetipo95/storybook-github-swiss-knife/releases/download/v0.7.0/storybook-swiss-knife-0.7.0.tgz init

init writes the workflows and the config, creates the label, and prints what's left: install the addon, make the two checks required, and push once to the main branch so there are screenshots to compare against. The addon isn't on npm yet, which is why the command points at a release file.

It's still 0.x, so expect rough edges. If you try it on your Storybook and something breaks, open an issue on GitHub. That's exactly the feedback I need.

What I took away

Most of the work went into one question: when is a green check allowed to be green? Two rules answer it. The check can't come from the code it's checking, and an approval stops counting as soon as the screenshots it covered change. I only got the second one right in 0.6.0, after I found the label could approve a commit nobody had looked at.

The same two rules made the agent easier to work with. 106 of the 112 commits are co-authored with Claude, and every one had to keep them. Checking a diff against two sentences is much faster than checking it for everything at once: a change that lets the pull request post its own check, or keeps an old approval alive, is easy to spot.