A GitHub Actions cache key for visual regression baselines

Visual regression testing is a game of spot the difference. You take a photo of every component before the change, a photo after, and let a computer find what moved.

At work we play it on every pull request, with every story in our Storybook: 686 of them. The "before" photos are called baselines. Taking them again every time is slow, so we keep them in a cache, like a folder in a drawer, with a label on it so the next pull request can find them.

For months, the next pull request almost never found them. The cache hit on 3 of the last 50 pull requests. Nobody noticed, because nothing failed. When the folder isn't there, the job quietly takes all the "before" photos again, compares, and goes green. Slower, never red.

Why keep the photos at all

The first question people ask: why not commit the screenshots to the repository?

Because the camera matters. A screenshot taken on my laptop and one taken in CI can differ by a few pixels, just because fonts are drawn a little differently on each machine. So the "before" photos have to come from CI anyway. And 686 images that change every time a shared style changes would make the git history heavier with every redesign. A hosted service like Chromatic solves both, but it charges per snapshot, and we take a lot of snapshots.

The GitHub Actions cache is free and sits right next to the job. A miss costs about a minute. That doesn't sound like much, but it's a minute on every push to every pull request, while someone waits for a green check.

A label nobody could match

In GitHub Actions, the label on the folder is the cache key: a string you build in the workflow. Ask with the same string and you get your files back. Change one character and you get nothing.

Our key looked like this:

key: visual-baselines-${{ runner.os }}-${{ env.ImageVersion }}-${{ hashFiles('**/package-lock.json') }}

In plain words: "the baselines, from this exact machine, with this exact list of installed packages".

I understand why it was written that way. If the machine changes, the pixels might change, so start fresh. It feels careful.

It was wrong twice.

The machine. ImageVersion is the version of the virtual machine GitHub runs your job on, something like 20260803.1.0. GitHub ships a new one most weeks and rolls it out over several days, so for a while two versions are in use at the same time. Picture an office with two identical meeting rooms, where you get whichever is free. You leave your folder in room A, and the next day you're sent to room B. The folder is fine. You're in the wrong room.

The package list. The lockfile lists every package in the project, down to the exact version. Hashing the whole file means any change to it makes a new label: a new lint rule, a type definition update, anything. It's like labelling the folder with the full inventory of the building. Someone buys a new stapler, and you can't find your photos anymore.

So the label was strict about things that almost never change a screenshot. That made the cache useless.

How I found it

I wasn't looking for it. I was looking at where our CI minutes went. That's a job I give to an agent now: I point it at gh run list and gh run view --log for the last few dozen runs, and it tells me which steps eat the time.

One step stood out. "Restore baselines" was followed by a full capture of the baselines, on pull request after pull request, about a minute each time. gh cache list --key visual-baselines showed dozens of saved folders, each used once or never, each with a different machine version or package hash on its label.

If you go looking for the same problem, one detail can fool you. The cache-hit output of actions/cache says true only when the label matched exactly. If it found a close match through restore-keys instead, it still says false, even though you got files back. It's like asking a shop for a red sweater in size M, getting an M in dark red, and the receipt saying "not found". cache-matched-key tells you what you actually got.

One of the agent's ideas was to update our Node version to match what was already cached in CI. It would have sped up CI. It would also have broken the git hooks for every teammate still on the older version. I said no. That's the kind of call an agent can't make for you. The information isn't in the logs.

The misses were hiding a worse bug

This part took me longer to see. The old label had no date on it.

In GitHub Actions, a cache label can only be used once. Once something is saved under a key, saving again under the same key does nothing. Think of a photo album where a slot, once filled, is glued shut.

So a label without "which version of the code" doesn't mean "the latest photos". It means "whatever was saved first". If that key had ever started matching reliably, every pull request would have compared itself with photos of the app from weeks ago, and every change made since then would show up as a difference.

The constant misses were the only thing keeping it honest.

A label that describes the photos

The fix was a different question. Not "where were these photos taken", but "what can change what's in them?"

• The code they were taken from. A different version of the components, different pixels.
• The test setup and how the stories are split. We split the stories across several parallel jobs, and each job has its own set of photos.
• The browser. Playwright pins the exact browser it uses, so a new Playwright version means a new Chrome, and that's the most likely thing to move a pixel.
• The fonts. This is where screenshots really drift between machines. We install our own fonts in the job, so they're part of the label.

The machine version and the rest of the package list went out. In the workflow it's one step:

- name: Baseline cache key
  id: key
  run: |
    base=$(git merge-base origin/main HEAD)
    playwright=$(npx playwright --version | cut -d' ' -f2)
    files=${{ hashFiles('tests/visual/**', 'playwright.config.ts', 'fonts/**') }}
    echo "key=visual-baselines-v1-$base-pw$playwright-shard${{ matrix.shard }}-$files" >> "$GITHUB_OUTPUT"

Three parts of it need a word.

git merge-base finds the commit a pull request started from. If you edit a shared document, you want to see your changes compared with the version you started from, not with what a colleague saved this morning. Same here: a pull request is compared with the main branch it branched off, so other people's later commits don't show up as its differences. git merge-base needs the git history, so the checkout uses fetch-depth: 0.

The Playwright version is read on its own instead of hashing the whole lockfile. Out of the whole package list, that's the one line that matters here.

The v1 is a reset button. If a new GitHub machine ever does change the pixels, it would show up as the same tiny difference on many stories at once. It hasn't happened to us yet. If it does, changing v1 to v2 throws away every saved photo in one commit.

Now the two meeting rooms produce the same label, because nothing in them that takes the photos is different.

Where you keep the folder matters

GitHub has rules about who can see a saved cache. Something saved by a pull request stays with that pull request, like a locker only its owner can open. Something saved on the main branch goes on the shared shelf, and every pull request can take from it.

So the baselines are saved by the run on main, once per commit, and every pull request that started from that commit reuses them. When a pull request misses, it takes its own photos, uses them, and throws them away. Saving them would only fill a locker nobody else can open.

One folder per commit adds up, but it cleans itself up. GitHub removes anything nobody used for a week, and when a repository goes over its limit, 10 GB by default, the oldest unused entries go first.

The result

The cache now hits on roughly 9 pull requests in 10. The misses are the honest ones: a pull request that started from a commit whose photos aren't saved yet, or a Playwright update.

The cache alone took the visual regression suite from 2.1 minutes to under one, because the minute of taking the "before" photos is gone. Together with splitting the suite into parallel jobs and skipping docs-only pull requests, the full checks went from 8.5 minutes to about 4.

What I took away

Label the folder by what's in it, not by the circumstances you saved it in. A cache key should describe the output: the things that change the result. Leave out the things that just happen to be around. Build systems like Bazel have worked this way for years, and a CI cache is no different.

And check your cache hit rate once in a while. A fridge that stops cooling doesn't beep. You just keep buying milk.

Update, October 2026

I later built the same idea into Storybook Swiss Knife, my open-source tool for visual and accessibility checks. Its label starts with the commit the photos were taken from, then a fingerprint of the camera: every file that takes or compares the screenshots, hashed together.

export function captureFingerprint(runnerRoot, visualConfig) {
  const hash = crypto.createHash("sha256")
  const files = [
    "package-lock.json",
    "playwright.config.js",
    ...fs.readdirSync(path.join(runnerRoot, "lib")).map((file) => `lib/${file}`),
    ...fs.readdirSync(path.join(runnerRoot, "tests")).map((file) => `tests/${file}`),
  ]
    .filter((file) => file === "package-lock.json" || file.endsWith(".js"))
    .sort()
  for (const file of files) {
    hash.update(file)
    hash.update(fs.readFileSync(path.join(runnerRoot, file)))
  }
  hash.update(JSON.stringify(visualConfig))
  return hash.digest("hex").slice(0, 20)
}

The lockfile here belongs to the screenshot tool, not to your project, so it only changes when the camera changes. The comparison settings are in it too, because a stricter "spot the difference" means different results.

The machine version is still in the label, but at the very end. If no saved folder matches the exact machine, restore-keys falls back to the same commit's photos from another machine, and the job prints a warning. Trimmed down, the two steps look like this:

- name: Baseline cache key
  id: key
  run: |
    prefix="$KEY_PREFIX-$ImageOS-s${SHARD%%/*}-"
    echo "prefix=$prefix" >> "$GITHUB_OUTPUT"
    echo "key=$prefix$ImageVersion" >> "$GITHUB_OUTPUT"

- name: Restore baselines
  id: baseline
  uses: actions/cache/restore@v6
  with:
    path: .swiss-knife/snapshots
    key: ${{ steps.key.outputs.key }}
    restore-keys: ${{ steps.key.outputs.prefix }}

Two machines can draw a pixel slightly differently, which is why the warning exists. I'd still take a warning over taking every "before" photo again on every pull request.