Visual Regression Testing Without Percy

What is visual regression testing?

Visual regression testing automatically compares screenshots of your UI across code changes to catch visual bugs that unit tests and type checkers miss: layout shifts, font changes, colour regressions, z-index breakages, missing components. A baseline screenshot is captured at a known-good state; new screenshots are pixel-diffed against the baseline on every pull request or deploy.

Visual regressions are the most common class of bug that ships to production despite passing all automated tests. A CSS specificity conflict, a missing font, a changed z-index — none of these fail a unit test. A pixel diff catches all of them.

The cost problem with SaaS visual testing

Percy, Chromatic, and Applitools are powerful tools — but they are priced for enterprise budgets. The cost structure makes them inaccessible for solo developers, small teams, and open-source projects:

Percy's free tier covers 5,000 screenshots per month — roughly 165 screenshots per day. A project with 50 pages running visual tests on every commit burns through that in hours. The paid tier starts at $299/month with no intermediate option.

The free stack: Openkova + pixelmatch

The self-hosted alternative uses two open-source tools:

• @openkova/cli — screenshot any URL or HTML file using your local Chromium. No server, no API key, no Docker required. npm install -g @openkova/cli.
• pixelmatch (or odiff) — pixel-level image comparison in Node.js. Returns a diff count and renders a visual diff image highlighting changed pixels.

Install dependencies

npm install -g @openkova/cli
npm install pixelmatch pngjs

Capture baseline screenshots

# Capture baselines for a list of URLs
kova screenshot https://localhost:3000 --output baselines/home.png
kova screenshot https://localhost:3000/about --output baselines/about.png
kova screenshot https://localhost:3000/pricing --output baselines/pricing.png

Store baselines in git. They are the ground truth your CI will diff against on every subsequent commit.

Diff in CI (Node.js)

// scripts/visual-diff.ts
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
import fs from 'fs';

const THRESHOLD = 0.1; // 10% pixel difference tolerance
const PAGES = ['home', 'about', 'pricing'];

let failed = false;

for (const page of PAGES) {
  const baseline = PNG.sync.read(fs.readFileSync(`baselines/${page}.png`));
  const current = PNG.sync.read(fs.readFileSync(`screenshots/${page}.png`));

  const { width, height } = baseline;
  const diff = new PNG({ width, height });

  const numDiffPixels = pixelmatch(
    baseline.data, current.data, diff.data,
    width, height,
    { threshold: THRESHOLD }
  );

  const diffPercent = (numDiffPixels / (width * height)) * 100;
  fs.writeFileSync(`diffs/${page}.png`, PNG.sync.write(diff));

  if (diffPercent > 1) {
    console.error(`FAIL ${page}: ${diffPercent.toFixed(2)}% pixels changed`);
    failed = true;
  } else {
    console.log(`PASS ${page}: ${diffPercent.toFixed(2)}% pixels changed`);
  }
}

if (failed) process.exit(1);

GitHub Actions workflow

A complete GitHub Actions workflow that captures screenshots, diffs against baselines, and uploads the diff images as artifacts for review:

# .github/workflows/visual-regression.yml
name: Visual Regression Tests
on: [pull_request]

jobs:
  visual-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: |
          npm install -g @openkova/cli
          npm install pixelmatch pngjs

      - name: Start application
        run: npm run build && npm start &
        env:
          NODE_ENV: production

      - name: Wait for app to be ready
        run: npx wait-on http://localhost:3000

      - name: Capture current screenshots
        run: |
          mkdir -p screenshots
          kova screenshot http://localhost:3000 --output screenshots/home.png
          kova screenshot http://localhost:3000/about --output screenshots/about.png
          kova screenshot http://localhost:3000/pricing --output screenshots/pricing.png

      - name: Run visual diff
        run: |
          mkdir -p diffs
          node scripts/visual-diff.js

      - name: Upload diff artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: visual-regression-diffs
          path: diffs/

Updating baselines

When you intentionally change the visual design — a rebrand, a layout update, a typography change — update the baselines:

# Recapture baselines after intentional design change
kova screenshot https://localhost:3000 --output baselines/home.png
kova screenshot https://localhost:3000/about --output baselines/about.png

# Commit the updated baselines
git add baselines/
git commit -m "chore: update visual baselines after nav redesign"

Testing private URLs and staging environments

One significant advantage of the self-hosted approach: @openkova/cli uses your local Chromium, so it can screenshot any URL reachable from the machine running the test — including:

• Localhost and http://127.0.0.1:PORT
• Staging environments on internal networks
• VPN-protected development servers
• GitHub Actions environment with your app started as a service

SaaS visual testing tools run on external servers. To test non-public URLs, you need to set up a tunnel (ngrok, Cloudflare Tunnel) — adding complexity, latency, and a potential security concern for staging environments.

Comparison: pixelmatch vs odiff

Two open-source pixel diff libraries are commonly used with this stack:

For most projects, pixelmatch is the better default — it is pure JavaScript, well-maintained, and has no native compilation step. odiff is faster on large screenshot sets (hundreds of full-page screenshots) but requires native binaries.

Full guide

The detailed step-by-step tutorial — including baseline storage strategies, handling flaky screenshots, and integrating with PR review workflows — is in the dedicated blog post:

Visual Regression Testing Without Percy: A Free Setup →

Frequently asked questions

What is visual regression testing?

Visual regression testing automatically compares screenshots of your UI across code changes to catch visual bugs — layout shifts, colour changes, missing components — that unit tests cannot detect. A baseline screenshot is diffed against a new screenshot; if the pixel difference exceeds a threshold, the test fails.

What is a free alternative to Percy?

@openkova/cli for screenshot capture plus pixelmatch for pixel diffing is a free, self-hosted alternative to Percy. Percy starts at $299/month for teams. The Openkova + pixelmatch stack is MIT-licensed and runs entirely in GitHub Actions with no external service.

How does visual regression testing work in GitHub Actions?

Start your app, capture screenshots with @openkova/cli, diff against baselines with pixelmatch, and fail the job if the diff exceeds your threshold. Upload the diff PNG as an artifact. The full workflow YAML is in the setup guide.

Can I test pages behind a login or on a staging URL?

Yes. @openkova/cli uses your local Chromium and can screenshot any URL reachable from the CI runner — localhost, internal staging URLs, VPN-protected services. SaaS tools require a public tunnel to reach non-public URLs.

What is the difference between Percy and Chromatic?

Percy is framework-agnostic and starts at $299/month for teams. Chromatic is Storybook-specific — it only tests Storybook stories, not live pages — and starts at $149/month. Both require uploading screenshots to an external service. The self-hosted alternative works on any URL and requires no external service.