Visual Regression Testing Without Percy: A Free Setup

What Percy does (and what it costs)

Percy captures screenshots of your application on every pull request and compares them against an approved baseline. Reviewers see a side-by-side diff in the Percy web UI and accept or reject changes. When accepted, the baseline advances.

It's a well-designed product. The constraints are architectural: screenshots are processed on BrowserStack servers, there is no self-hosting option, and you are coupled to Percy's SDKs and review workflow. Percy does offer a free tier (5,000 screenshots/month), but usage above that scales in cost, and the SaaS dependency remains regardless of plan.

The two-component replacement

Visual regression testing has two steps:

1. Capture — take a screenshot of the page
2. Compare — diff it against the baseline

Percy bundles both. Open-source tools handle both individually:

• @openkova/cli (kova) — headless Chromium screenshot capture; MIT-licensed, no API key
• pixelmatch — perceptual pixel-level image comparison in pure JavaScript; MIT-licensed

Installing the tools

# Screenshot CLI
npm install -g @openkova/cli

# Comparison library
npm install --save-dev pixelmatch pngjs

@openkova/cli requires Chrome or Chromium. On macOS with Google Chrome installed, it works with zero additional setup. On Linux, install chromium-browser:

sudo apt-get install -y chromium-browser

Step 1: capture a baseline

On the first run, you create the baseline screenshots that future runs will compare against. Store them in your repository (or S3 for larger sets):

mkdir -p baselines

# Screenshot your app's key pages
kova screenshot https://localhost:3000 \
  --full-page --output baselines/home.png

kova screenshot https://localhost:3000/about \
  --full-page --output baselines/about.png

kova screenshot https://localhost:3000/pricing \
  --full-page --output baselines/pricing.png

# Commit the baselines
git add baselines/
git commit -m "chore: add visual regression baselines"

Step 2: write the comparison script

// scripts/visual-diff.mjs
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
import fs from 'fs';
import path from 'path';

const PAGES = ['home', 'about', 'pricing'];
const THRESHOLD = 0.1;   // perceptual tolerance (0.0 = exact, 1.0 = very tolerant)
const MAX_CHANGED = 100; // fail if more than this many pixels differ

let passed = true;

for (const page of PAGES) {
  const baselinePath  = `baselines/${page}.png`;
  const currentPath   = `screenshots/${page}.png`;
  const diffPath      = `diffs/${page}.png`;

  if (!fs.existsSync(baselinePath)) {
    console.warn(`[skip] No baseline for ${page} — copy screenshots/${page}.png to baselines/`);
    continue;
  }

  const baseline = PNG.sync.read(fs.readFileSync(baselinePath));
  const current  = PNG.sync.read(fs.readFileSync(currentPath));

  if (baseline.width !== current.width || baseline.height !== current.height) {
    console.error(`[fail] ${page}: size changed (${baseline.width}×${baseline.height} → ${current.width}×${current.height})`);
    passed = false;
    continue;
  }

  const { width, height } = baseline;
  const diff = new PNG({ width, height });

  const changed = pixelmatch(
    baseline.data, current.data, diff.data,
    width, height,
    { threshold: THRESHOLD }
  );

  const pct = (changed / (width * height) * 100).toFixed(2);

  if (changed > MAX_CHANGED) {
    fs.mkdirSync('diffs', { recursive: true });
    fs.writeFileSync(diffPath, PNG.sync.write(diff));
    console.error(`[fail] ${page}: ${changed} pixels changed (${pct}%) — diff saved to ${diffPath}`);
    passed = false;
  } else {
    console.log(`[pass] ${page}: ${changed} pixels changed (${pct}%)`);
  }
}

if (!passed) process.exit(1);

Step 3: GitHub Actions workflow

# .github/workflows/visual-regression.yml
name: Visual Regression
on: [pull_request]

jobs:
  visual-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Install Chromium
        run: sudo apt-get install -y chromium-browser

      - name: Install kova
        run: npm install -g @openkova/cli

      # Start your app (adjust to your stack)
      - name: Start app
        run: npm run build && npm run start &
        env:
          NODE_ENV: production

      - name: Wait for app
        run: npx wait-on http://localhost:3000 --timeout 30000

      - name: Capture screenshots
        run: |
          mkdir -p screenshots
          kova screenshot http://localhost:3000       --full-page --output screenshots/home.png
          kova screenshot http://localhost:3000/about --full-page --output screenshots/about.png
          kova screenshot http://localhost:3000/pricing --full-page --output screenshots/pricing.png
        env:
          CHROMIUM_PATH: /usr/bin/chromium-browser

      - name: Compare against baselines
        run: node scripts/visual-diff.mjs

      - name: Upload diffs on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: visual-regression-diffs
          path: diffs/

When a regression is detected, the diff image is uploaded as a GitHub artifact. Open it from the Actions run to see exactly which pixels changed.

Updating the baseline

When you intentionally change the UI — a redesign, a new component — you need to update the baseline. The pattern is:

# Capture the new screenshots locally
kova screenshot http://localhost:3000 --full-page --output baselines/home.png

# Commit the updated baseline
git add baselines/home.png
git commit -m "chore: update visual baseline — new hero section"

Percy automates this via its review UI. In this setup, you do it manually with agit commit. For small teams, the explicit commit is actually useful: it creates a clear audit trail of intentional visual changes.

Handling dynamic content

Timestamps, user-specific data, and live prices change between runs and cause false positives. The most reliable solution is to run visual regression tests against a staging environment with static, seeded data — so all dynamic values are deterministic between runs. For HTML snippet workflows, replace dynamic values with static fixtures before the screenshot step.

Faster comparisons with odiff

For large images (2000px+ tall full-page screenshots), odiff is 40–100× faster than pixelmatch because it's written in Rust and compiled to a native binary:

# Install odiff
npm install -g odiff-bin

# Compare two images
odiff baselines/home.png screenshots/home.png diffs/home.png

# Exit code 0 = identical, 1 = images differ, 2 = error
echo "Exit: $?"

odiff does not have pixelmatch's perceptual threshold option, so it is more sensitive to minor anti-aliasing differences. For most web screenshots, pixelmatch with threshold: 0.1 produces fewer false positives.

What you give up vs Percy

Percy provides a managed review UI where non-developers can approve or reject visual changes with a click. This setup produces diff PNGs as CI artifacts — useful for developers, but less accessible for designers or PMs reviewing changes.

If that review UX matters to your team, a lightweight alternative is reg-suit — an open-source visual regression toolkit that generates an HTML report and can post a link to it in your PR comments. It composes with @openkova/cli for the capture step.

Frequently asked questions

How do I set up visual regression testing for free?

Install @openkova/cli for capture and pixelmatch for comparison. Store baseline PNGs in git, capture current screenshots in CI, and diff them. Fail the build and upload diff images as artifacts when regressions are found.

What is the difference between pixelmatch and odiff?

pixelmatch is a JavaScript library with a configurable perceptual color threshold — useful for ignoring minor anti-aliasing differences. odiff is written in Rust and is 40–100× faster for large images, but less configurable. Both are MIT-licensed.

How do I prevent false positives from dynamic content?

The most reliable approach is a staging environment with seeded, static data — timestamps and user-specific values are predictable between runs. For HTML snippet workflows, replace dynamic values with static fixtures before passing the HTML to the capture step.

Can I store baselines in git?

Yes. A typical full-page PNG is 100–400 KB. For 10–20 pages, storing baselines directly in git is fine. For hundreds of pages, use git LFS or an S3 bucket and pull them in your CI step.

Also read: Screenshot Any Webpage in CI/CD with Zero Config — or compare with Percy alternatives for visual regression testing.