Using BackstopJS to Spot the Difference

Two nearly identical images of a house with the sun and clouds

When I was a kid I loved doing the “spot the difference” games in Highlights magazine. Anytime we visited the doctor or dentist, I’d go through multiple times to see how fast I could see what changed. I got pretty good at that over the years.

A gif showing the dev environment of a site on Pantheon next to the test environment

Today I play my own adult version of spot the difference every time I work on a website. I’m constantly doing visual QA between Dev and Live to make sure nothing changed unexpectedly. It takes a lot of time and I still make mistakes.

These inconsistencies are a form of visual regression. “Regression” is defined as “a return to a former or less developed state.” Therefore, “visual regression” on a website refers to visual elements changing in an undesirable way, or simply in a way you don’t expect.

The good news is that computers are way better at spotting differences than I am.

Anyone can write a test with BackstopJS

Screenshots taken by BackstopJS of two versions of the same website

BackstopJS takes screenshots of two versions of the same website and compares them pixel by pixel. It then generates a report (or returns a pass/fail on the CLI) so that you can quickly see if that change you made broke anything.

I don’t write a lot of code, but even I can write tests. Once you install BackstopJS, a few simple commands go a long way!

backstop init

If you don’t already have BackstopJS setup, backstop init will create a default configuration file and project scaffolding in your current working directory.

Once you’ve done this, you’ll want to edit the backstop.json file and enter the details pertinent to your project. In particular, you’ll want to visit the scenarios section and edit:

  • url: what you’re checking (usually local or development environment)
  • referenceUrl: your live production site

Go back and see what the other lines in backstop.json do later on. BackstopJS is incredibly powerful and feature-rich, but it’s okay to use it only as a pixel comparison tool. The readme on GitHub provides a lot of useful information when you’re ready to dig deeper.

backstop reference

Do you know what’s frustrating? Content creators and editors keep changing the production site when you’re trying to make everything perfect. As a result, your development site gets out of sync and visual regression tests fail.

The backstop reference command sets the baseline that you’re testing against and makes sure that you’re not comparing against a moving target. It locks in the current version of production by generating screenshots for each referenceUrl defined in scenarios in backstop.json. You should only need to do this when you’re ready to sync up your development environment with production.

backstop test

Are you ready to run your first test? Just run backstop test and you’ll start seeing text stream across your screen like you’re a hacker on TV. When it’s done, your default browser should pop up with a report generated by BackstopJS.

A gif showing a report generated by BackstopJS

It’s that easy. Anytime you make a change to your development site, you just run backstop test again.

Opportunities for visual regression testing

Have you ever changed something accidentally? I find this happens a lot in the following situations, and testing for visual change can help you make sure you don’t commit a change you don’t expect.

Routine security updates and patches

Updates are essential for the good health of your website. It’s usually safe to assume that when you do updates nothing should change visually. But this isn’t always true, and it’s very difficult to tell how your unique weave of modules or plugins interact.

Theming and site building

Unlike updates, when you’re actively making changes to CSS or configuration, you do expect to see a change. But how often have you made a quick fix only to realize later that it unexpectedly changed the site somewhere else?

BackstopJS isn’t foolproof. There are actually a lot of false negatives at first. You can usually reconcile with the settings we didn’t mention in backstop.json, but really just start with the basics. If the test is important enough to stick around, you’ll continue to make it more accurate.

At the end of the day, Backstop.JS is still faster than asking me to look at your site.

Topics Development, Drupal, Testing & Optimization, Training and Education, Website Technology, WordPress