When I was a kid I loved doing the “spot the difference” games in Highlights magazine. Anytime we visited the doctor or dentist, I’d go through multiple times to see how fast I could see what changed. I got pretty good at that over the years.
Today I play my own adult version of spot the difference every time I work on a website. I’m constantly doing visual QA between Dev and Live to make sure nothing changed unexpectedly. It takes a lot of time and I still make mistakes.
These inconsistencies are a form of visual regression. “Regression” is defined as “a return to a former or less developed state.” Therefore, “visual regression” on a website refers to visual elements changing in an undesirable way, or simply in a way you don’t expect.
The good news is that computers are way better at spotting differences than I am.
Anyone can write a test with BackstopJS
BackstopJS takes screenshots of two versions of the same website and compares them pixel by pixel. It then generates a report (or returns a pass/fail on the CLI) so that you can quickly see if that change you made broke anything.
I don’t write a lot of code, but even I can write tests. Once you install BackstopJS, a few simple commands go a long way!
If you don’t already have BackstopJS setup,
backstop init will create a default configuration file and project scaffolding in your current working directory.
Once you’ve done this, you’ll want to edit the
backstop.json file and enter the details pertinent to your project. In particular, you’ll want to visit the
scenarios section and edit:
url: what you’re checking (usually local or development environment)
referenceUrl: your live production site
Go back and see what the other lines in
backstop.json do later on. BackstopJS is incredibly powerful and feature-rich, but it’s okay to use it only as a pixel comparison tool. The readme on GitHub provides a lot of useful information when you’re ready to dig deeper.
Do you know what’s frustrating? Content creators and editors keep changing the production site when you’re trying to make everything perfect. As a result, your development site gets out of sync and visual regression tests fail.
backstop reference command sets the baseline that you’re testing against and makes sure that you’re not comparing against a moving target. It locks in the current version of production by generating screenshots for each
referenceUrl defined in scenarios in backstop.json. You should only need to do this when you’re ready to sync up your development environment with production.
Are you ready to run your first test? Just run
backstop test and you’ll start seeing text stream across your screen like you’re a hacker on TV. When it’s done, your default browser should pop up with a report generated by BackstopJS.
It’s that easy. Anytime you make a change to your development site, you just run
backstop test again.
Opportunities for visual regression testing
Have you ever changed something accidentally? I find this happens a lot in the following situations, and testing for visual change can help you make sure you don’t commit a change you don’t expect.
Routine security updates and patches
Updates are essential for the good health of your website. It’s usually safe to assume that when you do updates nothing should change visually. But this isn’t always true, and it’s very difficult to tell how your unique weave of modules or plugins interact.
Theming and site building
Unlike updates, when you’re actively making changes to CSS or configuration, you do expect to see a change. But how often have you made a quick fix only to realize later that it unexpectedly changed the site somewhere else?
> It'll just be a quick fix
Narrator: It wasn't.
— I Am Devloper (@iamdevloper) August 9, 2018
BackstopJS isn’t foolproof. There are actually a lot of false negatives at first. You can usually reconcile with the settings we didn’t mention in
backstop.json, but really just start with the basics. If the test is important enough to stick around, you’ll continue to make it more accurate.
At the end of the day, Backstop.JS is still faster than asking me to look at your site.
You may also like:
[VIDEO] Visual Regression Testing with BackstopJS (90 minutes from Drupal Europe 2016)