Automating WordPress Core and Plugin Updates With Visual Regression Testing

Introduction

scalewp.io is a community site about scaling WordPress with the content managed on GitHub. Pantheon sponsors and hosts the site. My team, Agency and Community Engineering, is responsible for maintaining the site, including WordPress core, themes, and plugin updates.

I volunteered to take this responsibility but found myself to be a bottleneck in getting WordPress updates out. For example, I often travel to attend WordCamps and am fully  engaged there rather than manually checking for WordPress updates.

The Issue

There were really two problems here. 1) WordPress updates, including security updates, are released sporadically and need to be checked daily, and 2) Applying those updates was time-consuming and took away from other, more high-value activities.

Automate WordPress updates on the site could alleviate both problems. WordPress has an Automatic Background Updates feature which can run some of these updates, but having them run automatically without a safety net was more risk than I wanted to allow.

The Solution

I decided to automate all WordPress updates—core, themes, and plugins. I wanted this to be a scripted process that could be run reliably on a daily basis, without sacrificing confidence or introducing significant risk.

Handling WordPress Update Automation With WP-CLI

Starting with a script that could run on a daily basis, I chose to use WP-CLI instead of Automatic Background Updates as it can be run within a script and can handle major WordPress updates, not just minor versions. Automatic Background Updates does not.

WP-CLI has commands to update WordPress core, plugins, and themes. Looking at plugins, I don't want to run updates unless they are available. WP-CLI's plugin list command takes a field option to limit the output. We can use this to get a list of plugins with updates available in a Bash script and only run updates if they are available.

Taking a look at the repository for this project, you can see the plugin update functionality on lines 43-60 of wordpress-at-scale-auto-update.sh. First, I use Terminus and WP-CLI to check if updates are available with terminus wp "plugin list --field=update" --format=bash. Then, if there are updates, they are applied with terminus wp "plugin update --all". Finally, the updates are committed with terminus site code commit --message="update WordPress plugins" --yes.

I do the same check and update for themes on lines 62-79. Core updates are handled with Terminus, Pantheon's CLI, using Pantheon upstream updates on lines 23-37. If you aren't on Pantheon, you can use WP-CLI here as well.

Mitigating Risk With a Dedicated Environment

Besides the lack of WordPress major release support one of the reasons I dislike Automatic Background Updates is that they typically run on the live website, which introduces a significant amount of risk. What happens if an update breaks functionality on the site? Or worse, crashes it entirely? It could be a user or client that notices before you do. Then you'll have to roll back the updates by restoring a backup. Doesn't sound like fun to me.

To avoid this situation I run all WordPress updates in a Pantheon Multidev environment. This allows me to ensure the updates go smoothly before deploying them to the live environment. If there is an issue I can resolve it before deploying.

Adding Confidence With Visual Regression Testing

The whole point of this was to automate WordPress updates being deployed, adding convenience and saving me time, without sacrificing confidence. Deploying the updates without some sort of quality control would have given me automation and convenience but my confidence in the site functioning would have been low.

I decided to add visual regression testing to my script, using BackStopJS. The visual regression tests check the URL of every page on the site and compare screenshots between the live environment and the Multidev environment where the updates have run. I set the tolerance to 0%, meaning if any visual difference exists between the live site and the site after updates, deployment will not happen. Manual intervention will be needed if the visual regression tests fail, but that small sacrifice in convenience is worth the confidence gained knowing updates won't cause a major issue with the site.

Deployment With Terminus

Once all the updates have been applied and the tests pass, the Multidev environment is merged and deployed using Terminus. This allows me to have a clear deployment log in the Pantheon Git history.

The script also creates a backup before each deployment. Even though the visual tests have passed, I want an extra layer of confidence in case other functionality has issues, and I need to roll back. You can see the Terminus deployment on lines 121-142.

Update Visibility

Since this script runs daily, I need a way to keep track of its success or failure. There are multiple places I can access the deployment history.

Every site on Pantheon has Git version control integrated. I can view deployment history in the Git logs.

The auto update script is running on CircleCI and there is a CircleCI log created every time it is run. For example, here is the log for run #150, in which updates were applied, tested, and deployed. If you're wondering how we run the script on CircleCI without pushing to the Git repository, it's by pinging the CircleCI API on a cron that runs daily.

Lastly, the deployment script posts a message to Slack whenever something happens, e.g. updates are deployed.

Image

Next Steps

Implement The Script

The final script is open source on GitHub. The README.md has Setup Instructions for getting this up and running on your WordPress site on Pantheon.

Add More Tests

The scalewp.io site is fairly basic—only displaying content without dynamic pieces, such as a contact form or shopping cart. If your site is more complex, I encourage you to add more tests tests to increase your confidence in the updates running smoothly. Behat is a great option for Behavior-Driven Development tests.

To get started with Behat, check out my co-worker Steve's blog on Testing WordPress Plugins on Pantheon with CircleCI.

Conclusion

I hope you've been able to see how you can make your life easier by automating WordPress updates—without sacrificing confidence thanks to proper testing.

If you use the script, leave a comment and let me know what you think! If you run into issues please post them to the GitHub repository issue queue.

Happy automation!