Unit testing is a helpful strategy for ensuring a high degree of code quality, but isn’t a panacea. Even when a WordPress plugin has 100% test coverage, this doesn’t necessarily mean it will work when you install it on your site.
Pantheon maintains several WordPress plugins used by customers and the broader WordPress community. Recently, we started on a project to help ensure we can definitively say “Yes, this WordPress plugin works on the Pantheon platform.” The key advantage is that this implementation automates the testing we might perform manually, and runs the test suite on every changeset. We run scripts that use Terminus, Pantheon’s command line interface, to install our plugins on vanilla WordPress sites on the platform. Then we run Behat tests to ensure the plugin works as expected, and WordPress works as expected with the plugin active.
It turned out much nicer than we expected. Read on for the full details.
How We’re Using Pantheon Multidev and Behat to Provide Integration Test Coverage
On a high level, here’s how we’re able to add integration tests alongside our existing unit tests:
In addition to the .travis.yml file that defines Travis CI’s test suite, we added a circle.yml file for the Pantheon integration tests.
With every CircleCI build (every commit pushed to a pull request, every nightly run on the master branch) a new Pantheon Multidev environment is created using Terminus.
CircleCI runs Behat tests against that environment.
CircleCI cleans up after itself by deleting the Multidev environment.
Behat shines brightest when it is used as part of Behavior Driven Development where scenarios are written in the language of the users. Most of our tests fall short of that ideal, because we wanted to keep things as simple as possible, and instead rely on Behat’s (Mostly Mink’s) ability to script browser interaction. True Behavior Driven Development with Behat usually involves writing custom contexts that add complexity which allow Behat’s .feature files to be clear and direct.
For instance, a scenario to verify the homepage loads as expected looks like this:
Scenario: Verify that WordPress loads with the plugin active
Given I am on the homepage
Then I should see "Hello World"
In this way, these Behat assertions act as smoke tests verifying functionality on Pantheon.
Why Use CircleCI and Travis CI Together?
Keep in mind, we’re still using Travis CI on these WordPress plugins to run the full suite of existing PHPUnit tests. Travis CI is popular in the open source world for the ease with which tests can be repeated against a matrix of conditions like PHP versions. This is great for fully testing a WordPress plugin.
By splitting out these two types of testing onto different services—Travis and Circle—we get result more quickly. We also like how easy it is to get inside CircleCI containers while a test runs using SSH. This feature is a lifesaver when debugging the tests themselves.
CircleCI also has a freemium model that makes it more attractive to many of Pantheon’s customers who want to test private repos of full Drupal or WordPress sites. CircleCI provides one free container, even for private repos. This allows teams of developers unsure about continuous integration to experiment freely and use our tests as examples.
Integration Testing: It’s Not Just for WordPress Plugins
We are using similar techniques beyond WordPress plugins. Every night we use CircleCI to run WordPress core’s own test suite against Pantheon with the latest commits made on wordpress.org. We have a dedicated repository just for that task. (Thanks to Aaron Jorbin for the encouragement!) A separate repository is used for Behat testing of our own copy WordPress Core upstream. A forthcoming Drupal 8 module for Pantheon Solr integration uses the same Behat-based testing methodology as the WordPress plugins. And in my last blog post, I mentioned a repo that just uses CircleCI to verify Drupal 7 to Drupal 8 data migrations working on Pantheon.
Tradeoffs and Rough Edges
As you saw above, the setup and execution of the CircleCI tests is done through a series of Bash scripts. Executing our automation in this manner allows us to experiment quickly and independently with each plugin. The Bash scripts should also be very easy to follow for any developer interested in seeing exactly how one of our plugins can be installed and configured. The downside, of course, is that each of these repos duplicates code in their scripts. The script names could be slightly different, tasks could be done in varying orders, and maintenance will become increasingly difficult as our repositories and tests grow in number and complexity.
To address those problems we have a number of options. We could consider an abstraction layer designed for systematizing the interaction of CMS extension and a hosted CI service as drupal_ti does. We could use the Behat contexts in Terminus to abstract the interaction with Pantheon (separately from the way Behat can abstract interaction with a website). We could even use a task runner like Robo, or a devops abstraction like Ansible. We certainly welcome input and feedback on the best strategies.
Beyond answering “Does this plugin work on Pantheon,” the process of writing these tests has helped us in a number of ways. We have found places to enhance the plugins themselves, like the addition of new wp-cli commands that streamline plugin set up. We have generated ideas for improving Pantheon itself and our developer tools. And perhaps most beneficial for me in my job is that it keeps me in the mindset of a Pantheon customer relying on Pantheon’s tooling to test and run the code I write.
- Why Pantheon for WordPress [DATASHEET]
- Why WordPress? Six Reasons Why WordPress Is Ready for the Enterprise
- What Top Developers Think About When Choosing a WordPress Hosting Provider