We’re Officially a Unicorn Pantheon Announces Series E Read the News

How Pantheon Tests Drupal 8 Updates

Drupal 8 Logo in Clouds

Automated testing is an important part of any active project workflow. When a new version of Drupal 8 comes out, we want to smoke test it on the platform to ensure that everything is working well before we expose it to our customers. We have confidence that Drupal itself is going to work well; we confirm this by running the unit tests that Drupal provides. Most importantly, we also need to ensure that nothing has changed to affect any of the integration points between Drupal and the Pantheon platform. In particular, the behaviors we need to test include:

  • Site install, to ensure that the Pantheon database is made available to the new site without interfering with the installation process.

  • Configuration import and export, to ensure that the Pantheon configuration sync directory is set up correctly.

  • Module installation, upgrade, and database updates, to ensure that the authorize.php and update.php scripts are working correctly with our nginx configuration.

This level of testing requires functional tests; to ensure that everything is working as it should on the platform, we use Behat. The thing that is most challenging about the repository we are testing, though, is that it serves as the upstream for a large number of Drupal sites.  Each one of these sites may want to do testing of their own; therefore, we do not want to add a .travis.yml or a circle.yml to our repository, as this would cause conflicts with the downstream version of the same file whenever we made changes to our tests—something we definitely want to avoid.

Fortunately, Circle CI has a feature that fits this need perfectly. Most of the directives that can be placed in a circle.yml file can now be filled in to the project settings in the Circle CI web interface.  The relevant settings are in the “Test Commands” section.  A screenshot of the dependency commands below shows what we do to initialize the Circle container for our tests:

Drops 8 Screenshot

In the pre-dependency commands, we need to set the timezone to use to avoid PHP warnings in our tests. When using a circle.yml file, we would usually specify the version of PHP that we wanted to use; then, we could adjust the php.ini file directly using a well-known path.  If we do not have a circle.yml file, then we must use whatever version of PHP Circle wants to give us, as the settings in the web interface have no affordance for this. We find the appropriate ini file like so:

echo "date.timezone = 'US/Central'"  >  
  /opt/circleci/php/$(php -r 'print PHP_VERSION;')/etc/conf.d/xdebug.ini

There are a couple of things to note about this line. This is just an ordinary bash expression, as is every field in these panels, so we can evaluate expressions and use output redirection here. The path to the PHP ini files contains the PHP version; we use a short bit of PHP to emit this so that we do not need to evaluate the output of php --version, or anything of that nature. Every line is implemented in a separate subshell, so it does not work to set a shell variable on one line, and then try to reference it on the next. If persistent variables are needed, they can be set in the environment variables section, as usual. Finally, we write our setting into the xdebug.ini file in order to turn off xdebug, which speeds up composer considerably. Xdebug is necessary if you plan on generating code coverage reports, though, so you can change the output redirection operator > (overwrite) to >> (append) if you want to keep xdebug enabled.

After that, we install a few projects with Composer, including:

  • hirak/prestissimo is installed to speed up composer operations.

  • consolidation/cgr is used to ensure that other tools we install do not encounter dependency conflicts.

  • pantheon-systems/terminus is installed so that we can do operations directly on a remote Pantheon Drupal 8 site.

  • drush/drush is also installed on Circle, just in case we want to install the Drupal site to do functional tests on Circle prior to running the tests against the Pantheon site. At the moment, though, Drush is not used, and would not necessarily need to be installed.

Note that we specify the full path to the tools we install, rather than adjusting the $PATH variable, because it seems that the $PATH variable is initialized by Circle sometime after the environment variables section is evaluated, making setting it up there ineffective.

The final step of the dependency commands is to clone the project that contains our test scripts.

We keep our tests in a separate tests repository in order to keep our repository-under-test as similar to the base Drupal 8 repository as possible. The tests repository is cloned into the folder specified by the TESTING_DIR environment variable. This is defined in the environment variables section of the Circle CI project settings. There are a number of variables necessary for the scripts in the ci-drops-8 project to work. They include:

  • ADMIN_PASSWORD: Used to set the password for the uid 1 user during site installation.

  • GIT_EMAIL: Used to configure the email address for the git user for commits we make.

  • TERMINUS_ENV: Defines the name of the Pantheon multidev environment that will be created to run this test. Set to ci-${CIRCLE_BUILD_NUM}

  • TERMINUS_SITE: Defines the remote Pantheon site that will be used to run this test.

  • TERMINUS_TOKEN: A Terminus OAuth token that has write access to the terminus site specified by TERMINUS_SITE.

  • TESTING_DIR: Points to a directory on Circle CI that will hold the local clone of our test repository. Set to /tmp/ci-drops-8.

In addition to these environment variables, it is also necessary to add an ssh key to Circle, to allow the terminus drush commands to run.  You should create a new ssh key for use only by this test script. Add the ssh public key to the site identified by TERMINUS_SITE (create a user specifically for this purpose, attach the public key in the SSH Keys section of the Account tab, and add the user as a team member of the site), and place the private key in the Circle “SSH permissions” settings page:

SSH Key Screenshot

Note that the domain drush.in is used for any ssh connection to any Pantheon site. Initially, this was only used for Drush commands; now, WP-CLI and Composer commands are also ran this way. 

The test scripts are executed via the commands in the Test Commands section of the Circle settings pages:

Drops 8 Screenshot

This part is pretty simple; it first runs the Drupal phpunit tests on Circle CI, and then runs the create-pantheon-multidev script, followed by the run-behat script, which tests against the Pantheon site on the prepared multidev environment.

The create-pantheon-multidev script prepares a new multidev environment to run our tests in; the new environment is named according to the TERMINUS_ENV environment variable. The TERMINUS_SITE variable indicates which site should be used to run the test; the TERMINUS_TOKEN variable must contain a machine token that has write access to the specified site. The script then uses the local copy of the drops-8 repository that Circle prepared for us, and adds the behat-drush-endpoint to it. See Better Behavior-Driven Development for Remote Servers for more information on what this component is for. Once that is done, the resulting repository is force-pushed to a new branch on the specified Pantheon site. The multidev environment is created after the branch is pushed to avoid the need for the platform to synchronize the code twice. The multidev sites created for testing purposes are left active after the tests complete. This provides an opportunity to inspect a site after a test run completes, or perhaps re-run some of the test steps manually after a failure to help diagnose a problem. A script, delete-old-multidevs, is run in the post-dependencies phase, just prior to the creation of the new multidev environment for the latest tests. This script deletes all of the test environments except for the newest two.

The run-behat script is just a couple of lines, as its only job is to fire off the Behat tool with the appropriate configuration file and path. Note that we define BEHAT_PARAMS in the script to supply dynamic parameters, so that we do not need to rewrite any portion of our configuration file.

In typical usage, most projects that use Behat to test Drupal sites will first install a fresh site to test against using Drush. In our case, however, we want to test the Drupal installer, so we will instead wipe the site with Terminus, and use Behat to step through the installer dialogs.  The Gherkin for one of our installer steps looks like this:

  Scenario: Profile selection
    Given I am on "/core/install.php?langcode=en"
    And I press "Save and continue"
    And I wait for the progress bar to finish
    Then I should see "Site name"

The interesting step here is the step “And I wait for the progress bar to finish”. There are a number of places in the Drupal user interface where an action will result in a progress dialog. Getting Behat past these steps is a simple matter of following the meta refresh tags in the response. A custom step function in our Features context provides this capability for us.

Sometimes you might want to use a secret value in a Behat test—to provide the password to the admin account in the installer, for example. To do this, we use a custom step function that will fill in a field from an environment variable. We use this step function to reference the previously-mentioned ADMIN_PASSWORD environment variable.  Also, there are a number of other useful step functions in the context for running terminus and drush commands during your tests; you might find these useful in your own test suites.

Finally, the ci-drops-8 project contains a circle.yml.dist file that shows the equivalent content for testing a drops-8-based using a circle.yml file rather than placing the test scripts in the Circle admin settings. While the techniques used in this post are somewhat atypical compared to most Drupal Behat setups, it would be possible to adapt them to work on a standalone Drupal site. Perhaps we’ll do that in a future blog post. In the meantime, the circle.yml.dist file can be used as a starting point to re-use these tests for your Drupal site. 

Topics Development, Drupal Planet, Drupal

Let’s get in touch