Continuous Integration with Drupal 7 to Drupal 8/Drupal 9 Migrations

Editor’s Note: This blog post and conference presentation was written for teams looking to migrate to Drupal 8. The ideas here are just as applicable to teams now looking to migrate to Drupal 9. The big deal about Drupal 9 is that it is not a significant architectural change, compared its previous version the way Drupal 8 was a significant architectural change to Drupal 7.

I joined Pantheon in September of 2015, two months before the release of Drupal 8.0.0. I had spent many years building client sites in Drupal 5, 6 and 7. Now I was starting a job where I would talk a lot about building Drupal 8 sites on Pantheon. To stay sharp, I felt like I had to actually build a Drupal 8 site on Pantheon. Moreover, I wanted to have high confidence that I was doing it right

So I decided to take one of my personal Drupal 7 sites and rebuild it in Drupal 8. With each new push of Drupal 8 code, I would automatically run a data migration from 7 to 8 and test the result. For more details about this project, you can watch my recent DrupalCon Seattle presentation, 1,001 CI Builds: The story of Drupal 7 to 8 migration.
 

To deliver on this idea, I pushed my Drupal 8 code to a public repo on GitHub and had CircleCI execute a workflow for every push to a pull request. The CI process I used evolved as I built out the Drupal 8 site. By the time I was done, I had six separate CircleCI "jobs".

We can simplify even further and group those six jobs into three groups:

1. Get a functional Drupal 8 site running. That was the responsibility of the "build" and "merge" jobs. These jobs did things like install dependencies via Composer and push to a Pantheon multidev environment.

2. Migrate all the content from Drupal 7 to Drupal 8 via "drush migrate-import --all" command.

3. Verify that the Drupal 8 site is correct. That was the responsibility of the behavioral testing job, the Backstop visual regression testing job, and the migration check job.

For the rest of this blog post, I will focus more on the third group: verifying that the Drupal 8 site is correct. For more information on CI with Drupal 8 and pull requests, check out this blog post from 2017. If you want more on the nuts and bolts of configuring and writing migrations, there is a recent and thorough post from Juampy NR of Lullabot.

How Do I Know The Expected Behavior is Working?

Three of the jobs in my workflow ran tests to verify that the behavior of the site was correct. The first of those was the "behavioral" job that ran Behat scenarios. Behat is a behavior-driven development tool that allows for a team, even a team of one, to start with a plain English description of the expected behavior of the site. 

My site tracked performances in theaters, and I wanted those event listings to automatically be removed from the homepage once the performance date had passed. The Behat scenario looked like this:

 @api
  Scenario: Events in the future
    Given I log in as a content_administrator
    When I create an Event with a date in the future
    Then that event appears on the events page in the upcoming events section
    And that event appears on the homepage

 @api
  Scenario: Events in the past
    Given I log in as a content_administrator
    When I create an Event with a date in the past
    Then that event does not appear on the homepage
    And that event appears on the events page in the past events section

By writing those expectations first, I could then configure content types and Views that fulfilled the expected behavior. Then this continuous integration job would run all of the Behat scenarios on fresh Drupal 8 sites with every pull request to verify that no behavior regressed.

How Do I Know I Migrated All of the Data?

Behat is most helpful when the scenarios are written first as it increases clarity about what you are doing and provides the regression checks in perpetuity.

Sometimes Behat is still worthwhile to use just for the regression part. That is what I did to verify the data migrations continued functioning correctly. Working on a Drupal data migration generally means having the old (Drupal 7) site open in one browser window and the new (Drupal 8) site open in another window, while also using a text editor tweak configuration. After each tweak, I would run a partial migration and check the Drupal 8 window to manually verify if the data had moved over as I expected. That one-time task of personally checking if the Drupal 8 site has the right representative data can be automated and then repeated as a regression test. 

For instance, here is the Behat scenario I used to check that some taxonomy terms continued to migrate:

  Given I log in as an administrator
  When I visit "admin/structure/taxonomy/manage/member_designations/overview"
    Then I should see the link "Current Member"
    Then I should see the link "Emeritus"
    Then I should see the link "Founding Member"
    Then I should see the link "Honorary member"
    Then I should see the link "Viewable bio page"

I share this scenario here because this particular test proved the general idea had some merit. Shortly after I introduced this test, it failed! The test failed because of a genuine bug between Drupal 8.1.1 and 8.1.2. The automated test helped me catch the bug much faster than if I were just manually checking for missing taxonomy terms. This post on my personal blog has more details on the bug.

How Do I Know I Am Done?

As I undertook this side project, I needed to limit the scope if I ever wanted to finish it. One way I decided to do that for myself was to rebuild the Drupal 8 site to look and function exactly like the Drupal 7 site, down to the pixel, at least on the public-facing theme. I kept the same CSS file between Drupal 7 and Drupal 8 and modified a few selectors. My Drupal 7 site already had a hacky Twig implementation that meant I could even carry over some of the same Twig files.

These constraints allowed me to take screenshots of the real, live Drupal 7 site and compare them to screenshots of the in-progress Drupal 8 site. Early in the project, when the Drupal 8 site was barely started, all of the screenshots were very different. As I completed content types and important Views, the screenshots got more and more similar. 

Once all 25 representative pages were visually identical between Drupal 7 and Drupal 8, I knew I was done. For more information on using visual regression tests more generally, see this post from David Needham.

Starting Your Own Migration

If you are interested in building a Drupal 8 site in a similar way, I recommend getting started with our example repo for combining GitHub, Composer, and CircleCI. You can spin up a new project with a single command using our command line tool. When it comes time to set up the migration, I have a reference repo that configures a migration from scratch and runs that migration. If you find yourself getting stuck, check out our community forum site where you can ask any question about developing on Pantheon.