Running Drupal 8 Data Migrations on Pantheon Through Drush

Editor's Note: The structure of Terminus commands have changed since this post was written. The example commands have been updated to reflect the current structure of terminus command:subcommand <site>.<env>. Also when this post was written, Drupal 8 was the newest version of Drupal and was a large technical departure from Drupal 7 (hence the need for robust migration processes described in this post). Drupal 9 and 10 maintain the same technical structure as Drupal 8 and the techniques described in this post should continue to be effective for migrations from Drupal 7 to Drupal 9, 10, or higher. 

Earlier this month we had a blog post from Ryan Weal walking through some of the concepts behind migrating to Drupal 8 from earlier versions of Drupal as well as how to do a migration using the browser-based user interface. In this post, I'll walk through how to accomplish a similar result using command line tools like Terminus and Drush. This workflow will allow you to repeatedly run migrations as part of a continuous integration process. For many projects there is a need to run Drupal 7 to Drupal 8 migrations nightly—or on some other cron interval—so that the Drupal 8 site being built has the latest data.

So the intended use case for this workflow is a project in which a Drupal 8 site is being built over the course of some weeks or months while a Drupal 6 or Drupal 7 site remains live and constantly has fresh content being added.

The Idea

While this post will walk through the details of how to execute this workflow on Pantheon, the basic idea is the same regardless of where you host your sites. You can find more generic documentation here on Drupal.org. In essence, we need to:

1. Install helper contrib modules: Migrate Tools, Migrate Plus, Migrate Upgrade.

2. Define the connection details for migration source database (the username, password, etc. for the Drupal 6 or 7 database)

3. Define the data migrations for Drupal 8 to understand  

4. Run the migration (repeatedly if we choose to)

Defining Your Database Connection

Drupal sites use settings.php—and files included therein like setting.local.php—to define the connection to their primary database. For sites running on Pantheon, we take care of injecting database credentials through settings.pantheon.php. We do this so that developers do not have to worry about database connections as they move between Dev, Test, and Live environments. Connecting to an additional database—the Drupal 6 or 7 migration source–does take a bit of additional work. And just like the primary database you have to keep in mind that:

• You don’t want to commit your database password to version control.

• You want the connection details to vary based on environment (whether the site is running on your local machine or on Pantheon)

To fulfill these requirements, we currently recommend doing the following steps.

In your settings.php file, include settings.migrate-on-pantheon.php with the following code:

# When on Pantheon, connect to a D7 database.
$migrate_settings = __DIR__ . "/settings.migrate-on-pantheon.php";
if (file_exists($migrate_settings) && isset($_ENV['PANTHEON_ENVIRONMENT'])) {
 include $migrate_settings;
}

And the setttings.migrate-on-pantheon.php can include:

$secretsFile = $_SERVER['HOME'] . '/files/private/secrets.json';
if (file_exists($secretsFile)) {
 $secrets = json_decode(file_get_contents($secretsFile), 1);
}
if (!empty($secrets['migrate_source_db__url'])) {
 $parsed_url = parse_url($secrets['migrate_source_db__url']);
 if (!empty($parsed_url['port']) && !empty($parsed_url['host']) && !empty($parsed_url['pass'])) {
 $databases['drupal_7']['default'] = array (
   'database' => 'pantheon',
   'username' => 'pantheon',
   'password' => $parsed_url['pass'],
   'host' => $parsed_url['host'],
   'port' => $parsed_url['port'],
   'driver' => 'mysql',
   'prefix' => '',
   'collation' => 'utf8mb4_general_ci',
 );
 }
}

This file is reading from a secrets.json file created with our Terminus secrets plugin. Essentially what’s happening here is that we need a way to ask for the migration database connection information. If we wanted to be even more secure about the way this string is handled, we could use Lockr, but for now this secrets.json file is a fine starting point.

Now, to populate secrets.json from your local machine, you can use the Terminus secrets plugin. Run this command after setting the names of your Drupal 8 and Drupal 7 (or Drupal 6) sites and Multidev environments as global variables:

export D7_MYSQL_URL=$(terminus site connection-info --site=$PANTHEON_D7_SITE --env=dev --field=mysql_url)

terminus secrets:set $PANTHEON_SITE.$PANTHEON_BRANCH migrate_source_db__url $D7_MYSQL_URL

Notice here that we are reading from the dev environment of the Drupal 7 site. That is a question of preference. You might want to read from the live Drupal 7 database to ensure you always have the latest content. I personally like to use Multidev on the Drupal 7 site for the sole purpose of controlling exactly when the migration source database gets updated.

Configuring the Migration

All this work to set up the database connection only matters if you really want to keep the source database password out of version control (which is something you should want, in my opinion). Now to actually read from that source database, we will use a Drush command from the Migrate Update module (you will also need migrate tools and migrate plus).

export D7_MYSQL_URL=$(terminus site connection-info --site=$PANTHEON_D7_SITE --env=dev --field=mysql_url)

terminus drush $PANTHEON_SITE.$PANTHEON_BRANCH "migrate-upgrade --legacy-db-url=$D7_MYSQL_URL"

You might notice that in running this command we passed the source database credentials again, even though they are available in that settings.migrate-on-pantheon.php file we just set up. As it stands now, Migrate Update assumes that the source credentials are passed into the Drush command. The credentials are then saved into the resulting configuration option. I have opened an issue on the Migrate Upgrade project drupal.org to simplify this command to the following:

terminus drush $PANTHEON_SITE.$PANTHEON_BRANCH "migrate-upgrade --legacy-db-key=drupal_7 --configure-only"

With the migrations configured, we can export our Drupal 8 site configuration:

# Make sure your site is in sftp mode so that .yml configuration files can be exported

terminus connection:set $PANTHEON_SITE.$PANTHEON_BRANCH sftp

terminus drush $PANTHEON_SITE.$PANTHEON_BRANCH "config-export -y"

Before committing the exported YML, open up the file migrate_plus.migration_group.migrate_drupal_7.yml via SFTP and delete the database connection credentials. If you use my patch from Drupal.org you won’t need to do this step. Commit the YML files. You now have migrations defined that you can run through

terminus drush $PANTHEON_SITE.$PANTHEON_BRANCH "migrate-status"

terminus drush $PANTHEON_SITE.$PANTHEON_BRANCH "migrate-import --all"

At this point you might want to include the running of migrations in a continuous integration workflow. As a reference, and to validate that the suggestions I make here work, I’ve created a GitHub repo with CircleCI integration that does all of the steps above with each build. If you are doing a real Drupal 6 or 7 migration to 8 on Pantheon, let us know if and how your process varies. Rebuilding a site is difficult enough. We hope the data migration can be one of the easier steps.