Migrating from Drupal 7 to 9

This post was written collaboratively by Dharizza Espinach Barahona of Montreal-based Drupal experts at Evolving Web, along with Eric Michalsen of Pantheon. 

*Note, the metaphorical island we're referring to here is the Drupal community's support of Drupal 7. But Pantheon won't let you drown! Pantheon will continue to support Drupal 7 sites, even after this version has reached end-of-life status. Find out more on Pantheon's support of Drupal 7, 8, and 9. 

Drupal 7 is a large, active, and vibrant island, rich in contributors and characters. And now you are getting kicked off. 

The years have clicked by quickly, and a LOT has changed, save for the fact that Drupal 7 still sits on that island. And that island is quickly sinking. Never mind you've been given a grace year (because of Covid). November 28, 2022, is coming at us. Fast.

But you are not alone. With the Herculean task set before you, moving many hundreds of Drupal 7 sites into Drupal 9, planning and patience will pay off. Those that went before you are laying down best practices, and new tools have been built to help you succeed. But no matter how big or small the project, the same processes apply.  

Review, plan, configure, migrate, then test.

Review

The first point, Review, should start as self-reflection and gather the strengths and weaknesses of your team to tackle the upcoming hurdles. 

• How familiar is your team with the Drupal 9 architecture?
• What is the level of competence with Object-Oriented Programing (OOP) in PHP
• Has the team worked with the Symfony framework?
• Is your front-end team ready to develop in Twig?

Answering these questions honestly will allow you to better understand the requirements of labor and time that will be required.

• How much customization is being moved from:
  • Custom modules
  • Panels and Theme layouts
• Your team's skills:
  • Applicable skills brought to the project
  • Learning while on-the-job training (OJT)

Once you have taken an assessment of your team's abilities, you can begin to review the differences and find the common ground between Drupal 7 and Drupal 9.

Image

As you can see from the quick summary above, many items will be familiar to you within Drupal 9. Items in the admin menu, with small changes and additions, along with the content structure, will allow your site and content managers to feel at home. 

Popular contributed modules have been brought into core with improvements. It is still Drupal, only now with Symfony, Composer, Twig, configuration management, Layout builder, Workspaces ... you get the idea. 

It is still Drupal, now only more so. 

Of course, there is much that is different. Some would say it’s a whole new beast.

There are enough new changes for content creators, editors, and site designers, to make everyone anxious. Drupal 9 allows for a new way of doing, managing, and laying out content, and as a part of your migration process, training should be factored in.

But not all is lost. As I mentioned earlier, tools have been developed to help facilitate your migration process:

• Migrate UI
• Migrate Status
• Drupal Migrate Upgrader

In finishing up the Review process, you should have a clear understanding of what modules must be ported over, if there is no current D9 upgrade, along with decisions on layout, branding, and user interface (UI). However, moving from one theming engine to another will not be a clean path. 

This may be a good opportunity to review your site as a whole and how you wish for it to look and work — for both your visitors and content providers. 

Drupal 9’s Layout Builder is a powerful new tool that enables your content creators the power to lay out their work in the way that best reflects their work. Drupal Workspaces is a new experimental Drupal core module that allows you to publish multiple content items in your production site at the same time in a controlled manner.

Plan

There are multiple points in the planning process that can be farmed out in the decision making process. Reviewing content and mapping source fields to destinations, deciding what to omit, and auditing external API sources, and how they will connect to your new system. And with these points there are tools and methodologies that help.

Module Assessment

Module assessment is nothing more than an extensive spreadsheet listing every asset in your Drupal 7 site and confirming its readiness in Drupal 9. Here’s an example of modules:

Image

This is an audit for all those responsible to be held accountable.

Data Transfer Table (DTT)

The Data Transfer Table is another spreadsheet. It is an audit of the data that is being transferred, from nodes to users, metatags to aliases. This is also a key point where the decision to not move some content would be rendered.

Management

The business structure of the migrations team starts with the management, overseeing the budget of time, resources, and scheduling, as well as managing the upstream development, design/user experience (UX), and user acceptance testing (UAT) teams. They must keep open communication with this group of stakeholders, in terms of setting expectations and freezing sites.

Under the Upstream Development Team was the DevOps, internal development, and Migration Toolkit Developers at Evolvingweb.

Configuration

This stage is where the building of the content structure in the Drupal 9 site happens, it is where we'll set the foundation of the migration, so that the content can be finally moved. Here, the site starts to take form and it includes creating configuration corresponding to the content types, block types, fields, user roles, menus, layouts and so on, all of this based on the results of the review and planning stages.

University of Waterloo Case

Based on the results of review and planning stages, University of Waterloo found an intersection of commonalities in their sites and upon this, the Custom Upstream was created. The content of the Custom Upstream includes modules, themes, and shared configuration between the sites — along with the possibility to have a specific configuration per site. This specific configuration may include what modules we need to enable on each site, at each time.

For building the Migration Toolkit, one Drupal 7 site was created, this "parent" site included all configuration shared between the sites (content types, fields, vocabularies, entity types, and so on). It was used as the source for developing the migration toolkit and made it easier to test.

University of Waterloo migrated their 900 sites with one Upstream, but if the need suits, many upstreams could be designed. 

All 900 sites were broken into Groups, which provided smaller, more easily handled tasks for the migration team, and a schedule to meet expectations of the site owners and maintainers. 

Custom Upstreams

Custom Upstreams act as a scaffolding for new sites, allowing developers to roll out sites with the same configuration and theme. Waterloo, using only one custom upstream, pushed out over 950 sites, with modules and configurations enabled on a group by group basis.

Pantheon’s Custom Upstreams were vital in the migration workflow for Waterloo, as shown below. Migration scripts could be tested locally before being committed to the organizational upstream, which in turn served as the parent for all sites.

Using Pantheon’s Terminus and Quicksilver, migrations could be managed with Jenkins, and deployment notifications sent to Slack, Trello, or perform drush commands.

Migrate and Testing

Migrate

During the migration phase, you will create the migration scripts needed for moving over all your content automatically. This includes content itself, mapping references, migrating redirects, aliases, multilingual content, and so on.

Ensure there is a content freeze, so that the final migration can happen. This final migration involves running all the migration scripts created and also doing the manual migrations for the items that need to be migrated manually (according to the Data Transfer Table created in the Planning stage).

It is recommended, when migrating hundreds of websites, to create one migration module that can be reused in all of the sites. For the Waterloo University, a Migration Toolkit was developed, it is basically a Drupal module that includes all the migration, source, process and destination plugins — to migrate all the content from the Drupal 7 sites into Drupal 9 structures.

The Migration Toolkit also includes a test suite where each of the plugins created are tested. This module is included in the Custom Upstream, so it is distributed to and used in all the 900+ Waterloo websites and can read specific configurations per site.

Testing

Testing is performed throughout the migration process, from Unit Testing in creation of the migration scripts, through to the UAT. UI (such as behat) and visual regression testing can also be put in place to help the UAT team in their job.

What is important is that issues are discovered early in the process; the build and migration times are expensive.

Migration and testing together 

(Nuts and gum, together at last!)

The migration and testing stages are not isolated islands, they are processes that need to happen together, hand-in-hand, during the whole project. When we get to this stage, we're talking about content migration and when doing the content migration for hundreds of sites, these stages are not sequential — they have to be tightly integrated. It is a cyclical, interactive process where we need to add a lot of emphasis in the testing.

In the case of the Waterloo University, with more than 900 websites, they rolled the migration in chunks of 5 to 15 sites, starting first with a pilot group that would cover 90% of the more complex items to migrate.

Each Group would begin the migration process by the creation of each Drupal 9 site based on the Custom Upstream, followed by an initial test using the Migration Toolkit test suite and the Group’s live data (from Drupal 7). If the test failed, it was sent back to the build kit to update, and thus updating the Upstream. 

With a successful migration test, the group would be notified of a content freeze on their production sites, and the migration process would be deployed in the new live site in Pantheon using their most recent data. 

This process included both the automated migrations provided by the Migration Toolkit and all those elements that required manual migration. Once the process was done, this would pass through UAT prior to releasing the Group back to its owners and the migrations team moving onto the next Group. With a failure, the Group would be given back ownership of their sites while the Testing and Build Kit was evaluated for fault.

Below is a representation of the migration workflow.

Image

If the UAT passed they would queue up for the next migration. If failed, that group would go back to the build kit.

Timing for the migration was dependent on complexity of the sites in the group. The more complex sites were done towards the beginning of the process, speeding up the migrations downstream. It was not uncommon for the time between group migrations could be days, a week, or the next day.

Real-world example of time and resources required by University of Waterloo

Below is a real-world timeline of both the time and resources required by the University of Waterloo during their migration from Drupal 7 to Drupal 9.

*Please note, at the time of this post being published, their migration is still ongoing. 

• October 2020: Evolving Web and University of Waterloo begin planning.
• January 2021: Evolving Web begin migration toolkit development
• April 2021: First version of toolkit released
• Summer 2021: Various versions of toolkit released as University of Waterloo begin migrations and front-end development
• October 2021: University of Waterloo still migrating

Key Takeaways

Hopefully the above has stressed the importance of planning, from mapping the fields for the migration process, to estimating the time it will take to actually run the migrations, and test. Some key points of the journey include:

• Plan for QA and identify test cases from the Planning Stage, along with who can properly QA those migrations.
• Keep your communication lines open and your internal customers informed. They know big changes are coming their way, but keeping them abreast of scheduling changes, workflows, and all, mixed with a touch of empathy for what they will be going through, and you will have a much better experience.
• Create one migration module to be used everywhere — to help standardize the process.
• Divide the job into small groups, and then set reasonable goals.
• Test everything, remember to look for scenarios to test from early stages, create both automated and manual test cases. This will prevent you from stepping over your feet.

You are getting kicked off the Drupal 7 island, it is only a matter of doing so with confidence and grace, or kicking and screaming. Planning, testing, and communication are the keys to this complicated process that, if planned thoroughly, will reward your team with a smooth migration process.