Hot Fix Hot Take

Even the best development shops do it once in awhile.

The client’s complex site has been launched. The client is happy. New features are being built on a predictable sprint schedule. Feature branches are being merged to master. And then that urgent ticket arrives. A small change must be deployed to the live site and it must be deployed NOW. It’s Friday afternoon. It’s time for a hotfix.

The request could be any number of urgent needs:

• A drupalgeddon-level security hole.
• A JavaScript error that was not caught previously.
• CSS that looks broken on the client’s CEO’s model of Blackberry.

Developers will debate endlessly what types of changes warrant a hotfix, or even what a hotfix is. For the purposes of discussing hotfixes on Pantheon I will define a hotfix as such: “A hotfix is a change to the codebase that is deployed to the live environment through a process abnormal for the given website.”

For most sites on Pantheon changes are deployed in the following way through our dashboard:

• Changes are made to the master branch (either by directly committing to master or by merging feature branches, perhaps from a Multidev environment).
• The development environment reflects changes pushed to the master branch.
• At any time the team can move code from the development environment to the test environment using the dashboard. Under the hood, a git tag is made any time the movement of code from development to test happens. The git tag will be named something like “pantheon_test_123”
• The team moves code from test to live. A git tag is again made to coincide this this event; something like “pantheon_live_100”

Our documentation on hotfixing describes how developers can push tags directly in git which will trigger deployments to test and live. The naming convention will be very familiar to Drupal developers accustomed to writing update functions. If the most recent tag for test is “pantheon_test_123” then pushing a tag called “pantheon_test_124” will deploy that tag to the test environment.

Put another way, the Pantheon dashboard takes an opinionated stance on the branch workflow that generates the tags that trigger deployment. Hotfixing on Pantheon means using an alternate branch workflow to generate new tags. Those tags likely will not include recent master branch commits that unnecessary to the hotfix.

The Nitty Gritty

So let’s imagine that the most recent tag for live is “pantheon_live_2.” For sites with lots of tags you can easily filter down the the highest one with:

git tag | grep pantheon_live_ | sort -k1.15n | tail -1

A small hotfix needs to go live but the master branch has commits that cannot go live yet. Start by checking out that most recent tag.

git checkout pantheon_live_2

Note: checking out 'pantheon_live_2'.

You are in 'detached HEAD' state. You can look around, make experimental

changes and commit them, and you can discard any commits you make in this

state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may

do so (now or later) by using -b with the checkout command again. Example:

Make a new branch to work on your hotfix.

git checkout -b hotfix

Depending on the scope of the work you may need to push to a Multidev environment for testing and iteration. If the fix is simple or time is urgent simply commit your change

git commit -m 'The super serious hot fix commit'

And make a tag for the test environment.

git tag  -a  pantheon_test_5   -m  'Deploying the custom.module hot fix to the test environment.'

When you push the tags to Pantheon you will see in the dashboard that a deployment has been triggered in the test environment.

git push --tags

Counting objects: 9, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (8/8), done.

Writing objects: 100% (9/9), 952 bytes | 0 bytes/s, done.

Total 9 (delta 3), reused 0 (delta 0)

remote: 

remote: PANTHEON NOTICE:

remote: 

remote: The creation of tag "pantheon_test_5" has triggered a deployment of code on test.

remote: 

To ssh://codeserver.dev.ce2bbe07-8e4a-4779-9bb5-3e2fe88d7fc7@codeserver.dev...

Once the deployment is complete the test environment will not contain the master branch’s code.

So the dashboard will ask you if you want to bring in commits from the master branch.

But we’re doing a hot fix! We don’t want the master branch. So proceed with testing the hotfix tag in the Test environment. If the change is not satisfactory you can continue making and pushing tags that begin with “pantheon_test_”. If you need to do more debugging you can set up a multidev environment for your hotfix branch and work on it there until you’re ready to tag it again.

However, assuming that your fix has worked and been approved in the Test environment, use the dashboard to move code from test to live as per normal.

With the hotfix deployed to live take a few deep breaths. But don’t rest too long. Before you forget, checkout the master branch, merge the hotfix tag and push to Pantheon so as not to lose your save-the-day commits.

git checkout master

Switched to branch 'master'

Your branch is up-to-date with 'origin/master'.

git merge pantheon_live_3

Auto-merging sites/all/modules/custom/custom/custom.module

Merge made by the 'recursive' strategy.

sites/all/modules/custom/custom/custom.module | 4 ++++

1 file changed, 4 insertions(+)

git push origin master

Counting objects: 8, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (7/7), done.

Writing objects: 100% (8/8), 813 bytes | 0 bytes/s, done.

Total 8 (delta 3), reused 0 (delta 0)

To ssh://codeserver.dev.ce2bbe07-8e4a-4779-9bb5-3e2fe88d7fc7@codeserver.dev...

  87677e0..265a821  master -> master

Managing Risk

Hot fixing implies a level of risk. Our dashboard tools encourage you to test feature branches in Multidev and do integration testing in the dev environment and a final test deployment in the test environment. Cutting out any of those steps results in an incremental increase in risk. The question to ask when considering a hotfix is “does shortcutting our process to deploy the hotfix introduce more risk than waiting for the fix to be deployed through normal procedures?” Answering that question fully will require a lot of site-specific context to inform the second half of the question. The first half has a more generic consideration. Hotfixing on Pantheon does not mean shortcutting all testing.

When working on a hotfix we recommend you still consider:

• Testing your change in a Multidev environment.
• Making a tag to send to the test environment (You really shouldn’t skip this step).
• Using a suite of automated tests. See some of our past posts on Behat and Wraith. Such tests can make you more confident in your hot fix and even make the need for hot fixing less likely.
• Merging your hotfix tag back to master as soon as it is deployed to live. This will get your dev environment caught up and back in the normal workflow.

No one likes to hot fix. When the unexpected happens you can still use Pantheon’s tools to mitigate the risk of hotfixing. We also know that our dashboard user interface could add some options to make hotfixing possible in the browser. Let us know what tooling you want to better enable emergency procedures while still encouraging the preferred workflow.