What We Are Seeing With Drupal SA 2014-005

It's been 24 hours since Drupal SA-CORE-2014-005 was announced, and we are already beginning to see attacks in the wild. As a platform with 10s of 1000s of Drupal sites, we have a unique perspective on the problem.

(Updated with new attack patterns below)

This is not a drill: black-hat scripters from sketchy domains are working through lists of known Drupal websites probing for exploits. If you have not patched all your sites, stop reading and do it right now.

...

Ok, now that your websites are safe, here's what we're seeing.

Profiling and Logging Suspected Exploits

We learned of the vulnerability through our participation with the Drupal Security team, so we had a few days to prepare prior to the announcement. At that point, we were under obligation not to share details as part of responsible disclosure, but we did tweet and email customers to "be ready" for the update on Wednesday.

Beyond that, the first step was fashioning our own exploit to have something to build a defense against. I "owned" my personal blog several times getting this right.

With a sense of a potential attack signature, we developed platform-wide request filtering, WAF style. At our scale, we couldn't try to tweak every individual site: a platform solution was the only answer.

We got that deployed on Monday, giving us two days to see the results of real production traffic. We were able to eliminate false-positives while still detecting our PoC attacks, which gave us confidence that our filter would not impact legitimate traffic. That was an important moment, because it meant we could start locking things down.

Log and Block

With the SA announcement on Wednesday we switched the filter from "log" to "log and block". The first detected (and blocked) attack came in at 22:42 UTC (3:42 PM PT), about seven hours after the security announcement. It attempted to set up a fake user with id 9999 and a suspicious temp email address from trbvm.com.

Over the rest of the day we saw a handfull (20-ish) more attacks that looked like proof of concepts or penetration tests. We saw attempts to re-use a proof of concept posted in a Reddit thread, an attempt to create a user named "morpheus" with a pre-set password, and a few attempts to make accounts with the email address test@test.com and then elevate them to an admin role.

It Gets Real

Early this morning at 08:23 UTC (1:23 AM PT) we started seeing an attack that attempts to insert a new item into the menu_router table. This attack is originating from IPs from a VPS provider in the .ru domain space, and it appears to be working through a list of domain names alphabetically.

The attack seems to be the initial part of a multi-step process. The menu_callback it is attempting to create will try to use file_put_contents() to drop a file somewhere in the codebase. That file will pick up a subsequent http request with more of an attack payload in the $_COOKIE superglobal. This sophistication plus the alphabetical attack sequence suggests a professional exploit.

Note that this attack has a 0% chance of success on Pantheon. We block it, but even if we didn't live sites can't write files into the codebase, and a sophisticated $_COOKIE attack would also be stripped. Still, it's concerning.

This Is Not A Drill

It's barely 24 hours after the SA, and we have logged and blocked over 500 attempted attacks on sites on the Pantheon platform. We expect this rate to increase as exploit code is more widely shared and attacks become more automated.

The fact that we are blocking suspect traffic does not mean you delay updating. We're happy to be defending sites on our Platform, but the filter, like CloudFlare's WAF firewall rule is not a guarantee to secure your site. You need to get the update deployed and patch the vulnerability at the source.

If you need help, let us know. If you have friends who need help, lend a hand.

Credits

Credit to the Drupal Security team for organizing a responsible and orderly release. There was likely temptation to rush something out once the severity was realized, but they showed great professionalism by taking a more deliberate route. As soon as the fix was disclosed, black-hats would start working to weaponize the exploit, which we are already seeing.

I'd also like to thank Leonardo Finetti for chiming in based on some tweets with additional information about the menu_router attack. He has his own post up (in Italian) here.

Finally, I'd like to give credit to Greg "greggles" Knaddison for planting the idea in my head of using the reach of our platform as a way to monitor exploit attempts against sites running on Pantheon. Hopefully the data we're able to gather will help everyone defend better and build more secure software and platforms.

Update: more attack patterns

We'll keep updating this post with more attack patterns as they are identified. These are all blocked on Pantheon, but can be useful if your site was unprotected and you're worried that you've been hit.

• menu_router attack with file_put_contents
• the "megauser" role attack.
• another menu_router attack using assert as an access callback.