What We Learned Blocking 2.8 Million Drupal Exploit Attempts

It’s been just over 10 days since a proof-of-concept exploit was published for Drupal SA-2018-002—which the tech press has dubbed “Drupalgeddon 2”—and so far Pantheon has blocked over 2.8 million attempted exploits. Last week, we were blocking over half a million exploit attempts a day, though things have calmed down a bit since the weekend.

This is what happens whenever—inevitably—a vulnerability for a widely deployed piece of software gets exploited. Whether it’s accidental exposure of MongoDB’s default admin login, or un-patched Oracle Weblogic or PeopleSoft systems, attackers are constantly scanning for common vulnerabilities, and when found they will be used.

The time between exploits going public and weaponized automation is now measured in hours, and no stack is safe. The same pattern has been borne out time and again across Windows and Linux platforms, with open source tools and proprietary software, and even down to the library or framework level. If any of the code you run is popular, a future attack is a question of when, not if.

For web teams this presents a particular challenge. The demands of a fast-paced internet require innovation and make static websites a no-go, but the same fast pace creates full-stack security demands which are frankly unrealistic for most teams. And for those unable to meet these demands, the cost can quickly become unbearable.

DIY website operations has historically been a pretty good bet. That’s how the LAMP ecosystem really took off, and companies like Digital Ocean and Linode built their business based on the promise that anyone can be an Ubuntu sysadmin. But that’s no longer the case. If you aren’t investing in a fully staffed 24x7 operations team (and few are), you’re placing a crushing burden on people who will eventually burn out and leave, while simultaneously running a high risk of compromise or catastrophy.

More Than Just Blocking: We Kept Receipts

In addition to blocking these attempts, we kept the receipts. We have logs of who tried to do what. By dumping a few gigs of this data into Google’s BigQuery engine I was able to generate some interesting analysis:

Geographic Distribution

First off, where are the attacks coming from? By reverse IP lookup, we can cluster the attempted exploits based on their provider location:

Map of attempted exploits by location

The big circle in Ukraine is a single particularly active exploiter. While there were certainly a lot of attacks coming from traditional cybercrime hotspots like Eastern Europe and Southeast Asia, more exploit attempts originate from the US than any other nation.

It’s also worth noting that a lot of attacks came in on sites which had Cloudflare running in front. I’ll be interested to see if this changes if a signature for this attack starts to be something they filter for at the free service level. The takeaway for customers is that Cloudflare’s gratis service can’t be relied upon to turn around protection for these kinds of vulnerabilities in real time, which seems reasonable.

Exploit Payload Analysis

Secondly, and more importantly, what are they trying to do? By analyzing the attack payloads and generating a signature, we can cluster them by goal:

attempts distinct_ips signature
1402882 208 beach-head
760493 95 probe
237801 11 patcher
187176 23 deface
170434 21 botnet
97415 21 miner
26799 51 unknown

 

By far and away the most common exploit was an attempt to install a beach-head: a PHP file that could be used later on even if Drupal was patched in order to continue uploading new files and attempting further exploits. These ranged in sophistication from one-line scripts set up to eval() incoming POSTs, to full-on webshells with authentication mechanisms.

This common class of vulnerability is already mitigated on Pantheon, even without our specific mitigations for Drupal SA-2018-002, thanks to our immutable production environments. One of the big security benefits of building off version control and a structured deployment workflow is that live sites can’t add their own code, so the “blast radius” of a compromise is contained.

Next most common was a simple probe, either an attempt to get the website to echo back a specific result, to “phone home” to a remote server, or simply to place a harmless “I was here” file. These were sometimes followed up with other attacks, but in some cases might have simply been cataloguing for white-hat purposes, or marking territory.

After that we actually saw a class of exploits that attempted to patch Drupal. In some cases that was all an actor would do, maybe an “outlaw white hat” at work, but in other cases these were attempted only after malicious attack attempts. A “shut the door behind you” kind of action.

The pattern of trying to edge out the competition becomes more heated once we get into the real action—attacks sophisticated enough to set up a botnet or install cryptocurrency mining tools in one shot. These are not your typical “download this script from pastebin” type of approach. Many include several layers of obfuscation, attempts to kill off other competing infections, and scheduled tasks to restore the malware if it’s incompletely cleansed, all within a single attack payload.

While cryptocurrency mining attacks represent a little less than 10% of all exploits, in my opinion they are the ones to worry about. They are the most sophisticated, and given the upside of successfully stealing web server CPUs, this is a the pattern to watch for in the future.

One final thing to note is that we have not seen any exploits which attempt to alter the internals of the Drupal database, for example to create a privileged user or alter site content. There are some defacement attacks, but they’re restricted to trying to overwrite index.html with a “takeover” homepage and don’t suggest any Drupal CMS awareness.

The Zombie Apocalypse Is Real

The average time between an exploit going public to it being weaponized and showing up in scripted attack kits is under 12 hours. Systems which end up exploited via automation are frequently just used as additional attack nodes to exploit other systems, a phenomenon I’ve previously likened to a zombie apocalypse.

But now there’s an additional incentive to get your exploits out: the ability to turn a compromised CPU into cash more or less directly via cryptocurrency. It’s becoming commonplace for bad actors to figure out ways to maliciously use other people’s computing power to generate new cryptocurrency, and we should expect this to drive a new wave of exploits both on the client side and on the server-side, where powerful CPUs with backbone  internet connectivity make a very tempting target.

While the Bitcoin bubble may have burst, cryptocurrency is an established fact of life online, especially on the dark web. Four years ago the market and tooling wasn’t developed enough to support monetization in this way. Every time a self-managed server is compromised the zombie horde grows bigger, and the time between exploit proof of concept and weaponization goes down. A direct profit motive will only serve to accelerate things further.


You may also like:

Topics Drupal Hosting, Security, Drupal