Introducing WordPress at Scale

WordPress is increasingly being taken seriously by large organizations and for high-profile use-cases. This is good news for the web, and good news for open source, and today we’re happy to announce a new resource to help developers and organizations who are looking to get in on the action: WordPress at Scale.

Scaling is hard. And when it comes to WordPress, there aren’t many good resources for how to approach the problem. There’s a lot of confusion and uncertainty; too many forum threads to nowhere. This holds projects back, or worse, puts them at risk. That’s a problem.

The movement upmarket has led to an increasing demand for platforms and infrastructure that can handle WordPress at Scale. This is one of the main values Pantheon delivers, along with cloud-enabled development workflows for teams.

While we’re happy to be a part of the open source CMS success story, we don’t harbor any illusions that every single project will run on our website management platform. Furthermore, it’s strategically important for us that WordPress and Drupal continue to succeed at scale, even if that happens on-premise, via AWS, or even using one of our competitors.

For the past several months we’ve been collecting best practices and resources, and put together the microsite to help anyone thinking about scale, no matter how they want to get there. The content of the site is a community effort managed via GitHub (pull requests welcome!).

The goal is to increase the rate of success with organizations looking to run WordPress at Scale, period. You can check out the site now, or read on for Pantheon’s take on the problem.

How It’s Traditionally Done

Traditional architecture for scale involves building a “cluster” of servers. You have to get off one box if you want to scale. Being able to add more as needed is the whole point. This means deploying your application across many servers, routing traffic to them, and also managing all the data resources so they are consistent.

Typically you’ll also see people scaling their database via replication, adding multiple caching layers, and for WordPress especially, including an optimized search index. The end result can be fairly complex:

Image

There’s a deeper explanation on the WordPress at Scale site, but don't panic! This stuff is all doable, and there are DevOps professionals who are ninja-skilled at creating and maintaining this kind of implementation. At the same time, it’s a lot to bite off.

Clusters Don’t Scale

This is also a lot to handle if you're a service provider. The cumulative upkeep for lots of individual clusters can quickly become a challenge, if not outright unsustainable. This is why so many traditional “hosts” seem great at first, but end up descending into suckyness as they become more popular.

Likewise, cost and complexity often means that clusters are special snowflakes, with one-off rules and behavior. It almost always means there’s only a single implementation—production—meaning nothing like it is available for development or testing purposes. That makes clusters infamously difficult for development teams, and explains why site owners and project managers stress so hard around release time.

In short, while a cluster can scale an individual site, the model can’t address the internet-wide demand. The fact is the old ways of doing things just can’t keep up. Clusters impede agile web development, and there’s not enough DevOps or sysadmin talent in the world to take care of all the WordPress projects that want to scale. 

The Pantheon Way

We founded Pantheon with the mission of solving this problem: making sure every project was on an elastic hosting platform from day one, and that development teams had the tools they needed—including version control, continuous integration, and consistency across environments—to build and deploy fearlessly.

We solve all the problems that a custom cluster would solve, but in a way that has zero human intervention. We do this using a mix of our own software, battle tested open source tools, automated configuration management, and the power of Linux containers to deliver everything you need, on-demand, as a service:

Image

Every single site on Pantheon—even the sandbox instance you can spin up for free—is powered by the same core platform. We have one technology that brings all the capabilities of a custom cluster to not only every site, but also every environment (Dev, Test, Live) of every site. Here’s how it works:

The Edge: Handles incoming web requests. It is composed of multiple redundant clusters that perform three basic functions: HTTPS termination (via Nginx), reverse proxy caching with Varnish, and then load balancing using a custom tool we created using Golang. This layer pushes tens of millions of requests an hour, and can handle the complexity of routing requests to the hundreds of thousands of site/environments that are running on the platform.

Runtime Matrix: Where the website code (WordPress or Drupal) actually runs. Rather than set up individual VMs for every site, we run a large fleet of huge high-capacity servers onto which we provision lightweight Linux containers. Provisioning is completely automated, and new application containers spin in seconds, so horizontal scaling is a snap. It also allows us to easily deliver redundancy, which helps site uptime. This is how you deliver elastic hosting at scale.

Service Bindings: Serve the needs of the CMS for content and related services. We have a horizontally scalable database grid—running the latest MariaDB—that’s similar to the runtime matrix in design, as well as a custom built filesystem that keeps your media uploads in sync across all these containers. We also have developed containerized services for Object Caching (using Redis) and Search Index capability with Apache Solr.

Compared to managing your own infrastructure, Pantheon delivers huge value to teams and organizations. Not only is all this stuff dialed in and taken care of 24/7, but you also have a first rate dashboard, and best practice development workflows set up from the start.

It’s a little like the value described by David Whittaker in this tweet:

Show this to anyone who argues about whether that refactoring effort is worth the investment. pic.twitter.com/vxw5PvDXqO

- David Whittaker (@rundavidrun) January 22, 2016

With Pantheon, not only are you set up to scale, you’re also able to deliver new features faster, with less chance of things going awry. It’s more than just elastic hosting, it’s the tools you need to do the necessary iterations to put that scalability to the test.

Learn about scalable WordPress hosting with Pantheon.