Custom Upstreams: Enterprise-Scale WordPress and Drupal

Large organizations interested in Drupal or WordPress as CMS platforms have historically faced a fundamental conundrum: the main benefits of open source are directly undercut by the compromises inherent in Multisite architecture. Far too often this tension is unresolved and allowed to fester, even to the point of dooming entire projects.

Pantheon has a unique approach to website portfolio management we call the Upstream workflow. Upstreams allow big orgs to use WordPress or Drupal for hundreds, or even thousands, of sites without having to compromise on either creativity or centralization. It provides both the compelling economics and oversight of a central solution, with the flexibility and freedom that modern website operations demand.

Website Portfolios Need CMS Platforms

Organic bottom-up adoption of open source CMS usually starts with individual sites here and there, but an org-wide or institutional approach is naturally different. It means taking a step back, gathering requirements from many stakeholders, and then making a thoughtful longer-term investment.

Anyone evaluating WordPress or Drupal (or both!) at this scale will need to standardize a certain amount of code and process. It just doesn’t make sense to do each site as a completely custom build. While you can get whatever you want that way, you’ll re-invent the wheel frequently and create a long term maintenance nightmare.

Sometimes an organization tries to build a big flagship mega-site, one which can give editorial control over different sections to different teams, or allow for “sub sites” or workspaces. Unfortunately, this frequently leads to highly complex implementations which suffer from “design by committee” anti-patterns, death-march length implementation timelines, and high maintenance costs with low rates of ongoing innovation.

Smarter organizations embrace the fact that different teams with different goals will need to control their own website. That’s the only way for them to move fast and control their own destiny. They’re embracing the reality that most companies have hundreds of websites, not one, and thinking in terms of managing a site portfolio vs trying to get every requirement done in a single mega-site.

So, great, everyone gets a site, but how does that actually work? It can be deceptively simple to start knocking out individual installations from a common template. Just copy the theme from one site to the next and bob’s your uncle. And if they’re all on the same server, they’re technically centrally managed!

This obviously doesn’t work out well. Without any formal mechanism for sharing code, performing routine maintenance, or governing access, the “many independent installs” architecture quickly descends into the same chaos as doing every site from scratch. If anything, you just get there faster.

Organizations looking to adopt WordPress and/or Drupal as a solution for some or all of their total site portfolio, have to develop a platform-minded approach to CMS. It’s the only way to satisfy the requirements of enterprise scale.

The Multisite Letdown

Many would-be adopters of Drupal and Wordpress at this level are drawn to various flavors of Multisite, which initially do offer a lot of upside. They centralize the management of code and can be highly efficient in terms of server resources. However, they come with a monolithic architecture which creates a single point of failure that ultimately undermines much of the value of selecting an open source CMS to begin with.

Multisite architecture has a single point of failure in infrastructure terms (creating risk from a security perspective, or when handling a traffic surge), but even more crucially it’s also a restrictive single point of collaboration. Your sites might never go viral or get an internet-scale traffic spike, but if the codebase is under active development, at some point a human being will make a mistake—it’s inevitable.

A single point of failure magnifies the impact of human error, and this kills innovation. It often sticks senior developers in a grueling internally-facing role, consumed with release management, trying to anticipate every possible scenario when rolling out an update, and still not catching everything. After all, they’re only human.

This is why Multisite is only worth considering in completely cookie-cutter cases, and even then, we don’t recommend it for Drupal at all. With WordPress, the core Site Network capability lets you run blogs in a box pretty well so long as your users are just publishers, and you don’t plan on releasing changes other than updates coming directly from WordPress.org. But there’s just no good reason to cut yourself off from the value of open source if you’re thinking about WordPress or Drupal as an enterprise CMS platform.

Avoid Crumbled Cookies

The worst case scenario is when Multisite architecture is selected, but the discipline of keeping sites uniform isn’t maintained. This is the place from which Multisite horror stories arise.

If you have one codebase running a bunch of very different websites, or even worse you start doing per-site customization on a monolithic install, you’re just counting down the days until you’re faced with a burning platform. A release intended to give a much-needed feature to site X will break site Y. Or even worse, you need to get a critical security update out, and it turns into an all-nighter because a few of the sites didn’t take it well.

If you stick to a cookie-cutter use case, don’t mind a single point of failure, and can live with a cumbersome release process, there’s potential value to be found. But whatever you do, don’t let the cookie crumble.

Don’t Believe the FUD

Back when I first published Much Ado About Multisite, pointing out the above flaws, it sparked some energetic conversations, culminating in a packed birds-of-a-feather session at DrupalCon Austin called “the great Multisite debate.”

One of the things that really stuck with me from the session was a solutions architect from Acquia® telling me that Adobe and SiteCore sales people were sending around my blog post to dissuade people from choosing Drupal. In the event that something similar happens with this post, I want to make it clear: if you’re smart enough to find and read this, you should be smart enough to see through that kind of proprietary vendor FUD.

The Adobe Marketing Cloud is a suite of acquired technologies, stuck together with chewing gum and duct-tape; or “enterprise integrations” as they call them. The centerpiece is a decade-old proprietary static site generator—Adobe Experience Manager (AEM)—that requires custom J2EE Java code to extend or even theme. That’s a platform with some problems.

Adobe has zero ecosystem (seriously, try Googling anything about how it works) and gets sold based on a vaporware/visionary value of a “totally integrated omni-channel digital experience” that even salespeople will admit is years (and 10,000 billable hours from Accenture) away from reality. Even after all this implementation work, or maybe because of it, AEM websites tend towards the antipatterns of the flagship mega-site model I mentioned above: slow moving, full of stakeholder consensus compromises, and ultimately several steps behind the market.

They’re doing big business now, but I’ve heard more than one story of massive projects that are in year two or three of a death-march implementation slog without a believable launch date in sight. How many people will opt to replatform onto a faster moving and more widely supported stack once their contracts expire? Time will tell, and since they’re a publicly traded company it’ll be open for the world to see.

They liken themselves to the Rolls Royce of CMS, which is hilariously apt to me. Who in the world delivers business value via a Rolls Royce? They’re heavy, slow, gas-guzzling status symbols. Is that what you want your website to be? Enough said.

Have Your Cookie And Eat It Too

I’m here to tell you there’s a better way. You can have your cookie and eat it too, enjoying all the richness and agility of an open source CMS (or CMSs) at scale, without the maintenance nightmares of free-for-all install chaos, or having to make the Multisite compromise.

This value of open source is not the zero-dollar license cost: it is a rich ecosystem of add-on code, public best practices and how-to’s, and qualified professionals for hire that can be brought in to achieve goals quickly and effectively. The historical snag is that Multisite architecture is only really applicable in the cookie-cutter context, at which point you’ve left the richness of open source behind. You can’t tolerate meaningful deviation, and certainly can’t support site-specific development, or even parallel central development, be that by contractors, agencies, or even just separate internal staff.

When we first started Pantheon we took a fresh look at these problems from our own perspective. How would we run hundreds of thousands of Drupal and WordPress sites on one giant infrastructure, give developers unlimited freedom, and still keep things sane from a management and update perspective?

Thanks to two technologies that weren’t around when WordPress and Drupal were getting started—Git and Linux Containers—we arrived at the Upstream workflow, and we’ve used it ourselves for every single site running off a “vanilla” CMS core.

This has worked out tremendously well for us, and now we’re offering every organization that uses Pantheon the ability to run their own Custom Upstreams—whether that’s to knock out repeat use cases, directly adopt Drupal or WordPress as a platform, or to go out and pitch big clients with an organization-wide enterprise CMS solution.

The real beauty of this architecture and our platform is that it explicitly enables federated and fast-moving teams. We can empower the kind of hub-and-spoke organizational model that generates so much friction with either the flagship mega-site or monolithic Multisite approaches, delivering the full open source value to the entire organization.

With Upstreams, you can engage separate teams of developers to work on centrally managed shared code vs individual site implementations. That means you can have individual departments or business units handle their own last mile, or do it for them with a dedicated site implementation team, or even engage (or let them hire) an outside agency on a site-by-site basis.

Custom Upstreams: High Flexibility, Low Per-Site Cost

To recap the different approaches we reviewed:

  • Custom CMS

  • Flagship Mega-Site

  • Multisite

  • Custom Upstreams

Plotted on axes of flexibility and cost, it looks something like this:

With the Custom CMS implementations, whether you’re installing the same CMS over and over in one docroot after the next, or going down the path of building an all-encompassing flagship mega-site, you’re going to run into serious problems with maintenance and flexibility. It’ll be relatively costly to get off the ground and to maintain. Proceed with caution.

Multisite does offer value if you stick to the cookie-cutter/blog-in-a-box use case, but it can be deceptively expensive to get to an initial release. Since you’re effectively delivering a SaaS product your users need something that’s fairly complete/turnkey before they can start. Also, since you cut users off from all the open source value, you’re the sole source for new features and improvements.

Beyond that, Multisite’s monolithic architecture makes it inherently challenging from a maintenance standpoint. You can’t share the load with other teams, and there are locked-in single points of failure. Things can unravel quickly if there’s a big traffic spike, security breach, or a mistake in the release process.

I won’t belabor the point about Multisite gone wrong. Just Google it and you’ll get plenty.

Finally, we have our own Custom Upstreams option. Naturally, as I’ve created this quadrant, we’re in the upper right, but I think this is a defensible position. Using an Upstream approach allows you to start quickly, engage a wide pool of talent, leverage open source at every level, and it’s proven to scale—both by Pantheon directly and by many of our happiest customers.

Custom Upstreams are your ticket to freedom from Site Factory and other Drupal Multisite quagmires. They’re also the best way to elevate WordPress from the internet’s most popular blog to a truly enterprise grade CMS platform.

If you’re interested in learning more, you can try out the product today, or talk to one of our experts. We’re very excited to finally bring this solution to general availability, and to teach people how to understand how it fits their use cases. Whether that’s meeting ambitious client goals, bringing sanity to a fragmented internal CMS landscape, unwinding a troublesome Multisite, or finding a way to move faster than a flagship mega-site allows—let’s chat!


You may also like:

Topics Multisite, WordPress Hosting, Website Technology, Development, Drupal Hosting, Growth & Scale, Drupal, WordPress