A New Media Model Calls for New Media Technology at Patch

The Patch.com product—quality hyperlocal content across 915 communities and growing—presents demanding challenges, including those related to management and scale. Patch often publishes over 1,000 articles per day. 12,000 URLs are accessed via the Patch site at any given time within the last 10 minutes. Pantheon helps Patch tackle these challenges while mitigating risk, supporting flexible development, and guaranteeing uptime—without forcing Patch to maintain large, complex infrastructure. The net result has been helping the Patch team spend its time innovating to make a profitable national platform for hyperlocal news.

On Pantheon, Patch.com is running at 50 million page views & 20 million uniques a month at 99.969% uptime across approximately 35 million URLs, 40 terabytes of images, 500 terabytes of video, 6 million registered users, 15 million stories...and we have 4 developers."

—Abraham Brewster, CTO at Patch

As CTO, Abraham Brewster is constantly seeking new ways to innovate and improve efficiency at Patch. When he joined the team in September of 2014, he turned to Pantheon as a solution to manage infrastructure so his team could spend more time developing and growing the company. Managing a media site with as large a footprint as Patch is no small task, but Brewster and the team make it look easy—regularly scaling to millions of pageviews in a single day without breaking a sweat. Thanks to Pantheon, the Patch team can focus on building great technology without worrying about infrastructure or downtime.

Making “Everything Local” a Sustainable Business

Image

Founded in 2007 by Tim Armstrong, Warren Webster and Jon Brod, Patch was created to be an online local community bulletin board and news source. AOL acquired Patch and grew it into a national brand, spending hundreds of millions of dollars in the process. Under shareholder pressure to abate its losses, AOL partnered with Hale Global in January 2014 to turn the business around through a fresh approach anchored by technological innovation. Since Hale Global’s takeover of Patch and platform migration, traffic has doubled while achieving profitability, allowing Patch to grow its hyperlocal coverage while other major media publications have continued to dump local coverage due to cost and complexity. The only major media platform for local news, Patch strives to be in all towns.

Under its new Editor in Chief, Warren St. John, Patch now publishes between 800-1,000 stories a day to over 900 hyperlocal “Patches”. Writers and community members can post directly to a given Patch, fostering a community-centric news forum with the most relevant information for the local audience. 100,000 different articles are read on Patch.com on any given day, and the 24/7/365 news cycle leads to some 12,000 URLs being accessed via the Patch site at any given time within the last 10 minutes.

Patch is a unique synergy: a bunch of devoted people building a scrappy, non-startup startup. We're writing great content. We're serving our users and they love it; they keep coming back. Though we are obviously trying to make money, we have the soul of a nonprofit. We are trying to help local communities in a media landscape, which has completely forsaken them."

—Abraham Brewster, CTO at Patch

Building a Sophisticated Custom Stack

When Hale Global became majority owners of Patch in 2014, the team had a strict timeline to move off of proprietary AOL infrastructure, and needed to develop a solution to run the massive Patch platform. The Hale Global tech team built an impressive custom AWS stack using Drupal for their content management system. They had tried out a number of CMS solutions, but needed a backend that could be a workhorse, allowing the team to concentrate on UI work for both the front and contribution interfaces. Drupal's relational systems—coupled with Varnish to protect the query cost, along with its robust services support made the choice clear.

The team ultimately condensed Patch’s platform footprint as follows:

• 2684 domains down to 1

• 96 servers down to 15

• 903 websites down to 1

Brewster joined the team in September of that year. He immediately got to working finding solutions to help grow the platform, while nipping inefficiencies in the bud.

We’ve got to innovate like crazy. We need to be able to run stuff up, have ideas crash and burn, and have infrastructure that can support us."

—Abraham Brewster, CTO at Patch

“Leaving our infrastructure and processes in Pantheon's hands allows us to concentrate on innovation,” he added. “We, of course, have to worry all the time about scale: we have 15 million stories and 6 million users. Innovation, when you are running at this scale and speed, comes with both successes and mistakes, and unexpectedly some of them can cripple a system. The parallel scaling on this platform gives us the cushion to innovate.”

Patch requires high availability all the time, for everything. While the custom AWS setup worked well, a website management platform would allow the team to innovate faster with a better ratio of ops personnel to developers, and maximum uptime. Brewster suggested Pantheon to support their infrastructure and unique Drupal build.

A Decoupled Build: Innovating in an Ever-Changing Industry

Brewster knew that having a flexible build with lots of granular caching was necessary for a media site like Patch, and therefore suggested setting up the site as a decoupled Drupal build. However, instead of using Drupal solely for the backend and choosing a framework like Angular for the front end, Patch was built with one installation of Drupal to run the front end, and a separate installation of Drupal to run the backend.

“This build solves the problem of Drupal being able to do a lot of things well, but not all at the same time,” Brewster said. “Our front-end site is a Drupal site with no users, no nodes, no views. It has no content in its database—it sits there as a caching, configuration box and a theming box. It captures less-volatile information, digests it, and caches it locally for display. When you don’t ask Drupal to ‘think’, it can serve you a page in 200 milliseconds. If you ask it to both run the theming and display as well as the content-management and heavy database lifting, it can clog itself up—especially with a database as large as ours. There would be no way we could serve our content footprint and do the rendering at the same time, while doing all the other things that would have to happen on a service box—authentication and content relationships. We came to this solution because it seemed to be the best of both worlds. We let the CMS manage content and let the theming layer do the theming, and then put this massively scalable buffer between them called Varnish. The front end never calls a database. Instead of hitting a database, it’s hitting Varnish.”

One benefit of this unique build is that if someone adds malformed content to the service box, it’s not going to take the entire front end down with it. Brewster and the team don’t have to deal with big errors—if there’s bad JSON, the front end can trap it. Another plus for Brewster is that he doesn’t need to worry about hiring or contracting a dedicated front-end developer, because his whole team knows Drupal and can move around the platform.

“I don’t have to call a JavaScript person to come in when I only need him or her once a week. Ours is a very uniform, knowledge-base reinforcing team. If someone doesn’t know a function, someone else has handled the problem before. We have a group of people who can build stuff quickly and with innovation. If we really must to do something outside our wheelhouse, it’s not a big deal for a talented developer to have to learn one technology on top of a bunch of stuff that he already knows.”

Image

Choosing Pantheon and Migrating a Complex Infrastructure

When faced with the decision of where to host and manage Patch’s infrastructure, Brewster knew he wanted Pantheon. Maintaining a custom AWS stack was costly and the time required to manage it often kept the team from from innovating at breakneck speed.

“Everything was working well, so you could imagine Patch wasn’t thrilled about the idea of yet another migration, so I had to make the case for Pantheon,” Brewster said. “Pantheon was an architecture Patch could leverage and grow on. The real selling points for us were speed, risk mitigation, and flexibility of development. Since the whole thing is built on a services-based architecture, we can leverage its independent parts as the backbone of an app. Different pieces of the stack can be called individually, all insulated by Varnish wrappers that mitigate the risk of a ‘noisy neighbor’ process bringing down the whole neighborhood.”

The Hale Global team migrated the massive database from PostgreSQL to MariaDB. They did a huge migration first, while the New York team optimized the queries for Maria and MySQL. In doing so, they sped up a number of their most complex service calls by a factor of 10. “Some of it was the database, some of was the application horsepower and a lot of it was just some great coding done by our team with major assists from David Strauss and everyone at Pantheon,” Brewster said. “Our QA team and Pantheon facilitated relentless load and stress-testing to make sure everything was good to go.”

Ready for Whatever May Come

Patch is now running smoothly on Pantheon and the team is thrilled. The site runs with 8,000 page views at any given moment, and doesn’t break a sweat when spikes reach up to three million page views in a day. No downtime means the team isn’t worried about losing valuable advertising revenue or readership, and leveraging Varnish allows the team to test and innovate efficiently. The overall cost is comparable to their AWS solution, but with more capability and a sophisticated set of features and developer tools. Patch.com is running incredibly fast—a necessity in the 24/7 news media landscape—down from 2 seconds on AOL to 450-500 milliseconds on AWS to 240 milliseconds on Pantheon.

We handled three million page views in a day for one particularly viral story about arsenic in wine, and our hardware didn’t even know it. We had no reason to worry, so we kept developing." 

—Abraham Brewster, CTO at Patch

Moving Forward: Changing the Industry and Building Out Patches

Now with Patch.com running smoothly on Pantheon, Brewster and his team have the bandwidth to focus on innovating, building out more patches, and being “everything local.”

“We can launch a new Patch in minutes,” Brewster said. “In the end, it’s just a Drupal stack. It’s serving a taxonomy term and some JSON into our system once a day. We just spin it up. Patch is expanding. We launched a half dozen in the last week in an entirely new market.”

Patch is positioning itself to become a hyperlocal media powerhouse—to thrive, they must choose partners that can support continuous growth and scale. On Pantheon, Brewster and his team can innovate rapidly without worrying about the infrastructure.