We here at Pantheon have been using "WebOps" more and more to describe the culture of a successful web team. Just as DevOps asks developers and system operators work together to balance their competing priorities, WebOps brings together developers, designers, marketers, content editors and more. Everyone whose job it is to change the website in significant ways (the code, the visuals, the content, the measurements of success) needs to think of themselves on the same team and working towards the same goals.
Well, maybe you don't need to think in terms of WebOps. Maybe it's OK to think of your task as isolated. Maybe there are some corners that can be cut. Maybe it's OK to risk an occasional WebOoops.
Mystery CSS Regressions, Ooops
Over a decade ago, I was working long hours with a small team of developers. We did things the Cowboy Coder way. When our client wanted changes done fast, that meant SFTPing straight to the live environment. We weren't entirely naive. We knew there were lots of ways that workflow could go wrong. So we took some precautions. We used Dreamweaver's Check In / Check Out feature.
If someone else had styles.css "checked out" and I needed to change it, I would holler across the office and ask them to check it in. Sure it was kind of annoying to virtually hand this file back and forth, but it was better than overwriting each other's changes accidentally. It would be even more annoying to be editing a CSS file, refresh your browser and see your changes disappear, right?!
Except that did start happening one night, somehow. How were styling changes disappearing when only one person could be editing the one CSS file at a time? Well, one member of the team had gotten so fed up with Check In / Check Out that they made "styles2.css" and started writing more specific CSS rules there. That was a WebOoops. We needed WebOps.
PHP Warnings During a Demo for Investors, Ooops
Ok, that WebOoops happened before the site actually launched. After launching, we knew we had to be more careful. We needed a staging site. But did we really need to use the staging site for every change? I mean our client was asking for changes all the time. And they wanted those changes as fast as possible. They would appreciate it if I sent some changes straight to production from my local laptop development environment, right?
Well, they didn't like when I sent up a change that started throwing PHP warning messages. They especially didn't like that they were in the middle of a demo for potential investors. "It worked on my machine" could be the slogan for WebOoops. We really needed WebOps.
Moving a Few Gigabytes of Media Files, Ooops
A few months later, that site was running out of space. There were people signing up every day and uploading large media files. We needed a bigger server but the hosting quotes we were getting seemed astronomical. The database wasn't that huge. The traffic hitting PHP wasn't terrible. We just needed somewhere to put this giant directory of media files. Amazon S3 seemed like the right choice. But it was still relatively new and I just couldn't get it working with the metadata processing we needed done on each file. I needed Drupal to act on local files.
Eventually we found a new host that we could migrate to. We just needed to rsync the files over and we'd be good to go. We fired up the command and it sent over a file... and then another file… and then another file. At the rate it was moving, we'd have to wait somewhere around a week for the transfer to complete. One of the data centers was capping the transfer speed of the SSH connection.
It was time to get clever. I figured out that I could open SSH connections in multiple terminal windows and transfer files in each of them. I wrote a script that looped over the giant list of files in random order and sent them one by one via SCP from the old server to the new server. I opened 42 Terminal windows. I couldn't believe it was working. The whole set of files transferred in an hour or two, instead of multiple days. I felt so smart.
These user-uploaded files were the most important asset my client had. And there are a lot of terrible WebOoops ways this particular story could end. But the WebOoops here is that I was in this position at all. I spent a ton of billable hours solving (well, moving) a problem that I should not have had in the first place. I was an inexperienced developer and in over my head. I needed a fuller team and I needed a mindset that my pain points in running this website were not one-off problems to be Googled in isolation.
Building, launching, and iterating improvements on a website is a cohesive discipline of its own. We call it WebOps. Doing it well requires a cross-functional team that understand the constraints and platforms they are operating within. If you're a Pantheon customer, there's a good chance that you know this already. But maybe, like me, you had to learn it the hard way. If you have your own stories of website woes, please share them on Twitter with #WebOoops.
You may also like: