Cyber Monday: You’re soaking in it. If you’ve got the time to read this today, you clearly aren’t working in a company that has a major online retail component. Rather, you should be staring at your dashboards, religiously checking to see if anything has broken. Or if it’s broken now. Or now. Or maybe even now.
This kind of compulsive stat checking is entirely normal, so don’t feel strange for being glued to your dials and logs. What’s strange, rather, is the ways in which the Cyber Monday phenomenon will affect future teams of developers.
Why? Because development teams are changing. Out here in the Valley, I’m seeing the start of some very interesting experiments in agile, pair programming, onboarding, and generally a reworking of how software teams are built (and in how they build their software).
My personal favorite right now is a large-ish startup (more than 300 people) that employs no testers, admins or operations people. Instead, everyone is responsible for his or her own code. If your code breaks the build, you’re on the e-mail chains at 3 in the morning, and you’re the one rebooting instances in Amazon Web Services.
This no-ops, no-testers company (called Optimize.ly) is, actually, built of an amalgam of talent from operations and testing groups. But when they join the company, they’re quickly informed that they will also be coding. And coding is, essentially, their first obligation now.
That can mean they write unit tests all day or Chef recipes all night. In the end, it means that everyone is using source control for their work, and everyone is expected to be completely responsible for the assets they generate. Nothing is ever “thrown over a wall.”
This also means there’s never a situation where some poor operations person is called into the office at 2 a.m. because someone in development or testing dropped the ball. No one else suffers for your mistakes. (Well, no one who can’t look you in the eye over a meeting table the next day, anyway.)
And this does touch on one of the things I do find disturbing about this particular organization: meeting creep. If I had to point out anything that I felt it has sacrificed in order to function in this fashion, it’d be that their teams are sometimes in meetings all day long. No one likes to do that. But it does seem to be an important part of the transition, and it can be done in a way that leaves full days for coding.
For this particular organization, their developers are busy before Cyber Monday, and also during Cyber Monday. They’re not taking the day off and waiting to be e-mailed if there’s a problem; they’re the ones looking at the dashboards.
Another organization I have to wonder about on Cyber Monday is Uber. You may recall that we’ve been writing about their infrastructure since we saw their chief architect speak at QCon two weeks ago.
Uber embraces the chaos. They went from two services to more than 700 in two years. As a result, their infrastructure has been designed to help manage, rather than rein in, that chaos.
What this means, however, is that newcomers to the company can literally write greenfield code the first week. Everything is service-based, so the language and environment a developer uses are largely irrelevant: Everything will just end up as a Docker image in the end, anyway. That really cuts down on onboarding.
Instead of staring at some ancient and inscrutable code for months until productivity sets in, new developers write their code from scratch and need only familiarize themselves with the documentation for interfacing with the other services they’ll need.
There is a key to this work, though: Uber leaves things alone. This is something I’ve seen many organizations struggle with. Leaving a service alone, once it’s reached stability and reliability, is the absolute cornerstone of managing this type of infrastructure.
For Uber, Cyber Monday is just a bump in the road. New Year’s Eve is where the company really spends its hard-earned engineering dollars, as everyone wants a ride home after midnight.
That day requires 365 of engineering and scaling work. Every year Uber tests the limits of its infrastructure on New Year’s Eve, and every year they just squeak by.
Frankly, though, squeaking by is success in this business. When you’re a professional on stage, no one can see you sweat, and when the servers are hosted in the cloud, no one can see them fail.