You can cite the women’s vote. You can credit popularity among young voters. But it was DevOps that made the difference in the reelection campaign of President Barack Obama. So said the president’s campaign software team, speaking at an event hosted by New Relic on Jan. 10. The five engineers credited iterative development, Amazon Web Services, and the complete elimination of the development/operations barrier as the keys to their success.
Chris Gansen, dashboard tech lead for the campaign, said that DevOps was critical for the two-year project to build the campaign’s software. “Some organizations have a very clear line between engineering and ops. We worked very closely, so there was no line. It was absolutely incredible how closely we worked,” he said.
“One engineer would walk over and say, ‘What does Amazon have?’ Or, ‘I’m concerned about how this is going to scale.’ Or the admin comes to the developer and says, ‘This is running hot, make this change.’ These really quick feedback cycles were key to our overall success.”
Quick iteration cycles made a big difference as well, though the non-technical campaign staffers didn’t exactly understand the concept at first.
Scott VanDenPlas, DevOps tech lead on the Obama campaign, said that “It’s a culture clash. None of us has a political background. No one on the technology team came from the political arena. A campaign is something that pops up every four years, and it’s fairly resistant to external change and influence. And at the same time, we don’t understand how things work in politics. Both sides have a difficult time working around that. Iterative deployments are very foreign to a team that has never used technology.”
Due to this disconnect, things were difficult during the first summer of the campaign in 2011. So much so that Jason Kunesh, director of UI/UX on the project, said it was dubbed “The Summer of Messing Up.”
How the Narwhal overcame
Nick Leeper, finance tech lead on the campaign, said that staffers didn’t understand that they were looking at very early iterations of a growing platform. “They’d say, ‘This sucks,’ and we’d say, ‘We know that, but we are working to fix that.’ ”
This butting of heads also extended into the requirement-gathering process. Said Kunesh, “[Staffers were] used to working with vendors, to whom they say, ‘Go build this, come back three months later and we’ll yell at you because it doesn’t do what we want it to do.’ I would have to explain why we did things like talking to people in the field, or wire-framing apps.”
Other areas of difficulty came from the sheer scale of the project being undertaken. Using Amazon’s Simple Email Service was no small decision, as that service works out of a set pool of possible bandwidth and e-mails sent. That can get ugly when the Obama team was sending out e-mails to millions of people.
Thus, the team used redundant systems for almost everything. For donation processing, for example, the team used a third-party vendor, and then built an internal API that mimicked the vendor’s, allowing for redundancy in all donation-processing code.
And there were quite a few donations to process. Leeper said that the campaign’s websites took in a total of US$670 million from individual donations. Kunesh said the average donation was $55, but Leeper said the site “had crazy bumps. After convention, we were doing about $2 million an hour. We were just like, ‘What’s going on?!’ ”
VanDenPlas said that “The traffic patterns you see in a campaign are crazy. There are 25 million followers on Twitter, 35 million followers on Facebook. This e-mail list is multiple, multiple millions of people. One send through every channel hits 60 million people. The spikes go from 100 people using applications to 500,000.”
While Narwhal (the Obama campaign’s software platform for 2012) did perform properly on Election Day, the team said that in 2008, the Obama campaign’s Houdini software project was a disaster. That software failed within 40 minutes of launching on Election Day. The Narwhal team, therefore, started from square one. They even developed an application called Gordon, named after the man who killed Houdini.
“We were building everything from scratch,” said Gansen. “There was nothing held over from 2008. We started from five servers. The Narwhal guys are building the platform for all the data-integration tools while we were building the tools. They’re building platforms thinking the applications team might use this later on. We were building the airplane while flying. We started with nothing, and we ended with over 200 deployed applications.”
That list of applications ended up yielding only about two dozen that were ultimately used by the campaign at large. Those applications used a huge chunk of Amazon’s Eastern data center. “We had thousands of nodes,” said VanDenPlas. “We pushed 180TB of traffic with billions and billions of requests. We had 60% of all of Amazon’s medium [instances] in US East.”
Because of the reliance on Amazon, the team ran some game-day preparation tests internally to ensure they were ready for prime time. During one of these dry runs, Amazon actually experienced some issues, and the team went from addressing a fake issue to a real one.
And the rest, as they say, is history.
