As the New York City Velocity conference winds to a close, this year’s Web operations and performance event was highlighted by some engaging speakers and a laundry list of Web development news and releases from companies big and small.
In addition, Mikey Dickerson, a former Google engineering manager who’s now a member of the tech staff for the newly established U.S. Digital Service, gave a keynote. In “One Year After HealthCare.gov: Where Are We Now?” he gave the audience an inside look at how he got roped into overseeing the massive website cleanup and overhaul of HealthCare.gov.
What started as a favor to some desperate government friends and a multi-day trip to D.C. turned into—through what Dickerson dubbed “commitment escalation”—almost four months of 17-hour days overseeing hundreds of engineers and programmers from more than 50 different companies and contractors.
What exactly was wrong with HealthCare.gov? As Dickerson put it, the project went exactly as normal considering government IT projects fail spectacularly all the time. The only unusual factor was that failure becoming national news.
“Over 50 different companies were hired to build this site,” Dickerson said. “When a different company is responsible for each aspect of the site, you’re trapped in an eternal loop of damnation. The site was down in real time, days ticking away, but surprisingly there really was no sense of urgency.”
Because the HealthCare.gov project was completely mismanaged from the get-go, Dickerson said there were a lot of easy, basic fixes. He and the four other “experts” brought in to assess the damage created a “war room” in the Virginia office, installing TVs and desks and renting out an entire floor so that the disparate site contractors were all working out of the same place, with Dickerson often running the twice-daily operations meetings. As it said in the team’s initial report to the White House, one of the first things to do was to add monitoring.
“We sent out this report. Bad news: Everything has been done wrong. Good news: Because of that, there are a lot of easy, basic fixes. Will that be enough to make the site work? There was no monitoring, no dashboard, no nothing. No place to look to find out whether the site was up today or not besides CNN, which is how we found out about a lot of problems in the beginning. You’re dead in the water if you can’t see what’s going on on the site.”