The more I learn about the rollout of healthcare.gov, the sicker I get. According to Reuters and a June GAO report, the website has so far cost almost US$300 million, almost two-thirds of which involves money thrown at the train-wreck development over the past six months.
In the realm of national expenditures, $300 million barely warrants a mention, and what conversation there has been about the website has focused on its functional problems and, to a lesser degree, the 3x cost overrun. But reflect on the absolute number of $300 million, your own software budget, and your own experience of the industry. And let’s remember: This is green-field code for a website.
(Another messy software development issue: Architecture’s Big Ball of Mud)
My gold standard for spectacular incompetence came in the early 2000s, when I was brought in to deal with performance on the Web-based configurator for a Fortune 100 company. The project was three years old and had managed to burn through $30 million. To get a price, you had to set your browser’s timeout to five minutes and get yourself a cup of coffee, but at least it worked! (Protip: Don’t build your labels from a 30-language string-localization XML blob in your database.)
When I first heard the status of that project, I wondered how it was even possible for a software project to burn through so much money. I learned there were 50 full-time developers and 14 architects, which I always thought represented an incompetence event horizon: One more architect or full-time developer, and the energy needed to make a change would cause space-time to collapse.
But leave it to the government to take things to a new level! Close to $200 million in overrun costs over six months: What miraculous innovations in software development must this embody? There’re only so many developers you can pay better-than-Silicon Valley average and bill out at 4x their salary, and only so many Alienware rigs and 4P displays you can buy them. It still doesn’t add up! No, my friends, only pure bureaucratic BS can absorb so much money in such a short time. (Like a “pure function,” a “pure bureaucracy” consumes resources and has opaque internals, but a pure bureaucracy produces no value.)
So far, my favorite artifact is the “Periodic Table of I.T. Build Work Streams” from an article in the Washington Examiner. I bet it cost a million dollars, what with the salary paid to the nothing-better-to-do middle manager who conceived it and the design consultancy that probably billed out a couple hundred hours on fonts and color selection. And what does it mean? I thought I knew how to decode every single worthless project-management chart used in software, but I have no idea what to make of a trans-Uranium I.T. Build Work Stream. Is it an extremely short-lived by-product of colliding bureaucratic processes? (And, at the risk of just spluttering, can’t you just picture someone deciding, “Looks like we need another contractor in the noble gasses or it will ruin the symmetry”?)
Unable to bear thinking about the price anymore, I thought I’d take a look at the technology. According to this comical-in-retrospect Atlantic article from June, healthcare.gov is a static site, generated by Jekyll. That’s clearly only a part of the story, but although www.healthcare.gov/developers says that the source code for the website is open source, the GitHub link was returning a 404 as of this writing.
(Speaking of logging complaints: Let them know what you think)
An initial analysis of the static part of the website shows mistakes both trivial (too many connections downloading non-minified JavaScript) and petty (removing attribution in the Open Table library, a violation of its GPL v2 and BSD licenses), but the real functional mystery of healthcare.gov revolves around non-static elements.
In fact, according to the previously mentioned GAO report, more than 55 contractors were involved in the process. The major one was CGI Group, which had billed $88 million by June of this year. Their code is not open source nor have they been forthcoming about their technology stack. (Silent regarding healthcare.gov, their website’s press page instead contains such gems as “Railroad Commission of Texas taps CGI to fuel performance gains.”)
The finger-pointing has begun and things are obscure, but one theme seems to be a lack of integration testing between the Jekyll-generated front end, an identity validation module (provided by Experian), and CGI Federal’s components. There’s also the usual nonsense, from the disingenuous (“We were shocked—Shocked!—to discover the requirements were incomplete”) to the perverse (people claiming that the project was underfunded).
CGI Federal has an additional federal billing pipeline of $8 billion. I have a headache.
Larry O’Brien is a developer evangelist/advocate for Xamarin. Read his blog at www.knowing.net.