Healthcare gov

 

The ACA was signed into law on March 23, 2010, giving the government a little more than 2.5 years to build the healthcare.gov website. A lot was made of the fact that, during the election campaigns of 2008, the Obama campaign proved to be more technologically capable than its opponents, something many believe to have been a key element in Obama’s win. Unfortunately, development of the technology used in the 2008 and 2012 presidential campaigns wasn’t subject to the convoluted mess of regulations that dictate how the government procures goods and services.

 

What happens is you have a relatively small number of companies that have the expertise and resources (such as lawyers) to deal with government procurement. They’re the ones who write the proposals and are awarded the contracts and, in turn, subcontract with the companies (mostly large companies like IBM and Oracle, and some upstarts, like MarkLogic) to do the actual work. If you look at the publicly available information, you’ll see there were 50 direct contractors providing services related to the ACA implementation, with payments totaling about $450 million.

 

First lesson: The more entities involved in producing something, the greater the likelihood of failure. This seems like common sense, and yet the governmental procurement process virtually guarantees multiple companies (and layers within each of those companies) will be involved, making the kind of clear, rapid communication at the heart of nimble software development a virtual impossibility.

 

I’m amused when the government points to successes like the health care information systems that power Kaiser’s health system, or consumer-friendly websites like Amazon.com, and seek to emulate them. It took Kaiser 10 years and more than $4 billion to develop its system. Amazon has been refining its site since 1995. The 2.5-year schedule for healthcare.gov was ambitious to begin with. Even more so when you consider that both Kaiser and Amazon have complete control over their internal systems, and a lot of leverage when dealing with outside interfaces (for example, getting test data and images from outside labs).

 

On the other hand, while the public interface to the health insurance marketplace was being built from scratch, there were a large number of legacy systems with which its builders had to contend (Medicare, the Medicaid services in each state, insurance companies and so on). From experience, I can tell you that dealing with a legacy software system has two common problems. First, the interface may be primitive or nonexistent. These legacy systems were never designed to share data, and particularly not in real time. A lot of them are batch-oriented, which means information is usually between zero and 24 hours old, since updates (batches) are run overnight. Second, interfacing with outside parties can reveal hidden problems due to unanticipated and unfamiliar requests of the system.

 

The take-away: Modern websites aren’t as simple as they seem, and getting data in and out of another system isn’t as easy as you might think—and it gets harder the older that system is. In the programming biz, it’s known as a SMOP: a “simple matter of programming.” It’s a sarcastic way of saying that what looks easy and works well takes a lot of hard (and thoughtful) work behind the scenes. It’s true that it’s easier than ever to build a great website, particularly due to the availability of cheap cloud infrastructure (such as Amazon Web Services), which used to cost millions to acquire and operate in the late 1990s.

 

Not long after healthcare.gov launched, a small group of programmers made news when they claimed to have built a website that did everything healthcare.gov did in a couple of weeks. First, that ignores the very real problem of legacy system interfacing. But second, it ignores the problem of scale. It’s relatively easy to create a website that will handle one or two simultaneous visitors (although even going from one to two has its hidden pitfalls if you don’t understand concurrency). It’s much harder to build a site that can, on its first day of operation, handle thousands—or more—of simultaneous connections. Most sites do a “soft launch” so they have time to see and correct problems with loading as traffic increases over time. Not healthcare.gov—having a fixed and very public launch date virtually guaranteed a disaster.

 

There were certainly other idiocies at work as well. A project of this scale takes months of testing, and testing is the first thing to get squeezed or omitted. There was no experienced “integrator,” the group responsible for making sure all the individual pieces work together. There was an expectation that everything would work right out of the gate. And there was a lot of denial as the project got closer to deadline. It was an embarrassment for a President whose team seemed to be so good at technical stuff.

 

Eventually, healthcare.gov will work as well as any other high-traffic website (or nearly so). But its initial technical problems created a huge political nightmare for the Obama administration. Lest you think these problems are confined to government, Standish Group International found that, in 2012, only 10 percent of IT projects with a value of more than $10 million were successfully completed on time and within budget.

 

It’s 2014 (or will be shortly). Drop me a line at mduffy@northbaybiz.com and let me know what topics you’d like to see covered in this column during the coming year. It’s more fun for both of us that way. Happy New Year!