Fixing HealthCare.gov

The HealthCare.gov debacle seems like a textbook case of everything that can go wrong in a large software development project, especially in the public sector where the procurement rules make it almost impossible to apply the hard lessons the software industry has learned in the past forty years—although some mistakes are inexcusable, like failing to appoint a single executive who was responsible for this and this alone, all day every day, with the authority to get it done. That's the quickest way known to mankind to screw up a project like this.

I might blog on that and other mistakes at length some other time, but for now I'm more interested in what happens next. I can't help but laugh at the Obama administration's promises that it will all be fixed by the end of this month.

I have been designing, planning, and implementing large, complex, mission-critical government systems for the past twenty years. If I had been planning this project, I would have allowed at least three months for testing and fixes. By the end of this month, they won't even understand what all the problems are, or even have a concrete plan to fix it, let alone have time to actually fix the problems.

Testing systems that have thousands of inter-dependent functions in them is hard, and it requires a very methodical approach. You need a plan that ensures you test as many of the different paths through the process as possible. If the system has lots of rules and options and interfaces with other systems, and this one does, testing becomes a major project in its own right, and unless you apply the same discipline to testing any fixes, you run the risk of introducing more bugs without knowing it.

I don't have any inside information, but I ask myself what's going on inside this project right now, and my answer is that they are all running around like headless chickens in response to the enormous political pressure they must be feeling. Which means they are just spinning their wheels and that each week that passes doesn't bring the problem a week closer to resolution, it just pushes the true resolution date out another week.

My prediction is this thing won't be working in any meaningful sense of that word for at least three months and probably six.

Incompatibility

If you want to understand the key to programmer productivity (and any other task that involves the manipulation of a large number of inter-dependent ideas) read Joel On Software.

What it comes down to is this - interruptions are really, really, really bad for productivity when you are doing this type of work (because every time you are interrupted you need to reassemble all the ideas you were holding in your head in exactly the relative positions they were at the point you were interrupted). The lack of interruptions is one of the main reasons I like working from home (the other reasons are being close to my espresso machine, my wife and the ski lifts, but not necessarily in that order).

However when I'm visiting one of my company's offices I usually end up working at a cube in what we call a 'business centre'. The idea is widespread in big consulting companies now and is usually referred to as 'hot desking'. It's driven by bean counters who believe that it saves money which on the face of it is true because you can squeeze a lot more people in the same amount of space. Of course it's only a saving if you completely ignore productivity, which in my experience is about 25% of what it is when you have a quiet private space (like my home office), at least if you measure productivity in terms of producing the stuff that we deliver to clients and actually get paid for as opposed to 'busy work'.

But someone in our office in Wellington, NZ has taken the concept to new levels of stupidity because the 'business centre' is also the coffee break room. Try preparing a presentation to a client to close a multi-million deal while a dozen people are taking their lunch break around you. I really wonder how the guy that came up with this can justify taking a salary.

Distortion is rewarded, that's why it happens

Quite by accident I ran across an interesting article today by Edward Tufte. If you are not familiar with Tufte, he's probably the world's leading authority on how to present data for maximum impact with minimum distortion. I purchased a copy of his seminal work The Visual Display of Quantitative Information many years ago from a brick and mortar bookstore but you can conveniently buy it at Amazon.com.




Anyway, Tufte was engaged by the Congressional investigation into the explosion on re-entry of the space shuttle Columbia. In the article he is quite critical of the use by Boeing consultants of Microsoft PowerPoint to communicate the findings of their investigation into the inital collision on launch of a piece of insulating foam with the wing of the shuttle (which ultimately led to the disaster).

As bad as the Powerpoint presentation is, and all of his criticisms in the article are perfectly valid, I think he misses the point. PowerPoint is simply the medium. I don't beleive it caused the mis-communication, it simply facilitated it. (1)

I say this because I stumbed across the article after a day spent frustrated at a similar phenomenon (which not co-incidentally also has as its source a government agency), and it's not the first time I've had this type of experience.

Imagine trying, as I have been, to design a complex hardware and software system which must balance multiple competing objectives (cost with functionality with usability with security with compatibility and so on). A system which will cost millions of dollars, be used by thousands of employees and affect hundreds of thousands of the agency's clients and be distributed across hundreds of locations nationally and globally.

Now imagine being asked repeatedly, as I have been, to reduce your design to a single PowerPoint slide. What's more, imagine being asked to avoid using any of the standard notations that the software industry has developed to ensure semantic precision in its models, but instead to limit yourself to the standard slideware pallete of boxes and arrows.

In doing so you have to ensure that your slide communicates the complex design patterns that you have adopted to incorporate everything the software industry has learnt in the past 40 years. But you also have to keep it simple because the people selecting this multi-million dollar system don't have any knowledge relevant to evaluating your design.

Here's my point. In the end people sell what people buy. The Boeing consultants who created the PowerPoint slides that Tufte so rightly criticises did so because that's what their market demanded. The root cause of the problem is systemic and has to do not with deficiencies in the people doing the analysis, but in the processes which weed them out in favour of those who can spin a line to those too ignorant to know better, but unfortunately empowered to decide.

Had they wanted to use PowerPoint to communicate a clear message that forced people in positions of authority to make hard decisions they certainly could have. The problem is you only get to make such a presentation to politicians and their lackies once.


(1) Having said that, let me add this. Anyone who uses one of the standard PowerPoint outlines for a serious presentation is an idiot. I've used PowerPoint a thousand times, and I've yet to have a message to communicate that fitted one of those cheezy outlines. Perhaps when I start selling real estate (and hell freezes over) that will change, but until then create your own outline, one that logically follows the train of thought that led you to whatever conclusion you want to present.

Goodbye Peter Drucker

Noted management writer and thinker Peter Drucker died last Friday aged 95. The world will be a poorer place without Drucker who virtually invented modern management theory in the 1930s.

What I most admire about him was that he represented the triumph of substance over appearance. Long after he became famous he remained at a small college where he could focus on his work, he was never interested in the received wisdom or popular theories but instead had his own deeply considered views on things and he was much more interested in understanding the daily grind of making a company efficient and profitable than the cult of leadership that infects modern business.

And of course he managed to stay active and engaged and continue to make an real contribution at an age when most of us are long gone (if not physically then at least mentally). A wonderful role model for anyone who believes in the importance of ideas and intellect.

Virtual workers of the world unite!

I work for a large multi-national firm which has its head office outside Philadephia and my division is headquartered outside Washington DC. But I live in Steamboat Springs, three hours drive from the nearest office in Denver, where I have never been. I'm what my company calls a virtual worker.

This is something that has been talked about for a while now, but I suspect many people are inclined to dismiss it as a niche phenomenon. Well a little while back Routt County, where I live, commissioned a study to identify what proportion of its residents work for an employee that is not located here. The answer of 10% surprised even me, but in retrospect it shouldn't have, especially given that Steamboat is a very attractive place to live but has a fairly narrow local employment base. There's plenty of jobs for retail clerks, construction workers, coal miners and ski instructors but not people like me who have a deep specialisation that can only generate returns in a national or global market.

How many people go to an office every day and spend most of it behind a computer screen or on the phone? Don't they have these things at home?

If this is you, ask yourself the following questions. Can I live without the commute? Am I living here because it's where I really want to be, or simply because this is where my office is located? Can I live without the endless hours spent in pointless meetings? Can I live without needing to wear a tie most of the time (ok, I hate ties).

Then ask your boss why you're not a virtual worker!

Explaining regression and CEO compensation

There's an interesting discussion posted at Marginal Revolution on how you go about explaining regression analysis to people not trained in statistics in the particularly challenging context of a late night television show where the attention span for anything not involving naked women is probably 15 seconds or less.

I don't think the writer nailed it, so here's my attempt:

Most real world outcomes are caused by multiple factors. To understand the effect of any one factor we need a way to control for the other factors, that is, we need a technique to let us figure out how much of the change in the result is caused by changes in the factor we are studying.

Or as Bert and Ernie demonstrated in an excellent Sesame Street episode, the fact that there are no alligators in Sesame Street does not mean Ernie's technique of sticking a banana in his ear to keep the alligators away actually works.

One of the most egregious violations of this principle occured during the dotcom boom / late 90's stock market bubble, namely the linking of CEO compensation to stock prices without controlling for the general effect of a rising market. Shame on those boards who lined the pockets of their CEOs with millions of dollars in bonuses and stock options when they should have been compensating them only for that part of the rise in the stock price that was caused by the CEO's performance.

What must it have been like to sign such a contract? Well, for a CEO who understands statistics, it must have been like knowing in advance that you've bought the winning lottery ticket.