The Hubris of Analysis – Commentary by Aaron Katz

OK, first, the mea culpa. I am a policy analyst, I teach policy analysis, I have led myriad analyses big and small that seek to help policy makers make decisions based on what we know.

We look to the past to help us determine the future. That is, we take observations – analysis, data, stories – of what has occurred to create models of what might or should occur. The former is “known” or measurable (granted, history is often a subject of debate), the latter is “expected” or “projected.”

The future often involves many factors and forces only some of which we know and even fewer have influence over. So, we’re hampered by our inability to actually know the future or, as climate experts Tom Knutson and Robert Tuleya have said, “Observations of the future are not available at this time.”

My point is, beware the projection. And in health policy we have many. Just think of the ACA – the plethora of projections by the Congressional Budget Office and every flavor of interest group about how much the law would cost, how many people would enroll, what premiums in health insurance exchanges would be.

Remember the screams about the law’s goal to reduce excess to Medicare Advantage plans payments (estimated by MedPac as 14% more than equivalent spending for traditional Medicare), the managed care plans offered as an alternative to traditional Medicare? The insurers said, “Cut our payments and we’ll have to cut the extra benefits and increase beneficiary premiums.” That, said the analysts, will surely lead to lower enrollment in these plans.

Wrong. The CBO projected in 2010 that MA enrollment would drop from about 11 million that year to about 9 million this year. The “observed” future? Nearly 16 million MA enrollees in 2014. What we think we knew (do I sound like Donald Rumsfeld?) – that lower subsidies would lead to higher prices and therefore fewer buys – was not sufficient to accurately predict the future. Now CBO projects that 22 million people will be in MA plans by 2020, despite the ACA’s phased reductions in payments through 2017 … but don’t bet on it!

My point is not that we should cease analyzing and projecting, but that we should be very humble about how much stock we put in those prognostications. For any policy initiative, but especially for one as complex as the ACA, we should analyze, project, and act… but then expect to make frequent course adjustments as we realize the limits of our knowledge.

In a functioning democracy, anyway, that’s what we’d be doing.