Should ineliminable incompetence play a bigger role in economic and political thinking?

My trigger for asking is a piece in the Times by Oliver Kay on the appalling mismanagement of football clubs such as Blackburn Rovers, Charlton, Leeds and Blackpool. But of course, incompetence is much wider than that. Trains are late and overcrowded; building projects run over time and budget; utilities, banks and broadband providers often have poor quality service that can't wholly be explained by profit-maximizing; and you all have examples of bad management in your own workplace.

Which brings me to a paradox. Whereas our own eyes tell us that incompetence is ubiquitous, standard economic theory regards it as merely a temporary deviation. It thinks that agents are incentivized to optimize; that badly managed assets will be bought cheaply by people better equipped to run them; and that competition will drive incompetent firms out of business.

But this doesn't happen - at least not fully. Even the best incentives can't put in what God left out: I have an incentive to become a Premier League footballer or Nobel laureate, but you'd be ill-advised to bet on me becoming either. Credit constraints, among other things, prevent assets flowing to their best managers. And Bloom and Van Reneen have shown (pdf) that, in all countries, there is a "long tail of badly managed firms", which tells us that competition doesn't fully weed out idiots.

The failure here, though, isn't merely one of Econ 101. Real people make the same mistake.

We know that stock-pickers pay too much for growth (pdf) stocks and ones on the verge (pdf) of collapse. One reason for this might be that they underestimate the ubiquity of incompetence - the fact that managers can't grow firms or turn them around as easily as thought.

And in politics neoliberals overstate the extent to which incompetence can be removed by competition and/or well-paid management whilst the left exaggerates the benefits of nationalization. Not only do we over-estimate state capacity, but we also over-estimate management capacity*. Perhaps we underestimate the extent to which success is a matter of luck and then habit plus marginal gains.

Even the English language misleads us here. Think of the synonyms and near-synonyms for incompetence: mistakes, errors, ineptness, ineffectual, bungling, incapability and so on. All of these suggest a shortfall from a norm of competence. But perhaps the opposite is the case - that it is incompetence that is the norm. As Paul Ormerod said (pdf):

Failure is all around us. Failure is pervasive. Failure is everywhere, across time, across place, and across different aspects of life...Yet the existence of failure is one of the great unmentionables.

Here, we - or at least we economists - perhaps need a paradigm shift. It's a cliché that economics has a powerful theory of optimization. But we also have a powerful theory of incompetence. The cognitive biases programme explains how it persists. For example, the planning fallacy predicts that complex projects will over-run budget; the overconfidence effect tells us that assets will sometimes be bought not by those best able to run them but by those who over-estimate their ability to do so, such as the Venkys at Blackburn; Bayesian conservatism predicts that bosses will under-react to feedback; and so on. I might add that the optimism bias causes us to under-rate these mechanisms.

My point here is perhaps an old conservative one - that our institutions, public and private, are deeply imperfect and might well remain so, that there is, as Adam Smith said, "a great deal of ruin in a nation." Remembering this might not be good politics: there's a danger it'll reinforce the status quo bias. But it would be good for our mental health.

* One thing I found irritating about much of the coverage of the collapse of BHS was the lack of distinction between the failure of the company, which was forgivable and perhaps unavoidable, and the plundering of the pension fund, which was neither.