Simon Wren Lewis asks:

There is no doubt that Lawson was far better qualified than many other Chancellors, and also that he was clever, so why did he get these major judgements wrong?

It's a good question because it broadens. Clever people gave us the poll tax, the invasion of Iraq, financial crisis and collapses of Silicon Valley and Credit Suisse.

So why do they so often get it wrong. There are many reasons.

One, which Simon attributes to Lawson, is arrogance. In ignoring his advisors, Lawson did not go as far as Kwasi Kwarteng (double first, PhD and Browne medal) who sacked one of his. And observers agree that one of the defining features of Fred Goodwin which contributed to the collapse of RBS was his arrogance. Being clever tempts men to ignore Burke's advice:

We are afraid to put men to live and trade each on his own private stock of reason; because we suspect that this stock in each man is small, and that the individuals would do better to avail themselves of the general bank and capital of nations and of ages.

Instead, they are prone to what Daniel Kahneman has called an illusion of skill. They believe their intellect equips them to foresee or control events which are largely unpredictable and uncontrollable in a complex world.

We see this in finance. Although intelligent investors do perform slightly better than others, they still make errors - such as buying expensive but poorly-performing funds - because they think they have the ability to spot good assets.

Closely related to arrogance is the failure to see that intellectual ability is often context-specific, that there are no general purpose experts. They forget Charlie Munger's advice: "one skill is knowing the edge of your own competency." Many men have made fools of themselves by spouting off outside their field of expertise: think of William Shockley, Bobby, Fischer, James Watson or Richard Dawkins...

Failing to know one's limitations isn't the only way intelligent people go wrong, as the story of Vicky Pryce shows. She went to prison because her great intellect was eclipsed by her hatred for her ex-husband. Which is a dramatic exemplar of a widespread tendency, for our intellect to fail us in times of stress. The better financial advisors tell people to never take decisions when we are emotionally aroused because this clouds our judgment and so "produces substantial financial costs".

Warren Buffett has famously warned of this. "Investing is not a game where the guy with the 160 IQ beats the guy with a 130 IQ" he has said. "You don't need a lot of brains to be in this business...What you do need is emotional stability."

The ancient Greeks had a good word - phronesis, which referred not to mere intellect, but to the ability to apply that intellect appropriately in the right context. You all know clever people who lack that skill.

One reason they lack it is that intellect is not the same as biasfreeness. In Human Inference, one of the earliest books in the cognitive bias literature, Richard Nisbett and Lee Ross wrote:

There is no inferential error that can be demonstrated with untrained undergraduates that cannot also be demonstrated in somewhat more subtle form in the highlt trained scientist.

And David Robson has written that "greater education and expertise can often amplify our mistakes while rendering us blind to our biases."

One such bias is groupthink. In The Blunders of our Governments Anthony King and Ivor Crewe say this was true of the emergence of the poll tax. The people charged with reviewing local government finance were very bright - William Waldegrave was a fellow of All Souls - but they also, say Crewe and King, "developed a considerable esprit de corps". And this can drain any group of self-criticism:

Some of the best and brightest in British government - none of those intimately involved was remotely a fool - nevertheless contrived to produce one of the worst and stupidest pieces of legislation in modern British history.

But we don't need groups to turn smart people into bad decision-makers. We can do it on our own. There is, for example, no link between intelligence and the myside bias. In fact, greater intelligence makes us more able to see the flaws in opposing ideas and more able to defend our own with sophistry. It's perhaps for this reason that George Orwell said that "some ideas are so stupid that only intellectuals believe them."

What he's getting at here is ideology. Of course, we all have ideology - yes, even (especially) "moderate" and "reasonable" centrists. What we must not do, though, is let our ideology distort our assessment of the facts. This is what Lawson and Kwarteng failed to do: the ideology that tax cuts would swiftly boost potential supply led them to underestimate the impact they would have on inflation, inflation expectations and interest rates. Similarly, proponents of the poll tax let the ideological view that users should pay for public services colour their judgment about the feasibility of that tax.

It's not just partisan ideology that can hinder our intellect. So too does professional deformation, the tendency of our professional training to distort our perceptions. Academics, for example, might be antipathetic to my point here because they are apt to overweight the importance of intellectual ability. In the same way, engineers to often see everything as a control problem; lawyers over-estimate the role of law in social change and - yes - economists can exaggerate the importance of incentives.

The same applies to politicians. Even the very smartest of them, merely by virtue of being politicians, will be biased. They are disproportionately likely to be over-optimistic about the ability of good policy to improve things; to exaggerate the competence and knowledge of top-down controllers (which contributed to the Iraq disaster); to under-play the value of emergence and freedom; and to overvalue hard work.

Raw intellect, then, is on its own useless and sometimes downright dangerous.

This is not to say decision-makers should be chosen for their stupidity: if this were so, we would now be living in a golden age of governance. What it is is an argument against epistocracy.

It's also an argument for ensuring that intellect exists not in individual brains but in institutions and habits. Except for a little seasonal investing, my investments are all passive: I merely make a regular monthly payment into tracker funds. I do this because good habits matter more than thinking about investing with all its pitfalls. The same should be true in companies. The best ones have organizational capital, routines which help limited people do good things. Well-functioning markets also do this, by weeding out egregiously incompetent companies. And the same could be true in politics: deliberative democracy could in principle mobilize the general bank and capital of wisdom.

But of course, our actually-existing institutions are far from ideal. The marketplace of ideas - of which the media is a big part - selects against good ideas and in favour of bad. Economic policy is bad not (just) because politicians are stupid but because there are powerful pressures from the forces of conservatism operating against good policy. Companies sometimes select in favour of psychopaths (pdf) and the overconfident. Academics produce bad research not because they are stupid but because of the compulsion to "public or perish." And the financial crisis was caused in part by bad incentives: bankers were paid well to ignore risks.

Which brings me to my point. We must avoid the schoolteacher attitude to politics and business, marking the work of politicians and businessmen as if it were a test of intellectual ability and singling out the best and worst students. Instead, we must consider institutions. Do we have those institutions which help filter out incompetence and bias, or which are resilient to error? The answer, for now, is: no.