It is a truth universally acknowledged that the best lack all conviction whilst the worst are full of passionate intensity. Which is another way of describing the Dunning-Kruger effect - incompetent people don't know they are incompetent. They are overconfident.

However, a new paper by Peter Schanbacher suggests that, in many cases, what looks like overconfidence might in fact be quite rational.

To see his point, take a different case. Imagine a coin is biased to land on heads 80% of the time and tails 20%. Predict the next 10 tosses.

A good answer to this question would simply be 10 heads. This will be more likely to be right than any random-looking sequence of eight heads and two tails. But this answer is biased - it overpredicts heads.

The lesson here is that there can be a trade-off between bias and variance. The "10 heads" answer is biased, but it has less variance (is less likely to be wrong) than (say) HHHTHHHTHH.

A similar problem faces the ignorant individual when asked about his chances of solving a problem correctly. The same lack of expertise that makes someone ignorant can also mean there is large variance around his estimate of his competence. One rational solution to this, says, Schanbacher, is for him to give a biased answer - to over-estimate his competence - just as a biased answer to our coin-toss question is reasonable.

In this sense, overconfidence might be a good answer to a bias-variance trade-off.

This is not the only way in which overconfidence might be rational:

- Overconfidence might be the only way of steeling oneself to take risky decisions, such as to start one's own business.

- Overconfidence can be mistaken for genuine skill. Being overconfident, then, is a good career strategy.

All this raises a distinction which is often overlooked.

I suspect that the large majority of people who are overconfident have not solved a bias-variance trade-off or chosen a career-enhancing strategy. They are just mistaken. In this sense they are irrational, in the sense of having a belief for which they do not have evidence. However, they act as if they are rational, in the sense of doing something which maximizes their chances of success.

In other words, rational behaviour is more common than rational thinking.