David Aaronovitch in the Times has a nice piece on how people fall for conspiracy theorists and conmen:

What a fraudster, a fantasist or a hoaxer needs to be able to survive, is to fall among people who are anxious to believe.

This urge to believe is not confined to extreme cases. It's much more common than that.

A while ago, Guy Mayraz did a neat experiment at Oxford. He asked subjects to use charts of the price of wheat to predict future price moves. Before doing so, he randomly divided them into two groups: "farmers" who would profit from a rising price, and "bakers" who would profit from a falling one. He found that farmers predicted higher prices than bakers - which is evidence of wishful thinking. What's more, this bias persisted even when subjects were given incentives for accurate predictions.

This finding applies in the real world. Brad Barber and Terrance Odean have shown that retail equity investors under-perform the market around the world in part because they tend to hold on too much to losing stocks and thus suffer adverse momentum effects; this is the disposition (pdf) effect. I suspect that one reason they do this is wishful thinking: the mere fact that they hold a share makes them unduly optimistic about it. (This is closely related to the endowment effect).

If wishful thinking exists even when people have material incentives to avoid it, it is even more likely to exist where they don't. I suspect, therefore, that it is common in politics. Political partisans are prone to over-estimate the beneficial effects of their party's policies and to under-rate the sheer difficulty of manipulating complex emergent processes such as society or the economy for the better.

Maybe, therefore, wishful thinking is more widespread than David suggests.

And in some ways, this is a good thing. In Human Inference, Richard Nisbett and Lee Ross wrote:

We probably would have few novelists, actors or scientists if all potential aspirants to these careers took action based on a normatively justifiable probability of success. We might also have few new products, new medical procedures, new political movements or new scientific theories.

And Donald Davidson has written:

Both self-deception and wishful thinking are often benign. It is neither surprising nor on the whole bad that people think better of their friends and families than a clear-eyed survey of the evidence would justify. Learning is probably more often encouraged than not by parents and teachers who overrate the intelligence of their wards. Spouses often keep things on an even keel by ignoring or overlooking the lipstick on the collar. (Elster (ed, The Multiple Self p86)

And here's the point. Because wishful thinking (like overconfidence) often has beneficial effects the learning and adverse feedback that would cause us to correct our errors is absent. Cognitive biases persist because they are not selected against, and might be selected for.

I say all this for a reason. It's very easy for those of us who write about cognitive biases to commit the Homer Simpson error: "Everyone is stupid except for me." In fact, we might in some respects be as irrational as everyone else.