Burton Malkiel claimed that "a blindfolded monkey throwing darts at a newspaper's financial pages could select a portfolio that would do just as well as one carefully selected by experts". He was too harsh - on the monkeys. Subsequent research has shown that they often beat experts*. For this reason, I recently suggested that stock-pickers should employ some randomization.

Two things I've seen since then have broadened the case for random choice. David Van Reybrouck revives the argument for sortition. And Steve Levitt shows that people who decide upon a difficult choice (such as whether to leave a job or partner) by tossing a coin are significantly happier six months later.

This poses the question: when should we use randomization?

One circumstance is where there is significant indeterminacy - when we simply don't have good reasons for preferring one option over another. This is the case (subject to caveats) for stock-picking: in many cases, we just can't predict which stock will out-perform. It might also be true when picking candidates for a job. if we can't be sure who is the best, we might pick one at random from those who meet the desired criteria.

Such cases might be common in economics. Keynes thought that investment decisions depended upon animal spirits not because businessmen were irrational but because rational deliberation was in fact impossible:

The outstanding fact is the extreme precariousness of the basis of knowledge on which our estimates of prospective yield have to be made. Our knowledge of the factors which will govern the yield of an investment some years hence is usually very slight and often negligible.

Another circumstance is where reason might go wrong. Levitt thinks this is the case in his study. He believes that the status quo bias leads people to stick with unsatisfactory jobs and relationships, so that choosing at random can avoid that error, thus forcing people into decisions that make them happier.

I'd add another example. If our judgment proves correct in one or two instances, it can lead us astray. This is because correct calls can embolden us to become reckless in later decisions. This is what happens to investors. It might also have afflicted Tony Blair: his successful military interventions in Kosovo and Sierra Leone led him to believe he could repeat those successes in Iraq.

All this poses the question: if there's a case for randomization, why is it so rarely used? You might think it absurd to decide matters such as whether to go to war by lottery**. And the excessive use of lotteries could lead to the sort of unpredictable world described (pdf) by Borges.

I suspect, though, that there are two bigger reasons why randomness is so rarely used. One lies in a version of the narrative fallacy; we need explanations, which randomness doesn't give us. As Jon Elster put it:

Human beings are...reason-seeking animals. They want to have reasons for what they do, and they create reasons where none exist. (Solomonic Judgements, p56)

A second reason lies in overconfidence***. This causes us to over-estimate the scope and power of our reason, and thus to apply it where it is in fact useless or even counter-productive. In this sense, the same overconfidence that leads us into error also causes us to fail to see that we sometimes need protection from that overconfidence.

* This is because most stock market indices are weighted by companies' market capitalization and because small stocks often beat larger ones. This means that that most stocks often beat the market, implying that a random selection will beat a closet tracker fund, which is what many funds actually are.

** Or is it? The lotteries needn't contain equal chances: they could be weighted to give particular low probabilities.

*** One virtue of Channel 4's Naked Attraction is that it reminds us of Kahneman's claim that overconfidence is ubiquitous: if you looked like those guys, would you really want to appear naked on TV?