Nobody will thank me for saying so, but I have a bit of a mancrush on Ed Smith. Yet again, he has raised a profound point in the social sciences:

Nakedly ambitious people rarely achieve their ambitions...Simplistic self-interest is not just bad PR, it is often bad strategy. It suffers from a fatal flaw: it is predictable.

This draws our attention to the fact that, in some domains, randomness is the best policy.

This is most clear - and has been formalized by game theorists - in the case of games such as rock-paper-scissors where predictability leads to defeat. In such circumstances, the optimal strategy is to randomize.

To what extent do such games exist in the real world? They're common in sport. The penalty-taker who always aims low and left will find goalkeepers quickly wising up. The batsman whose best shots are all on the legside will soon get most balls pitching outside off.

I suspect they exist in other fields. If you're negotiating with a nutter who might randomly choose violence, you might well make concessions that a more rational counterpart couldn't extract; Colonel Gadaffi stayed in power for so long in part because of this, and it seems to have worked for North Korean dictators. One reason why interviewers at Oxbridge or Google have traditionally asked candidates unpredictable questions is to test their powers of thinking on their feet. And one sign of genuine entrepreneurship is the ability to do - or blunder upon! - the unexpected, to wrongfoot rival companies.

However, randomness doesn't just work when the other player is trying to anticipate our actions. It can also work where there's environmental uncertainty.

Take a simple example. Strategy A has a 60% chance of a payoff of 3 and a 40% chance of a payoff of zero. Strategy B has a 40% chance of a payoff of 3 and a 60% chance of a payoff of zero. A is obviously the strategy with the highest expected payoff.

But it doesn't follow that it is the optimum strategy. If the payoffs are the number of descendants the members of a species has, then when that 40% chance comes up, followers of strategy A go extinct.

To maximize the chances of survival, a mixed strategy is needed. The species that randomizes will therefore survive. The one that "optimizes" does not*. This was the failure of the banking system in 2008. In trying to "optimize" banks ran the risk of extinction.

As John Kay showed, very often our goals are reached not through direct optimization, but obliquely.

And as Bruno Frey argues in a new paper, there's much to be said for using random selection in numerous contexts.

This poses the question: why, then, is randomness not used more often? And why do almost no decision-makers admit to using it, preferring instead to emphasize their "careful judgment"?

I suspect it's because we have fallen into a practice of cargo cult optimization and management. In business, politics and even in parts of our personal lives, we use the gestures and language of direct control in the hope that these suffice to achieve our aims. But sometimes, they don't.

* The scare quotes are because I want to duck the question of whether randomization is a form of rational maximization or an alternative to it. In biology, for example, mixed strategies are not consciously chosed by species but rather emerge by selection if they do so at all.