Stumbling and Mumbling

Diversity trumps ability

chris dillow
Publish date: Fri, 17 Oct 2014, 01:53 PM
chris dillow
0 2,773
An extremist, not a fanatic

Jon Lansman wants Labour MPs to be "ordinary people who have held normal jobs" rather than career politicians. There's a powerful piece of thinking on his side - the diversity trumps ability theorem. This is an extension of James Surowiecki's wisdom of crowds theory, but it has been mathematically formalized by Lu Hong and Scott Page, who summarise it thus:

When selecting a problem-solving team from a diverse population of intelligent agents, a team of randomly selected agents outperforms a team comprised of the best-performing agents.

I'll spare you the maths, but give you the gist. Let's suppose that we want to find the best possible policy, according to some objective criteria - that is, we are in the domain of epistemic (pdf) democracy (pdf). Suppose too that there is bounded rationality and limited knowledge and that each individual selects the best option using his own information set and decision rule.

In these conditions, each individual, if s/he is moderately competent, will find a local maxmimum - the best option, given his/her information and decision rule.

But local maxima aren't necessarily global maxima. And Hong and Page show that even experts might well not find that global maximum because their decision rules and information sets might not be wide enough to encompass the best option: this might be because of deformation professionnelle, or groupthink or simply because their Bayesian priors limit the number of options they search for.

Instead, widening the population of searchers increases our chances of finding that global maximum, because doing so brings more decision rules and information sets to bear on the problem. Cognitive diversity - in the sense of different ways of thinking - can therefore beat experts. It increases our chances of finding the best option. This might be why diversity within companies is associated with more innovation.

Note that this requires effective deliberative democracy, so that lesser options can be discarded in favour of better ones; simple "speak your branes" direct democracy is not sufficient.

Jon's call for an end to Labour's meritocratic preference for career politicians should be seen as an endosement of the diversity trumps ability theorem.

Now, I'm not saying this theorem is universally valid. The virtue of fomalizing it as Hong and Page have done is that it allows us to see more clearly when it is and when it isn't, and John Weymark shows that there are conditions in which it doesn't hold. My hunch, though, is that it might be sufficiently applicable to the Labour party to endorse Jon's call.

Herein, though, lies what some might see as a paradox. The diversity trumps ability theorem is the strongest part of the case for free markets. It is by increasing the diversity of firms that we increase our chances of finding good new products and more efficient processes. The fact that so much (pdf) productivity growth comes from entry and exit rather than from the growth of existing firms can be seen as corroroboration of our theorem.

In this sense, the case for free markets and the case against decision-making by tiny homogenous elites have a common root.

More articles on Stumbling and Mumbling
Discussions
Be the first to like this. Showing 0 of 0 comments

Post a Comment