Stumbling and Mumbling

"We'll have to look at the data"

chris dillow
Publish date: Sat, 14 Mar 2015, 01:35 PM
chris dillow
0 2,773
An extremist, not a fanatic

Here are some things I've seen this week:

- Describing the decline of universities, Marina Warner says: "not everything that is valuable can be measured."

- Some US newspapers have signed up to Blendle, allowing readers to pay to read individual articles.

- Dave Tickner says that Peter Moores' response to England's being knocked out of the World Cup - "We'll have to look at the data" - "sums up everything's that's wrong" with his management.

- South Yorkshire police diverted funds from sex abuse inquiries towards tackling car crime, which the Force was targeting.

- Mike Goodman says that much of Mesut Ozil's contribution to Arsenal doesn't show up in statistics.

These apparently disparate things all bear upon the same question: can information be fully quantified, codified and therefore centralized, or were Hayek and Polanyi right to claim that some knowledge is inherently fragmentary, tacit and dispersed?

This is of course not merely a matter of epistemology. It bears directly upon how organizations should be structured. If everything can be measured by a central authority then hierarchy is feasible. If not, then we might need more decentralized forms of organization. It's no accident that the increased use of metrics in universities has coincided with the rise of vice-chancellors' salaries. Claims about knowledge are also claims to power.

As with most questions in the social sciences, the answer here is: to some extent. Given the ubiquity of cognitive biases, data can tell us what really works; this is the message of Moneyball. For example, stats disprove the claim that Ozil is lazy. And targets and quantification can be used to identify lazy academics and policemen.

However, good ideas can be pushed too far, with counterproductive consequences: revenge effects are common. There are at least four different mechanisms through which this can happen:

- Gaming. David Boyle has claimed that school league tables led teacher to focusing excessively upon D-grade students at the expense of others, because converting D to C grades improved schools' performance. Similarly, South Yorkshire police were told to target car crime, they did just that to the detriment of tackling what we now know to have been more serious crime. This was an example of the drawback of centralized over dispersed knowledge; the belief that sex abuse was a serious problem was a hunch of a few junior officers, whereas knowledge of car crime was more quantified and centralized.

- Excessive investment in measured outcomes. This happens when academics are encouraged to write grant applications rather than teach students. And it'll happen if journalists are paid per click: we all know that the way to attract eyeballs is to write about celebs or shrill partisan pieces.

- Some stats are just bad and can give a mere illusion of knowledge. For example, in 2007-08 banks' risk models were based on data which over-sampled low volatility and under-sampled high. The upshot was that the crisis came as a shock. David Viniar, Goldman's chief financial officer, famously said: "We were seeing things that were 25-standard deviation moves, several days in a row." But in fact, a better inference would have been that risk was mismeasured.

- Adaptive markets. Companies or sports teams might get a competitive advantage from using statistical methods but they cannot retain it, simply because others will emulate them. This too is a message of Moneyball.

Herein lies my problem. I fear managers are underestimating these drawbacks. This might be simply because of deformation professionnelle - the tendency for one's professional background to warp one's perspective. Or it might instead be an example of Upton Sinclair's famous saying: "It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

More articles on Stumbling and Mumbling
Discussions
Be the first to like this. Showing 0 of 0 comments

Post a Comment