Would balanced and impartial news and current affairs journalism be dangerous, even if it were done with maximum integrity and competence? I ask because of a recent paper by Benjamin Enke and Thomas Graeber.

We've known for a long time that, in the words of Glaeser and Sunstein, balanced news can produced unbalanced views. In a famous experiment (pdf) in 1979, for example, Charles Lord, Lee Ross and Mark Lepper showed people mixed reports on the effects of the death penalty and found that people who supported capital punishment became stronger in their support of it after reading them, whilst opponents of the death penalty also became more dogmatic in their opposition. Balance, then, can lead to polarization.

This mechanism, however, works on partisans - those who fancy themselves already informed. For those who are not informed, another mechanism can be at work, which is described by Enke and Graeber.

They call it probability compression. People who are uncertain of their own minds especially tend to treat low probabilities as if they were greater than they actually are. This is consistent with experiments (pdf) by Baruch Fischhoff and Waendi Bruine De Bruin at Carnegie Mellon University. They found that when students were asked to estimate low-probability events such as their chances of being burgled or getting cancer, they chose 50% probabilities more than they should. We also see it in gambling: punters back outsiders more than they should - over-rating small chances - and favourites (pdf) not enough. There's a good reason why online bookies so often advertise accumulators: punters over-estimate the chance of long odds paying off. And we see it in stock markets too. Investors have paid too much for the slim chances that small speculative stocks will come good: it is partly for this reason that Aim stocks have under-performed mainline ones since their inception in the mid-90s.

Probability compression is slightly different from the tendency described (pdf) by Kahneman and Tversky for very low probabilities to be generally over-weighted. Kahneman and Tversky say we over-value small probabilities because we have a taste for long-odds bets, whereas Enke and Graeber say we over-estimate those probabilities.

It's this mechanism that populist enemies of technocrats exploit. Michael Weiss and Peter Pomerantsev describe (pdf) how Vladislav Surkov, Putin's eminence grise, puts forward competing stories "not to persuade (as in classic public diplomacy) or earn credibility but to sow confusion via conspiracy theories and proliferate falsehoods." He is weaponizing relativism. Trump does something similar when he attacks "fake news". He's not intending to establish that a story is false, merely to sow a seed of doubt. So too do climate change deniers (described well by Rebecca Long Bailey) when they claim that the "science isn't settled", even though the overwhelming majority of scientists think it is.

In all these cases, people are exploiting probability compression. They suggest there is a small chance that the established narrative might be wrong, and invite listeners to over-estimate that chance.

This means that our hypothetically perfect provider of balanced news faces a nasty dilemma. On the one hand, it must present challenges to the consensus, because it is important that we avoid groupthink. But on the other hand, doing so invites people to over-estimate the small chances that the consensus is wrong.

For example, Simon has criticized the BBC for inviting Patrick Minford to discuss Brexit without telling the audience that the overwhelming consensus among economists is that Brexit would be harmful to the economy. Even if the BBC had done so, however, some of the audience would have inferred that the chance of the consensus being wrong was higher than it is.

This is all the more true because in this case (and others) probability compression is reinforced by other biases. One is a halo effect: the appeal of Brexit to many for non-economic reasons can spill over into a belief that it's also good economics. Another is plain wishful thinking; if you want Brexit to succeed, you'll easily persuade yourself that it will. And then there's the familiarity effect: a falsehood you are familiar with can seem more credible than a true claim that's not so familiar. A balanced report that repeats a false claim (perhaps if only to debunk it) can end up causing the claim to become too widely believed.

We all criticize the BBC for its failure to be properly impartial. My point here, however, goes further than this. It's that even if the corporation were immaculately impartial it might end up inculcating false ideas. The problem with our politics, then, is not (just) bad journalism but the fact that voters can be systematically irrational.