"Economics research is usually not replicable" say (pdf) Andrew Chang and Phillip Li (via).

This problem is not confined to economics: it also seems to be the case in psychology (pdf) and medicine. Perhaps, therefore, the problem is a general (ish) academic one: pressure to publish and win research grants lead researchers to, ahem, tweak their findings.

Nevertheless, it poses a problem for those of us who are consumers of economic research: how should we respond to this?

First, we should remember Dave Giles' Ten Commandments, in particular:

Check whether your "statistically significant" results are also "numerically/economically significant"...
Keep in mind that published results will represent only a fraction of the results that the author obtained, but is not publishing.

Secondly, we should remember the Big Facts. For example, one the Big Facts in finance is that active equity fund managers rarely (pdf) beat the market (pdf) for very long, at least after fees. This, as much as Campbell Harvey's statistical work (pdf) reminds us to be wary of the hundreds of papers claiming to find factors that beat the market.

To take another example, Pritchett and Summers remind us of another Big Fact (pdf):

The single most robust empirical finding about economic growth is low persistence of growth rates...Episodes of super-rapid growth tend to be of short duration and end in decelerations back to the world average growth rate.

This warns us to be sceptical of findings that national policies, institutions or cultures have significant long-lasting effects on growth.

Thirdly, we should ask: is this paper consistent or not with other findings?

Take this paper which claims that fund managers perform badly after they have suffered a bereavement. This seems a mere curiosum. It's not. It's consistent with experimental evidence that sadness increases present bias, and with another Big Fact - that stock markets do better in winter, perhaps because longer nights in the autumn depress investors and share prices.

One of my favourite examples here is momentum. When I first saw Jegadeesh and Titman's claim that shares that have recently risen tend to carry on rising whilst fallers continue falling, I thought it was an interesting curiosity. But a similar thing has been found in currencies, commodities and even in sports betting. All this suggests that momentum is a reasonably robust fact.

But here we have an inconsistency: how do we reconcile this claim with the Big Fact that fund managers don't often beat stock markets?

This brings me to my fourth principle. We must ask: is there a sound theoretical reason for these findings, which reconcile them to the Big Facts?

Initially I thought momentum's out-performance was possible because fund managers were looking for accounting-based anomalies rather than behavioural ones. This, though, has become less plausible given the interest in behavioural finance since the 90s. But there is another possibility, pointed out by Victoria Dobrynskaya. It's that momentum stocks often have "bad beta"; they carry benchmark risk which makes them unattractive to fund managers with the result that they are under-priced to reflect such risk.

Now, here's the thing. It is, I suspect, rare for papers to satisfy these criteria. Of course, a lot of findings are worth thinking about. But few are actionable. And those that are are ones which fit other findings, Big Facts and theory.

My principles, I hope, are a form of informal Bayesianism. They are intended to steer the middle ground between a bigoted cleaving to our own prejudices on the one hand and gee-whizz gullibility on the other.

You might wonder why I've taken my examples from financial economics. It's partly because my ignorance of this field is less profound than others. But it's also to remind you of another Big Fact - that economic research is not, and should not be, about armchair windbaggery but rather about how to help people make better real world decisions.