It's a cliché that stock markets have been in panic mode recently. But why? The answer is not as obvious as you might think. And the issue matters not just for equity investors but for everybody interested in social science.

My chart shows the point. It shows that the All-share index has been largely predictable simply by the dividend yield. Since 1985 the correlation between the yield and subsequent five-year changes in the index has been a humungous 0.8. With the yield now over 5%, history suggests we have the best buying opportunity since 2009.

But, but, but. If investors had believed that the dividend predicts returns, we would never have seen any significant sell-offs of the sort we've seen lately. Any incipient rise in the yield would have triggered buying with the result that the yield and prices would have been stable. The pattern in my chart exists only because people didn't believe it would exist.

Why, then, has the yield been so volatile?

One answer is that it only predicts medium-term returns. Most stock market participants, however, don't care about these. For many, "the long-term is this afternoon." Over shorter periods, returns are unpredictable: the correlation between the yield and subsequent monthly returns has been only 0.1. A cheap market can get cheaper - hence the saying, "never try to catch a falling knife."

In the short-term, participants focus on two things other than the dividend yield.

One is trying to anticipate others' beliefs. As Keynes put it:

Professional investment may be likened to those newspaper competitions in which the competitors have to pick out the six prettiest faces from a hundred photographs, the prize being awarded to the competitor whose choice most nearly corresponds to the average preferences of the competitors as a whole; so that each competitor has to pick, not those faces which he himself finds prettiest, but those which he thinks likeliest to catch the fancy of the other competitors, all of whom are looking at the problem from the same point of view. It is not a case of choosing those which, to the best of one's judgment, are really the prettiest, nor even those which average opinion genuinely thinks the prettiest. We have reached the third degree where we devote our intelligences to anticipating what average opinion expects the average opinion to be. And there are some, I believe, who practise the fourth, fifth and higher degrees.

This means we face what Mordecai Kurz has called endogenous uncertainty - uncertainty not just about the so-called fundamentals, but about other investors' beliefs (or their beliefs about others' beliefs and so on).

A second concern in the danger of a positive feedback loop, whereby falls lead to further falls. These can arise simply because leveraged traders, facing big margin calls, are forced to sell. Or they are the result of "risk parity" trades, whereby traders trying to maintain stable volatility in their portfolios sell stocks when volatility rises. (This isn't as daft as it seems).

Of course, not all uncertainty is generated only within financial markets. Some exists because the world really is unknowable. What investors must worry about is not merely the central case scenario but the distribution of risks. As David Meenagh and colleagues have shown, small but reasonable changes in the probability we attach to a catastrophe can justify big swings in prices.

In this context, my chart overstates the predictability of share prices. For decades, the UK stock market has recovered from most setbacks*: we've enjoyed what Dimson and Marsh called the triumph of the optimists. Disasters have not materialized and it has therefore paid off nicely to buy when investors have worried about them.

But we have no assurance whatsoever that history will repeat itself. Instead, we face what Richard Bookstaber called radical uncertainty - the unknowable chance that the past won't be a guide to the future. Returns might be non-ergodic.

It's reasonable to worry about this now. As Paul Krugman says, "markets are implicitly predicting not just a recession, but multiple years of economic weakness." With conventional monetary policy almost maxxed out, central banks can do less than before to support equities. Covid-19 might accelerate deglobalization and the productivity gains it has brought. The low aggregate rate of US profit might ensure continued secular stagnation. "Sticky" economies (to use Banerjee and Duflo's useful term) cannot adapt well to shifts in the pattern of demand wrought by Covid-19. And so on.

Sure, a high yield in the past has been a predictor of high returns. But it might today instead be a predictor of low profit growth. And even if it isn't, it would be imprudent to bet the house this way.

We have known since Shiller's work (pdf) in 1981 that share prices are more volatile than underlying dividends. What's not so certain is whether this is because investors are irrational or not.

Now, I know a lot of you aren't interested in stock markets. Insofar as they are full of pompous posh white men, I share your lack of interest. But there's more to them than this. Markets raise questions of general importance: is there predictability in human affairs? Is history a guide or not? How do beliefs not only reflect the world but shape it too? Does the pursuit of individual incentives lead to efficient aggregate outcomes or not? What are the roles of rationality and irrationality? Markets matter not because of a few points here or there on the Footsie, but because they are a window onto the social sciences.

* Not all, though. The market is lower than it was during the tech bubble of 2000.