When does innovation work and when not? I'm prompted to ask by Ed Smith's wonderful new book, Making Decisions, which argues for the need for"anti-processes", original thinking and spontaneity to resist "the slide into bureaucratic inertia":

If your decisions do not diverge from conventional wisdom in any important respect, you are failing to do your job. Indeed, you no longer have a job. You have a sinecure.

All this is good and true. Following process and protocols can degenerate into groupthink, cargo-cult decision-making, and an irresponsible deference to algorithms.

But, but, but. I read Ed's book whilst cryptocurrencies were collapsing: whilst Elon Musk was ruining Twitter; whilst we are still awaiting the "sunny uplands" of Brexit; and a few weeks after Liz Truss's promise of "doing things differently" ended in abject failure. All of which show that there's something to be said for cleaving to the conventional wisdom. As Edmund Burke said:

We are afraid to put men to live and trade each on his own private stock of reason; because we suspect that this stock in each man is small, and that the individuals would do better to avail themselves of the general bank and capital of nations and of ages.

Reinforcing this point, whilst reading Ed I was listening to a lot of J.S. Bach, whose life shows that great genius sometimes consists not in originality but in sticking to rules.

Economics, perhaps, offers more examples of what I mean. It's not clear that recent new economic thinking - be it DSGE models or MMT - is much of an improvement on the functional finance and structural models we had before the 1980s. A while back, the Guardian used to run tiresome articles calling for new thinking about economics, when what I suspect they meant was a return to old thinking.

Nor is economics alone here. In Science Fictions, Stuart Ritchie describes "a deep corruption within science itself", which has a "dizzying array of incompetence, delusion, lies and self-deception." Hence the replication crisis which seems to have afflicted most academic disciplines. "New thinking" is often bad thinking with dubious results.

All of which raises the question: under what circumstances should we stick to conventional thinking, and which not?

Of course, the best time to innovate is when there is cheap money and lots of mug punters. The collapse of cryptocurrencies is yet another reminder that actually-existing markets incentivize bad innovation rather than good.

That aside, there are, I suspect, four conditions when we should reject Burke and Bach in favour of Smith.

One is that the conventional wisdom is sometimes merely the groupthink of the privileged, who use it to entrench inequality. Talk of a "fiscal black hole" for example is used to justify otherwise questionable austerity, just as talk of incentives or deregulation (pdf) can be a ideological cloak for inequality.

A second is that the cliche is true: necessity is the mother of invention. Bach had such abundant resources of genius that he could create brilliantly within existing conventions; he didn't need to innovate. The rest of us, though, often aren't so lucky. Francis Spufford's brilliant Backroom Boys describes how British boffins in the mid-20th century developed space rockets and computer games with little budget and puny computing power.

Ed had to do something similar. In 2019 England lacked top-quality opening batters and so experimented with a big hitter, Jason Roy, at the top of the order. That didn't work, but the subsequent performance of England's openers suggests it was worth a try.

A more successful example of innovation in response to lack of resources was "moneyball". The Oakland Athletics' small budget for players led their manager Billy Beane to evaluate players on the basis of statistics rather than conventional coaching judgements which led (briefly) to the recruitment of undervalued players and an improvement in the team.

Economic policy also provides an example. The UK adopted inflation targeting in 1992 in a panic and pretty much as a last resort after previous "monetary anchors" such as M3 targets, shadowing the Deutschemark or ERM membership had failed. Until recently, it worked much better than its predecessors.

A third circumstance favouring innovation and thinking outside the box is when the environment changes. Cricket provides another example. Jonathan Liew describes T20 as "an entirely different sport now" requiring the "shedding the decades of ingrained cultural baggage." In T20, a team's most aggressive batter should be not in the middle order but an opener, simply because he can then potentially face more balls.

The environment, of course, also includes technological possibilities. These don't just facilitate new companies, but also new strategies. As Stian Westlake and Jonathan Haskel have pointed out, modern intangible assets make production much more scalable which enables a few successful companies to get very big - hence the emergence of superstar (pdf) firms.

We see - or should see - this in politics too. The genius of Blair and Brown was to see that changing socio-economic conditions after the 1970s required a new form of social democracy. Equally, though, economic changes after the 90s requires yet another rethinking of social democratic politics. Most Blairites and Corbynites are too tribal to see it, but in important respects Corbyn was the heir to Blair.

But there is a limit here. It's easy to see a changed environment when in fact there is only fashion. One of the surest ways for investors to lose money is to believe "this time is different". Which it rarely is. Conventional thinking, such as intelligent momentum investing, beats believing hype about new technologies.

We could add a fourth circumstance when thinking outside the box works. In strategic interactions, be it sport or business, it can give one an edge over one's opponents. Arsene Wenger's use of healthy diets and more overseas players gave Arsenal big advantages in the late 90s.

The problem with these, however, is that one's opponents quickly learn to emulate such innovations, thereby neutralizing them. Wenger's use of sports science became less help as other coaches copied him, and Billy Beane's moneyball became less effective as rival teams adopted his methods.

The point generalizes. William Nordhaus has shown that the profits to companies' innovative activities are usually tiny simply because new products and processes get copied, thus bidding away initially high profits. Sustainable high profits come from conventional thinking about how to hold down costs, preserve brand power and fend off competitors thus maintaining what Warren Buffett calls economic moats.

All of which is to merely regurgitate an old saying of Niels Bohr: the opposite of a great truth is another great truth. Whilst there is sometimes (occasionally) a case for original thinking it is also true that conventional ideas are sometimes conventional for a very good reason. Which vindicates Ed's point: we will always need a "human dimension" if we are to have a hope of distinguishing the two.