Where are there so few free market pessimists? A post by David Henderson raises this question. Responding to Noah's claim that fossil fuels will run out, he says:

As the price of an energy source rises, entrepreneurs have a strong incentive to invent, develop, and produce alternate sources.

This is a longstanding and common position among supporters of the free market - that the market will help us overcome, or sidestep, energy shortages. And it has been correct for many years. As David says:

Every single time someone has predicted that they we will run out of oil, that person has been wrong. What reason is there to think that prediction today will be right?

However, this logic isn't watertight. The fact that things have happened in the past does not guarantee they'll continue to happen. As Bertrand Russell famously said:

The man who has fed the chicken every day throughout its life at last wrings its neck instead, showing that more refined views as to the uniformity of nature would have been useful to the chicken.

Which brings me to my question. Although David's optimism is about humankind's ability to overcome resource constraints is often associated with free market ideas, there is no necessary reason why this should be. One can, coherently, argue that whilst free markets are necessary to bring forth innovation, they are not sufficient. In fact, it's quite possible that it is now harder to make innovations than it was before, simply because we've already discovered the easy ideas.

Such pessimism is quite compatible with a belief in free markets. One could argue that, if economic growth is harder to achieve, then allocational efficiency becomes more important; if we can't produce more from existing resources, we should at least ensure those resources are put to best possible use. And a lack of growth makes it more important to incentivize entrepreneurs to exploit the few ideas there are.

It's entirely logical to believe both in free markets and in the view that diminishing returns will eventually overcome technical progress. This is especially true given that the pace and direction of technical change is largely unpredictable. In fact, it was pretty much what David Ricardo believed. And if it was good enough for him, it'd good enough for us.

Hence my question. Given that free market pessimism seems such a tenable view, why, AFAIK, is it so rare? Why does optimism about human potential seem to be more correlated with a belief in free markets than I'd expect?

I was going to suggest that free marketeers tend to have a more optimistic disposition than their critics; a lot of egalitarianism is founded upon Rawls's maximin principle, which is an ultra-pessimistic decision rule. But this seems inconsistent with research which shows that political conservatives (pdf) tend to be more risk-averse than lefties.

You could retrieve my hypothesis by arguing that a belief in free markets isn't politically conservative - but I fear you might struggle; the two are closely associated in the US at least.

Which leaves me with a genuine question: why has free market pessimism almost disappeared, despite being so reasonable? My more partisan readers might say it's because the free market right has become anti-scientific. Even if there's some truth in this, I'm not sure it's a complete explanation. For me, then, there's a puzzle here - and, given the virtue of cognitive diversity, something to be regretted.