Author Talks: What the history of progress and innovation tells us about disruption

In this edition of Author Talks, McKinsey Global Publishing’s Rick Tetzeli chats with Carl Frey, the Dieter Schwarz Associate Professor of AI & Work at the Oxford Internet Institute and director of the Future of Work at the Oxford Martin School, about How Progress Ends: Technology, Innovation, and the Fate of Nations (Princeton University Press, September 2025). Frey explores how innovation and technological progress alternate with periods of decentralization and fragmentation, shifts that have occurred repeatedly over time. His exploration extends from the Roman Empire to Silicon Valley, and from China to the United States. An edited version of the conversation follows. You can also watch the full video at the end of this page.

Why did you write this book?

The key motivation for writing this book was to communicate that we shouldn’t take technological progress for granted. If progress were a given, the first industrial revolution would have happened a lot earlier in human history.

If progress were a given, most places around the world would be rich and prosperous. If progress were a given, we wouldn’t see booms and busts, and we would see places steadily improving their material standards. But if we look historically, no country or society has been the technology leader for very long.

What is so important about saloons and coffee houses over the centuries?

If you know something but you never tell anybody about it, that piece of knowledge will at some point be lost forever. Innovation depends on people communicating, knowledge being passed on to the next generation, ideas being shared, and giving other people an opportunity to build on them or execute on them.

Saloons and coffee houses were very important for innovators, creative minds, and intellectuals to meet and share discoveries and ideas. Saloons were for many the place where they met after work, and they discussed and explored. That’s how progress happens.

But during the Prohibition era in the United States those networks essentially collapsed.

Say more about why technology slowed during Prohibition.

Working people are the key drivers of innovations in this period. It’s not the upper tail of human capital that drives technology in the late 19th century, early 20th century.

Very much innovation is achieved by tinkerers and independent inventors experimenting. When saloons closed, those hubs for meetings, discussion, conversation, and exploration essentially shut down. People had to rebuild their social networks, and that process takes many years.

Why did the fall of the Roman Empire lead to Europe becoming a hotbed of innovation for centuries?

The reason that the decline of the Roman Empire led to an upsurge in innovation is that fragmentation and decentralized competition are essential for innovation. If you compare this to China, for example, the Song dynasty was very technologically progressive for its time.

But most innovations centered on technologies that aided the state, such as soil mapping, which was really important for the bureaucracy to know what farmers were producing, to be able to set taxes accordingly. Another example is the introduction of a unified writing system, which allowed them to develop population records, track people, and build bureaucratic governance structures.

China built a strong state on the back of those technological discoveries.

The Roman Empire was not as centralized. The Roman Empire essentially expanded by integrating the armies of the communities that it conquered into its war machine.

But that gave them a fairly significant degree of autonomy. As an empire expands, however, either you have to centralize to control the territory that you acquire, or something has to give. Unfortunately, in this case, the Roman Empire collapsed.

What that meant was that intellectuals and innovators could move between territories. For example, when Louis XIV revoked the Edict of Nantes and made Protestantism illegal in France, the Huguenots could migrate to Germany, Switzerland, the Netherlands, and England, where they then imported new technologies and skills as they settled. Similarly, people like John Locke or Voltaire continuously moved around in response to the political environment in which they interacted.

In China, such mobility wasn’t possible. If Galileo had been born in China rather than in Europe, he would probably have become a bureaucrat, as that was the most lucrative path of social mobility in China.

The fragmentation Europe experienced was very important for people to be able to move around and escape the tyranny of the state. That competition eventually led to the gradual decline of craft skills in Europe, which controlled production for a very long time.

That first happened in Britain, in large part because wages in Britain were very high, and the ruling classes in Britain were concerned that they wouldn’t be able to compete in international trade for that reason. The political elites come to favor mechanization and then decide to clamp down on the craft skills through acts of parliament.

Second, the construction of roads and turnpikes in Britain undermined the political clout of the guild that didn’t extend beyond a given city. That introduced competition to the system, and that is what unleashed the forces of creative destruction in Britain earlier than anywhere else.

In China, it took another 200 years for the guilds to be abolished, and it’s only really with the opening of treaty ports and competition from abroad that that system eventually disintegrated. In Europe, it took a longer time as well.

But as places like Germany followed, they clamped down on the guilds as well. So did Napoleon. That enabled industrialization to spread across Europe.

The central thesis of the book is that there is a need for both decentralization and centralization. Can you explain why you think both are necessary for progress?

I’m an academic, and if I want to write a book or a paper, first I have to decide what I want to write about. I have to decide whether that idea makes sense. I have to read. I have to talk to people. I have to explore.

In the early days of the process, I discard most ideas. But you need to go through that process for exploration. You have to do the same if you’re an entrepreneur or an inventor. Since you don’t know how successful your idea will be or how good your idea is until someone has invested in it and it is released to the marketplace, you must have a system where people can diversify.

Consider the U.S.S.R., the most centralized economy there ever was. Say you were an aircraft engineer, and you wanted funding for your project. You could go to the army, and if the army declined funding, you may have a few more options to try. Yet if those were declined, your project was dead. That’s quite different from Google, for example.

Bessemer Ventures famously declined to invest in Google back in 1999, but others invested in it. The project did not die with that failure to invest. Katalin Karikó, who’s behind the mRNA vaccines [in response to the COVID-19 pandemic], spent decades in the United States; mRNA technology was considered futile at the time.

We are dependent on people taking those risks and making those bets. The more people who take different bets, the more exploration we have and the likelier we are to end up on a productive path. That’s why decentralization is really critical to innovation.

And how does centralization assist in some cases?

At some point, you want to make your product available to the market, and you want to scale. That means you need to build managerial hierarchies to support distribution networks, to be able to take advantage of new technologies such as factories, and to mass-produce products.

Scaling is a very different activity from innovation and exploration. If you’re producing something, you don’t want everybody on the factory floor to question every single procedure. You will not get much productivity out of such a system.

But if nobody questions anything, you won’t get innovation, either. These are two very different activities. An argument that I try to make in the book is that you need both to be able to grow in the long run, unless you can import technologies.

If you want to understand both the growth and the collapse, then you have to understand that you can grow very rapidly just by scaling and absorbing technology from other places. But at some point, when you encounter something new—that new encounter was the computer in the 1990s—then you need to adjust to those changing circumstances.

You write, ‘At the technological frontier, continued progress depends on the willingness of governments to override vested interests to restore competition.’ Why are you in favor of such pronounced government intervention?

I’m not sure if I would call it pronounced government intervention. You need people to take risks, to invest, and to have the desire to grow their enterprises and scale. At the same time, you need to realize that, eventually, that runs into diminishing returns. In some instances, incumbents have an interest in safeguarding themselves from competition, blocking entry, and making it harder for new enterprises to emerge to challenge the monopoly they’ve been building.

I don’t think that there’s a case for governments to go in and break up companies just because they’ve grown big. I don’t favor that. What I do favor is governments having the ability to enforce the rules of the game, to enforce competition, and clamp down on anticompetitive behavior.

If you look back historically, and you compare the US and Japan, back in the 1980s, everybody worried that Japan was going to overtake the United States. It was growing much more rapidly.

It had already overtaken the US in automobiles and was much more productive in that industry. It was gradually taking over the semiconductor industry as well. But Japan completely missed the boat in the switch to software, and missed the boat in the switch to e-commerce.

The conglomerates that were essentially controlling Japanese industry were erecting quite formidable barriers to entry for outsiders. A lot of trade was just happening within those groups, and they were not very interesting competition.

Conversely, in the US, a case was launched against IBM. That eventually led to the unbundling of hardware from software and created space for firms such as Microsoft to enter the market. The breakup of AT&T meant that when the National Science Foundation essentially released ARPANET, the predecessor to the modern internet, to the world, it was not handed over to a telecommunications monopoly.

Those examples do illustrate that competition policy can be productive, but I’m the first to acknowledge that it can also become highly politicized.

You say that it’s extremely hard to predict the future of a disruptive technology. Why is that true for AI?

Much depends on what we use technology for. If all we had done since 1800 was automation or introducing new productivity tools, we would have productive agriculture, but we wouldn’t have much else.

We wouldn’t have vaccines, antibiotics, rockets, airplanes, computers. Most growth and most increase in material standards come from doing new and previously inconceivable things and from the introduction of new industries.

Anything that involves innovation is hard to predict, because we don’t know when the next breakthrough will emerge. For example, in the early days, steam engines were mainly used to drain coal mines, but they didn't do that particularly well. They were tremendously energy inefficient. It took James Watt’s separate condenser and a number of other innovations for steam to be applied to railroads and steamships. I think AI needs its own separate-condenser moment. Algorithms are tremendously data hungry. If the world were just a static distribution of events, then we could probably force the widespread adoption of AI. More data, more compute, you’re done.

But the world is not a static distribution of events. It changes all the time. And if the algorithms did not adjust to new realities, they would not perform particularly well. Most people know of AlphaGo, which beat the Go game world champion, Lee Sedol, back in 2016.

Fewer people would know that in 2023, human amateurs using standard laptops beat the best Go programs quite easily by employing new tactics that they would not have encountered in training. That raises important questions.

Even when AI has achieved superhuman performance, humans can make a comeback by exposing it to new situations and tactics. As the world then changes, how will these algorithms perform? Humans are very adaptable. We can learn from just a few examples and adapt to new circumstances. We are more resilient in that sense than AI is. Going forward, we will need AI that is more resilient as well.

I’m an economist. I don’t know the exact discoveries, tweaks, and innovations that will be needed to get there. The more I speak to computer scientists, the more I realize that they don’t know exactly, either. This is going to be a process that will require exploration. Exploration thrives in competition and decentralization.

Somehow you managed to connect Galileo to large language models. Please explain.

An algorithm that had been trained before Galileo would not place the sun in the center of the universe. In a similar fashion, had you asked an LLM in 1900 to predict whether human flight would be possible, consider the data set it would have had access to.

It would have been a long list of failed attempts by humans flying and trying to fly—numerous deaths. The best evidence that flight is possible would be evidence from birds, but even then, larger birds, such as ostriches, cannot fly.

No bird that weighs more than 30 pounds can get off the ground. So, an LLM would have said, “No, humans are not going to be able to fly.” Similarly, if you give an algorithm a database of Impressionist paintings, it will not come up with conceptual art.

These leaps still belong to humans. When it comes to frontier discovery, humans still have a comparative advantage. LLMs are very impressive. But at the end of the day, to write prose in the style of William Shakespeare is only semicreative. The reason that AI can write prose in the style of William Shakespeare is that Shakespeare existed.

Watch the full interview

Visit Author Talks to see the full series.

Learn more