Arvind Narayanan

As AI labs race toward Recursive Self Improvement, there’s a possibility that what they are actually racing toward is Recursive Self Extinction — of their business model. Here’s my reasoning. 1. Progress on verifiable tasks continues. In particular, improving the cost efficiency of existing capabilities is a highly verifiable task. For the past few years, cost per equivalent capability has dropped by at least 10x per year. I think this progress will be sustained for a few more years, and even accelerate as labs use AI more and more heavily for AI research in the pursuit of RSI. There is of course a ceiling to efficiency improvements, but I don’t think we’re close to that ceiling. 2. Progress on most unverifiable tasks remains frustratingly hard. AI inherits the same limitations (ambiguous success criteria, slow and noisy feedback) that limit human performance at tasks ranging from writing to governance. The optimization paradigm generally doesn’t result in superhuman capabilities on such tasks, because whatever proxy you optimize for gets Goodharted. For example, I recently pointed out how over-optimization explains Claude’s overly dense and ornamental writing style that’s since been dialed back. 3. RSI ends up being narrow. A consequence of the above two points — even if some version of RSI is achieved, it results in an explosion in some dimensions (efficiency, verifiable tasks) while falling short of producing something that is superintelligent in any real sense or capable of automating a broad swath of jobs in the economy. 4. More than one lab stays at the frontier. This means the market will remain competitive, and developers can’t charge high margins to compensate for rapidly falling inference prices. Any compute scarcity that allows higher margins is temporary, in my view. I also think it’s likely that most of the innovations continue to make their way into open-weight models with some lag, so open-weight models are at cost parity for inference or only slightly more expensive. Note that efficiency tricks and verifiable capabilities tend to leak much faster than unverifiable ones. Distillation could be one reason, but simply knowing that something can be sped up 10x tends to have a focusing effect on research and innovation. 5. Jevons’ paradox is not a magic bullet. If the above points hold, there are very few applications that will consume so many tokens that they could generate the astronomical amounts of revenue implied by the labs’ valuations. We’re talking about a 10x growth in tokens year after year, just to keep revenue flat given falling prices. Even today’s behemoth tasks like migration of million-line legacy codebases will likely become cheap within a few years. While new applications will open up, there won’t be orders of magnitude more new ones. As for unverifiable tasks, inference scaling rapidly hits diminishing returns. Ultimately you have to have a human in the loop who’s accountable for the result. Spending more on tokens doesn’t let customers escape that responsibility — and only makes the human review bottleneck worse. 6. What about scientific research? Science is an endless frontier, and more tokens could perpetually mean more discoveries and inventions, in theory. But the unfortunate thing about research is that researchers don’t get to capture the value it creates, which is why it requires government funding in the first place. Even if it’s true that AI could enable great scientific innovations, I don’t think AI companies will make much money off of that. I think they are currently investing in AI-for-science not as a future revenue stream but as a loss leader because they want to show the public that it could be beneficial to humanity. 7. The one escape hatch is moving “up the stack”, by which I mean quickly transitioning from selling tokens to an enterprise software like model that relies on high switching costs, deep integration into workflows, and control over systems of recor…