🔮 Unbounded self-improvement and its limits #599

Good morning!The limits to recursionWhat are the conditions under which AI systems could undergo recursive self-improvement (RSI) — and how long might that last? Cards on the table, I’m not wildly excited by the theory of unending accelerating recursive self-improvement for the simple theoretical…

Good morning!The limits to recursionWhat are the conditions under which AI systems could undergo recursive self-improvement (RSI) — and how long might that last? Cards on the table, I’m not wildly excited by the theory of unending accelerating recursive self-improvement for the simple theoretical issue of control and alignment. But I also think it’s not likely for practical and theoretical reasons.Now philosopher, Toby Ord, has done the heavy lifting for me. He argues that a key gating factor to RSI is the generation time: how long it takes of an AI system to go through a single loop of improvement (where it helps design and train its successor). Ord concludes that while it is mathematically possible for extreme RSI, intelligence rising without bound, the conditions are unbearably difficult to achieve. The key question is whether the entire research to training to development cycle can shrink to zero or not. Ord reckons unlikely, I too don’t believe it is possible. Generation time can’t get to zero because real-life intrudes: experiments take time; training runs take time; making new chips take time… lots of things take time. It might still feel fast, but it wouldn’t race to infinity. Eventually, physics intrudes too: the speed of light limits communication speed; the Bekenstein bound limits the information contained within a finite bit of space; and Landauer imposes an energy tax on irreversible computation.1The Universe, it seems, agrees with me. Unbounded RSI has its limits.You can read Ord’s paper here. Premium members can explore a plain English interactive version too. Subscribe nowThe market is the instrumentOpen-weight models are growing in popularity in the business world: their token share at Vercel hit a single-day record of 62%, up from 28% two months earlier. Some Western firms are even moving workloads to Chinese open weights — Thomson Reuters has developed its first in-house model based on Qwen to cut costs. You can tune cost-effective open weights to match, or sometimes beat, frontier performance on the tasks that matter to you. Take Bridgewater: working with Thinking Machines, it fine-tuned an open Qwen model on expert-labeled data and beat every frontier model it tested on its internal information-filtering tasks: roughly 30% fewer errors than the best closed model, at one-fourteenth of the inference cost. Trainloop, which I am an investor in, does something similar, using the tiny Qwen 3.7-27b model, and can outperform GPT 5.6 Sol on specific fine-tuned tasks at a fraction of the cost.The results that Trainloop is getting are pretty impressive, seeing as they are based on a pocket model—a 27b model will even fit on a desktop Mac.2 Openweight models are, of course, getting better and better. Z.ai GLM 5.3, released this week, completely reshapes the cost-performance Pareto frontier. Of course, it isn’t a small model, but smaller distillations will emerge from it. All of this speaks to a welcome competition in AI provision. Clearly, firms could move focused workloads onto the most-performant, fine-tuned small models they can. Where possible, they might choose large, generally capable open models. But the appeal of being at the frontier, which is more than just model performance—it is service guarantees, harness quality, reliability, and a host of other requirements—still drives significant business for Anthropic and OpenAI. We don’t think it has much impact on the question of whether revenues flowing into the industry will materially change. For one thing, we don’t have a counterfactual to test against. But more importantly, every open model still involves paying inference providers. We’ll be looking at this question in more detail in the comings weeks. A spicy entry to the compute ecosystemRecursive self-improvement may be entering the compute realm. OpenAI’s new chip, ‘Jalapeño’, was designed with a heavy helping hand from the company’s own models, which helped write kernels and cut roughly 10% from one of the chip’s main compute blocks. In around 16 months from first hire to tape-out, OpenAI has built a chip that beats comparable Nvidia silicon by 1.5–1.9x on tokens per megawatt at peak throughput. This suggests frontier models can compress the design cycle for competitive silicon. AI will result in far greater heterogeneity in chip architectures than we saw in prior computing markets. Personal computers battled between the x86 standard and the Motorola 68x, before today’s duopoly of Intel and Apple silicon. Different uses for AI will need compute optimised for intelligence, latency, power consumption, training, and inference. This creates lots of room for specialist firms. One example is that ChatGPT’s fast response mode is powered by Cerebras’ low-latency silicon. Another is Fractile, where I am an investor, which has a deal with Anthropic for its low-latency inferencing chips. Compute is becoming a highly segmented market, where chips aren’t a standardized commodity. This differentiated hardware demand will expand the market even if it potentially reduces Nvidia’s relative dominance.MorselsGood post from Chad Syverson on AI productivity. Some micro-evidence for improvements, not much at the aggregate level.AgglomerationsUnderstanding AI and ProductivityAs part of EIG’s American Worker Project, we are delighted to present this guest post from Chad Syverson, who is the George C. Tiao Distinguished Service Professor of Economics at the University of Chicago Booth School of Business. Click here for a PDF version of this post…Read more2 days ago · 42 likes · 2 comments · Economic Innovation GroupThe least bad place to hide from global catastrophe? Australia.The harness matters as much as the model: SwarmOS pushed GPT-5.6 Sol from 13.3% to 100% on ARC-AGI-3 Public.The University of Chicago’s Social Sciences Core is going back to paper, banning most classroom technology to deal with the AI-learning crisis.Meta considered shrinking some teams by up to 60% to become “AI native.”Would you take this bet? It pays out if, in any quarter up to and including Q4 2033, US real GDP per capita is at least 15% higher than its previous peak.3With one video, you can reconstruct a moving 4D avatar of a person and render it from novel viewpoints, like a video-game character.You can now teach an adorable “Pixar-ish” robot new tricks for only $399.You can now generate videos in less time than it takes to watch them.A walk down memery lane:1Reversible processors, like those from Vaire, will not pay the Landauer tax. 2I run a version of Qwen 3.7-27b on one of our local machines for various tasks.3Recorded at least four quarters earlier.

Source: Exponential View — Published August 30, 2026 at 02:51 UTC — Category: Business