If Democrats take control of at least one chamber of Congress and pick up a few governorships in the midterms, proposals to regulate frontier AI labs could get a more serious hearing. So I wanted to look at some of the ideas being discussed, particularly those that would change how labs develop and…
Annons
Annons
If Democrats take control of at least one chamber of Congress and pick up a few governorships in the midterms, proposals to regulate frontier AI labs could get a more serious hearing. So I wanted to look at some of the ideas being discussed, particularly those that would change how labs develop and release models. A quick disclosure: my P(doom), the odds I’d put on AI causing a catastrophe, is on the low side. I’m also cautious about new regulation. My default is to use existing laws to govern AI applications within specific sectors and industries. I’ve tried to approach these proposals with an open mind, but you may see some skepticism leak into my comments. Subscribe to our weekly newsletter Pausing RSI Requires Defining It First Some of the more ambitious proposals would pause recursive self-improvement (RSI) or even prohibit the development of “superintelligence.” I’ve written before about the distinction between an AI system making a bounded improvement and one that improves the process used to generate further improvements. That distinction matters if we’re going to regulate RSI. Does using a coding agent to improve a training pipeline count? What if the agent runs experiments and selects the best results? Without a clear, enforceable definition, I’m not sure what a ban would actually prohibit. Some proposals seem more practical, at least in principle. Requiring labs to evaluate internal models before using them to develop their successors is one example. But who sets the criteria, and who decides whether a model is safe enough for that work? I’m not sure a lab’s own assessment would be sufficient. I’m less convinced by proposals requiring three to nine months of public deployment before a model can be used for internal research. Public use may not reveal the risks of giving a model access to a lab’s research tools and training systems. Coordinated slowdowns also raise questions about competition, while short delays to complete safety testing seem like a reasonable compromise. Locking Down the Training Environment Of all the proposals, those involving training security strike me as the least controversial. Much of this is basic hygiene. Recent incidents involving frontier AI agents have shown why labs need to do a better job containing their own systems. OpenAI recently described its approach to securing frontier model training, including cutting off live internet access, adding multiple layers of isolation, and using its strongest models to test those defenses. In September, an internal model exploited a network misconfiguration to gain limited access to the internet. OpenAI paused some training while it strengthened its safeguards. This seems like an area where concrete security requirements make sense. I’d also support requiring labs to monitor for unauthorized agent activity, report serious incidents involving internal models, and regularly reassess their security controls. Researchers need realistic environments to test dangerous capabilities, particularly in cybersecurity, but that doesn’t justify unrestricted access to external systems. There are legitimate questions about how much incident information should be made public, especially when disclosure could help attackers. Still, these measures seem more practical and enforceable than some of the broader restrictions on model development. Who Tests the Frontier Labs? Security measures are only as good as the tests used to validate them. I like the idea of giving independent evaluators meaningful access to frontier labs, including internal models and safety procedures, not just models released to the public. Anthropic has already started giving researchers from METR, a nonprofit evaluator, employee-like access. Routine stress testing, safety plans that meet meaningful standards, and cooperation among international safety institutes also seem reasonable to me. Government agencies will struggle to hire enough people who understand these systems, and relying on labs to grade their own work isn’t sufficient. One idea I’ve heard is to encourage private testing firms, similar to credit rating agencies in finance. But that model has a well-known weakness. Before the financial crisis, rating agencies were paid by the banks whose securities they rated, and plenty of shaky mortgage-backed securities ended up with top grades. I’d rather see something closer to Underwriters Laboratories, with independent standards and a long tradition of testing how products fail. I recently recommended a book on test engineering, and I think AI teams have a lot to learn from that discipline. Credible independent assessments could also help CTOs compare models and insurers assess risk. Who Pays When Things Go Wrong? Holding specific decision-makers accountable and protecting employees who raise safety concerns seem reasonable to me. Liability is more complicated. When an AI application causes harm, how do we distinguish a problem with the underlying model from mistakes made by the company deploying it? Mandatory insurance is an interesting idea, and independent testing could help insurers assess risk. But someone still needs to figure out how to price risks that are difficult to measure and for which we have little historical data. Without that, I’m not sure how an insurance market would work. Other proposals would make voluntary safety commitments binding or prohibit labs from supplying AI for certain uses, including military applications and workplace surveillance. I’m skeptical that blanket restrictions on which customers private AI companies can serve would gain much traction. Regulating specific applications is one thing, but prohibiting entire categories of customers is another. I also don’t think labs should get to write their own rules, although regulators will need their technical input. The proposals discussed above may struggle to become law, but I wouldn’t be surprised if some became litmus tests in the 2028 Democratic presidential primary. The post Previewing the Debate Over Frontier AI Regulation appeared first on Gradient Flow.