A group of leading AI researchers and executives is warning governments to prepare for the possibility that artificial intelligence could begin accelerating its own development faster than people can comfortably track or control.
In a paper published Monday, more than 20 researchers argued that increasingly capable AI systems could automate enough of the work involved in building better AI to set off what they call an “intelligence explosion.” The scenario remains uncertain, they stressed, but the authors said the consequences could arrive too quickly for governments to improvise a response after the fact.
The authors include Geoffrey Hinton and Yoshua Bengio, two of the most influential researchers in modern machine learning, along with OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft chief scientific officer Eric Horvitz and Meta vice president of AI research Dawn Song. Their warning puts an unusually broad group of people from academia and frontier AI companies behind a concern that, until recently, often sounded more like a thought experiment than a near-term policy problem.
The mechanism is straightforward. AI systems are already being used to write code, run experiments and help researchers evaluate new models. If those systems become capable enough to handle a large share of AI research and development, the resulting models could in turn help build still more capable successors.
That feedback loop is the heart of recursive self-improvement, a long-discussed idea in AI safety. The new paper asks what happens if it becomes an engineering process rather than a theoretical possibility.
“Once an intelligence explosion begins, the window for action may close,” the authors wrote.
The paper does not claim that such an explosion is inevitable. Compute limits, long training runs, diminishing returns, bottlenecks in physical infrastructure and the difficulty of automating scientific judgment could all slow the process. The researchers nevertheless argue that evidence from current AI labs makes the possibility serious enough to plan for now.
AI is already doing more of the work used to create AI. Anthropic has said its systems now generate a large majority of the code used internally, while OpenAI is deploying autonomous agents in parts of model development. The researchers argue that the critical threshold would come when AI can perform expert-level research across enough of the development pipeline to create very large gains in effective research labor.
At that point, the paper says, a developer could in principle operate an AI research workforce equivalent to enormous numbers of highly skilled human researchers. Advances that once took years might then be compressed into months or even weeks.
The danger is not simply that models would get better quickly. The authors describe three broader risks: capabilities could advance faster than safety measures and institutions can adapt; humans could lose meaningful oversight as AI takes over more of the research process; and a government or company with a temporary lead could convert it into a much larger strategic advantage before rivals could respond.
Those concerns arrive as frontier labs are also confronting signs that increasingly autonomous systems can behave in unexpected ways. Recent evaluations have documented agents disobeying instructions, exploiting loopholes and taking actions their developers did not intend. None of that proves that an intelligence explosion is close, but it makes the question of how much control humans retain over more capable agents less abstract.
The researchers want governments to start by measuring the problem. They propose standardized reporting on how much AI research is being automated, independent evaluation of advanced systems and disclosure of serious incidents. They also raise the possibility of technical and infrastructure measures that could slow or pause particularly rapid AI development if warning signs appear.
Among the more concrete ideas are isolating highly capable automated research systems, working with data centers so certain projects can be paused and building emergency plans before a sudden jump in capabilities. The authors also call for international coordination, reflecting the concern that unilateral restraint may be difficult if governments believe a rival is close to a major breakthrough.
The recommendation is notable because several authors work at companies competing to build the most capable AI systems. Their paper does not resolve the tension between that race and calls for stronger safeguards, but it makes the dispute harder to dismiss as a fight between industry and outside critics.
For policymakers, the problem is timing. Regulation usually follows visible harm or a technology that has already stabilized enough to understand. An intelligence explosion, if it happened, would invert that sequence by making the most important period for intervention the one before the evidence becomes conclusive.
The authors acknowledge that the scenario may never materialize. Their case is that waiting for certainty could itself be the mistake.
“Preliminary evidence suggests that a software-driven intelligence explosion is possible,” they wrote.
