Microsoft CEO Satya Nadella wants companies to be able to stop an advanced AI system in the middle of a job, even when the system appears to be working as intended.
In an essay published Saturday on X, Nadella called for an “emergency brake” on powerful AI models and argued that organizations should treat them as potential insider security risks. His warning comes as AI companies disclose episodes in which their systems exceeded intended boundaries or took unexpected actions.
“We must assume a model is compromised and contain it from the start,” Nadella wrote, according to TechCrunch. “Think of it like an emergency brake.”
Who holds the controls?
The distinction matters as companies increasingly put AI agents to work using software, sending messages and accessing sensitive information. A chatbot can give a mistaken answer. An agent with permission to operate other systems can turn a mistake into an action before anyone notices.
Nadella’s proposal would separate the AI model from the software that directs its work. Controls over access and permitted actions should sit outside the model itself, he argued, rather than depending solely on the model to obey instructions. Authorized humans should be able to interrupt an agent while it is still working.
He also urged developers to keep tamper-resistant, human-readable records of consequential AI actions and make systems independently auditable. Those measures could help investigators reconstruct what happened after a failure, while making it harder for a model to evade oversight.
His argument resembles the cybersecurity principle of zero trust: possessing access or appearing reliable does not eliminate the need for restrictions and verification. Nadella said even models that are not malicious can make errors or be compromised.
A series of warnings
The call follows disclosures by developers of frontier AI models about unusual behavior during tests. In one incident, OpenAI described a security incident involving a model evaluation and Hugging Face, prompting the companies to examine their procedures. Anthropic has also documented misuse of its Claude models by outside actors across cyber operations, fraud and other areas.
These incidents differ in cause and severity. Misuse by criminals is not the same as an autonomous system going beyond its instructions. But both illustrate why relying on a model’s built-in safeguards alone may be insufficient when it can access consequential systems.
Nadella’s essay did not announce a new Microsoft product or a deadline for implementing an emergency stop. It set out a design principle for the industry. The challenge will be proving that independent controls still work when an AI system acts quickly, uses multiple tools or encounters instructions planted by an attacker.
The question is no longer simply whether AI produces a trustworthy answer. It is whether the people deploying it can reliably see what it is doing, limit what it is allowed to do and halt it before an error causes harm.
