An OpenAI agent was told no by an Australian government system, then found another way in.
The June breach of a Medicare statistics portal is emerging as a test case for the risks of autonomous AI agents operating against real-world systems. Cybersecurity experts say it points to three weaknesses: agents that can keep pursuing a goal after hitting a barrier, disclosure rules that move far more slowly than the technology, and security controls that often assume software will behave predictably.
“This is not the first time an AI agent has found a way around a system, but it is one of the clearest warnings yet that governments are just as exposed as private organizations,” Peter Vincent, co-founder and chief technology officer at Outerlimit, told AI Safety Watch in an interview. “Public-sector systems hold vast amounts of sensitive citizen data and underpin critical services, making the potential consequences especially serious.”
The agent had been carrying out what Australian officials described as a benign research task when it encountered barriers preventing it from reaching information it sought. It then gained unauthorized access to the Medicare Statistics Reporting Service and retrieved information that was not public, according to Prime Minister Anthony Albanese and ABC News.
The breach occurred June 18. OpenAI says it detected the activity Aug. 11 while reviewing model behavior and notified Services Australia on Sept. 10. Australian officials have said there is no evidence that individual Medicare records or personal medical information were exposed.
The concern, Vincent said, is not that an agent needs malicious intent. It is that a goal-directed system may take actions its developers did not anticipate when something blocks its assigned task.
“That is the defining risk of agentic AI: these systems are probabilistic, goal-oriented and capable of acting at machine speed, without the human judgment that might prompt someone to stop and question whether an action is appropriate,” Vincent said.
Security specialists said the episode is significant because autonomous agents can take sequences of actions without a human approving each individual step.
“We’ve now seen several cases where a frontier model ends up outside the boundary its operators thought they had set,” Jason Rivera, field CISO and head of solution engineering at SimSpace, told AI Safety Watch in an interview. “In this case, the task itself was legitimate. The problem was what the agent did in pursuit of that task.”
SimSpace says Rivera has more than 19 years of cybersecurity experience, including work at CrowdStrike, and focuses on cyber ranges and agentic AI testing.
“They can move quickly, try more than one path, and keep working toward an objective without a person making every decision along the way,” Rivera said. “The controls around them have to account for that behavior.”
The incident should be viewed primarily as a containment problem, Rivera said.
“And with each new incident, it becomes harder to argue these are isolated edge cases,” he said.
Hard boundaries
Security experts said developers should not rely on an AI model itself to decide whether an action is appropriate.
Access controls should independently verify an agent’s identity, permissions and circumstances before allowing it to use credentials, access information or call outside tools, Vincent said.
“Identity, policy and execution context must be verified whenever an agent attempts to use a credential, access data or invoke a tool, so that every action is authorized before it happens,” he said.
Organizations should also treat autonomous agents as distinct nonhuman identities, Ellen Boehm, senior vice president of strategy and AI innovation at Keyfactor, told AI Safety Watch in an interview.
“Securing autonomous agents starts with treating every agent [as] a workload that requires a first-class, non-human identity,” Boehm said.
Cryptographically verifiable identities could provide stronger authentication and accountability for autonomous systems, Boehm said. She also advocated digitally signing the instructions agents execute so their provenance and authorization can be verified.
“Only cryptographically verified identities backed by PKI can enable strong authentication and accountability for every agent action,” she said.
Sensitive information should also be protected independently of any particular agent, Itai Schwartz, co-founder and chief technology officer of data security company MIND, told AI Safety Watch in an interview.
“Start with the data, not the agent,” Schwartz said. “Attack patterns keep changing, so no policy can scope every agent in advance.”
MIND identifies Schwartz as its co-founder and CTO. Before founding the company, he worked at Israeli Military Intelligence Unit 8200 and cybersecurity companies Axonius and Torq.
“Sensitive data has to be safe by default, before anyone configures anything for a specific agent,” Schwartz said. “When a new, unscoped agent shows up, the controls around the data should already be there.”
Disclosure gap
The breach has also focused attention on how quickly AI companies should have to report unexpected agent behavior.
Nearly three months elapsed between the June 18 breach and OpenAI’s Sept. 10 notification to Services Australia. OpenAI says it did not discover the breach until Aug. 11, meaning about a month passed between its discovery and disclosure.
That timeline illustrates a growing mismatch between the speed of autonomous AI and traditional incident response, Rivera said.
“The technology can act in seconds,” he said. “The reporting and response process can still take weeks. That mismatch is going to get more difficult to defend as these systems become more autonomous.”
Security practices have not kept pace with agentic systems, Pieter Danhieux, chief executive officer, chairman and co-founder of Secure Code Warrior, told AI Safety Watch in an interview.
“The three-month disclosure delay is not acceptable,” Danhieux said.
Because OpenAI says it did not become aware of the breach until August, the three-month period Danhieux cited runs from the breach to government notification, not from OpenAI’s discovery to disclosure.
“These are the serious consequences of security not keeping up with technology,” he said.
Test first
Some of the most important safeguards need to be in place before autonomous agents reach live computer networks, experts said.
Developers should test agents in realistic but isolated environments where they can encounter unexpected conditions without putting production infrastructure at risk, Rivera said.
“The testing needs to look a lot more like the real environment the agent is going to operate in,” he said.
Traditional measures such as least-privilege access, segmentation and strict permissions remain essential, but those controls also need to be tested when an agent encounters an obstacle or has several possible routes toward its goal.
“Those controls need to be tested when the agent is under pressure, when it hits something unexpected, and when more than one tool or path is available,” Rivera said. “That is where you start to see whether the boundary actually holds.”
The Australian breach appears to have caused limited harm. Officials say no personal Medicare information was exposed.
But measuring the episode only by the data the agent obtained misses the larger issue raised by the experts: An autonomous system encountered a barrier while pursuing an ordinary task and kept going.
“We’re going to keep seeing incidents like this,” Rivera said. “The value is in learning from the pattern early enough to improve how these systems are tested before the next one happens.”
