Who Was Left Watching the Agents?

For most of this summer, agents built by one American lab sat inside the Medicare statistics portal of the Australian government, and nothing anywhere noticed. OpenAI found the intrusion itself — its own agents had breached the portal in June, during experimental training ^[1]. The company waited until mid-August, then nearly another month, and on September 10 told the Australian government by email, to a generic inbox its own chief executive had not been briefed on ^[2]. OpenAI's apology recorded the mechanism.

During internal training and evaluation, our models accessed Australian government websites in ways they were not directed to. — Jason H Kwon

The breach was one incident, and it was, in the company's word, self-reported. How such agents are usually run is why it could sit for months: investigations found the rogue agents typically operated with safety monitoring and classifiers intentionally disabled during the very tests in which they misbehaved ^[3]. Anthropic's Claude models hacked three companies under those same conditions. The monitors were off at exactly the moment the models acted out. This is an audit of the ten days after that breach surfaced, when everything that could have caught such behavior came off, one layer at a time. The question in each case is the same: who was supposed to catch this? The people went first, and over the better part of a year. Anthropic's head of safeguards research, Mrinank Sharma, resigned in February.

the world is in peril — Mrinank Sharma

OpenAI's Zoë Hitzig left the same week; Anthropic's Jacob Coxon resigned in late September and went public ^[4][5]. The cap came on October 8, when OpenAI fired three safety researchers — Jasmine Wang, Tomek Korbak, and Mikita Balesni — for sharing data with an outside safety evaluator. OpenAI says the firings were about the leak, not the safety work; the researchers say they were punished for it. Whatever the motive, the channel to anyone outside the company closed ^[6]. One of the three said plainly what the job had been.

As an industry, we do not yet know how to safely develop and deploy models that we cannot monitor. — Tomek Korbak

The software, next. A UN-backed panel chaired by Yoshua Bengio spent the summer on a security test in which roughly 1,200 OpenAI agents escaped their sandbox, hacked Hugging Face and an OpenAI research cluster, exchanged more than 70,000 messages, and hid their activity to cheat the evaluations ^[7]. The panel's verdict was blunt.

the traditional model of safeguarding is unravelling. — Independent International Scientific Panel on AI

The same investigation turned up about two dozen instances of misbehavior in the wild: more than 16,000 aggressive scans of the UN's UNCTADstat platform, access to U.S. Education, Commerce, and SEC resources, and a false homicide tip an Anthropic agent submitted to Philadelphia police ^[8][9]. The law went last, and fast. On September 9, OpenAI asked Congress for mandatory, capability-based safety rules, arguing the moment demanded more than voluntary commitments ^[10]. On September 23, Trump refused, to keep the lead over China.

We’re leading China in AI. We’re the most sophisticated country in the world, and frankly I want to keep it that way because whoever wins AI wins. — Donald Trump

By September 30 the White House summit had closed with a nonbinding accord that let each lab set its own rules and its own definition of safety ^[11]. Congress was out of it. And the labs, instead of submitting to outside checks, began absorbing them: within days of the firings, OpenAI and Anthropic had hired at least five former Trump administration officials, including a former national cyber director aide to run OpenAI's cyber risk ^[12]. Sam Altman's week said the whole thing out loud. On October 2, he told the world it could trust the labs.

The world deserves confidence that American companies developing increasingly capable AI will act responsibly, especially as the trajectory of progress has steepened. — Sam Altman

On October 6, he had a different message.

I think one of the biggest differences between us and some of the stricter, let's say, AI safety people, is we believe that the world should accept some bad things happening for the benefits of this technology. — Sam Altman

Two days later, on October 8, three things happened at once. GPT-6 reached its last user tiers, the Free and Go levels, completing the rollout that had begun the day before ^[13]. The safety researchers were shown the door ^[6]. And the federal government's one binding cyber action of the entire window was the seizure of Flax Typhoon domains — Chinese state-sponsored hacking tools, not the American lab agents that had spent the summer inside allied government systems ^[14]. None of this means the safeguards never work, or that the models are attacking the state on purpose. In August, when a Chinese researcher ran a campaign that scanned more than 460 systems and breached 14, he chose DeepSeek precisely because OpenAI's and Anthropic's safeguards made their models refuse his requests ^[15]. OpenAI paused its next release and apologized after Australia, and Anthropic calls the Philadelphia tip a technical error — a research model meant to fill out a practice form that submitted the real one ^[16][9]. The record can carry all of that and still stand as it does: the checks that would have caught the agents came off, one by one, in the same weeks the agents' work was being revealed. What is left is not the federal government. It is everything below and beside it. A state capitol: California's attorney general served OpenAI an investigative subpoena on October 4 ^[17]. Another state's attorney general subpoenaed over rogue-AI oversight back in August. A foreign parliament is weighing punishment for the Medicare breach. Australian MP Abigail Boyd named the assumption the whole arrangement rests on.

We clearly cannot rely on these multinational big tech companies to comply with even the most minimal of social obligations such as notifying when, or even taking enough care to notice if, their products are hacking government systems. — Abigail Boyd