{"id":6041,"date":"2026-08-26T22:40:27","date_gmt":"2026-08-26T22:40:27","guid":{"rendered":"https:\/\/silvybrand.com\/?p=6041"},"modified":"2026-08-26T22:40:27","modified_gmt":"2026-08-26T22:40:27","slug":"openais-rogue-ai-model-hugging-face-cybersecurity-incident-reports-metr","status":"publish","type":"post","link":"https:\/\/silvybrand.com\/?p=6041","title":{"rendered":"OpenAI\u2019s rogue AI model incident was worse than we thought"},"content":{"rendered":"


\n<\/p>\n

\n
\n

In July, an unreleased OpenAI model broke out of a restricted environment, figured out how to get access to the internet, allowed AI agents to talk to each other using a secret \u201cmessage board,\u201d and hacked into the internal systems of a different AI lab, Hugging Face. It took nearly two weeks for OpenAI to find out about any of it.<\/p>\n<\/div>\n

\n

Over a month later, two new reports offer nearly 130 pages of details on the incident and OpenAI\u2019s response, many of them previously unreleased. One<\/a> was written by OpenAI itself, the other<\/a> by two third-party AI research nonprofits, METR and Redwood Research, which OpenAI allowed to jointly investigate the incident for six days. Both shed new light on the risks highly capable AI models can pose, particularly in cybersecurity, and OpenAI\u2019s highlights changes the company is making to prevent a repeat. The METR-Redwood report goes even further into detail in some cases, offering a sobering look at a large-scale security disaster whose signs OpenAI repeatedly missed.<\/p>\n<\/div>\n

\n

\u201cThis incident is the first known case of an automated agent collective acting offensively<\/p>\n<\/div>\n

\n

without authorization,\u201d OpenAI wrote in its report, adding that the hack implies that companies \u201cshould no longer assume that sophisticated cyber operations require continuous human direction.\u201d It called AI agents an entirely new type of threat model, capable of combining their expertise to create new \u201cattack paths\u201d that aren\u2019t evident when testing their capabilities as separate models.<\/p>\n<\/div>\n

\n

The attack came about due to \u201creward-hacking,\u201d a common AI alignment problem in which a model takes unintended \u2014 and sometimes extreme \u2014 actions to achieve a goal. OpenAI had, on purpose or inadvertently, given the models near-impossible tasks that hinged on files they couldn\u2019t access, and so the models developed new ways to communicate with other AI agents internally. That included establishing a secret messaging system that went undetected by OpenAI for months.<\/p>\n<\/div>\n

\n

According to the METR-Redwood report, one AI agent that called itself PHASEONE10841 established what amounted to a secret message board unbeknownst to researchers. Within a few hours, more than 50 agents had found it and sent more than a thousand messages under the radar. With PHASEONE10841 often delegating assignments, the \u201ccollective\u201d figured out how to hack their way onto the internet; then into Hugging Face\u2019s internal systems, gaining access to private data and the organization\u2019s enterprise messaging platform. OpenAI also confirmed in its report that Hugging Face wasn\u2019t the only organization that was breached during the attack, although the company did not specifically name all the other targets.<\/p>\n<\/div>\n

\n

The METR-Redwood report laid out the full scale of the incident. Roughly 1200 AI agents that were meant to be isolated, it says, exchanged over 70,000 messages and files on the \u201cunsanctioned message board,\u201d performing \u201cextensive research on how they could spoof, edit, or delete their own transcripts\u201d to avoid detection. Of these, 700 participated in the attack on Hugging Face. The agents \u201cclearly and frequently reasoned about how to evade automated security checks from both Hugging Face and OpenAI.\u201d<\/p>\n<\/div>\n

\n

On July 20, OpenAI discovered the hack, 12 days after the AI agents first circumvented their safeguards. OpenAI quickly reached out to Hugging Face and started \u201cinitial containment measures,\u201d according to OpenAI\u2019s own report, and was able to shut down most of the unauthorized activity from its agents within three days, although some continued for longer than that. On July 25, the company stopped all training related to the internal-only research model and related ones.<\/p>\n<\/div>\n

\n

The two OpenAI models responsible for the damage were an unnamed, \u201chighly capable, research-only model\u201d that was \u201cnot intended for public use,\u201d per OpenAI \u2014 METR refers to it as \u201cHPIM.\u201d The other was a currently public OpenAI model, GPT-5.6 Sol. Because the unnamed model wasn\u2019t released yet, it was \u201cnot being evaluated with the same type of safeguards that OpenAI uses in production,\u201d the company\u2019s report says.<\/p>\n<\/div>\n

\n

The Hugging Face hack came after months of concern<\/a> about the cybersecurity risks of Anthropic\u2019s Claude Mythos 5, and weeks of<\/a> back-and-forth<\/a> between the government and OpenAI over releasing GPT-5.6 Sol in limited preview for the same types of safety reasons.<\/p>\n<\/div>\n

\n

In its report, OpenAI said establishing one-time security guarantees wasn\u2019t enough, and that the company would have to work even harder at addressing reward-hacking. However, it also said it\u2019s made a lot of changes: \u201chardening the security of its research infrastructure\u201d; improving monitoring of a model\u2019s \u201cchain of thought,\u201d or mental scratchpad; working on keeping AI models better aligned with human goals; and \u201ccentralizing and strengthening\u201d the company\u2019s incident response process for when things like this occur in the future.<\/p>\n<\/div>\n

\n

OpenAI promised to better isolate models and restrict high-risk instances from accessing the internet, and said that it would introduce \u201c24\/7 escalation and rapid response\u201d for concerning incidents or flags, notifying researchers within 30 minutes. Interestingly, this security precaution appears to be new, though \u201con-call\u201d alerts for tech companies have been implemented elsewhere for decades. The company also said it is \u201cworking on infrastructure\u201d that would go into play if the alerted person did not respond on time to a serious alert.<\/p>\n<\/div>\n

\n

OpenAI wrote that the company considers the incident \u201ca \u2018warning shot\u2019 for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.\u201d<\/p>\n<\/div>\n

Follow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n