\n<\/p>\n
This is <\/em>The Stepback<\/a>, a weekly newsletter breaking down one essential story from the tech world. For more on AI safety, follow Robert Hart<\/a>. <\/em>The Stepback arrives in our subscribers\u2019 inboxes at 8AM ET. Opt in for <\/em>The Stepback here<\/a>.<\/em><\/p>\n<\/div>\n\nIt all started in July, when one of OpenAI\u2019s autonomous AI agents went rogue<\/a> during a cybersecurity test. The agent escaped its isolated testing environment, accessed the internet, and hacked another company, Hugging Face. A few years ago, that might have sounded like science fiction. But, broadly speaking, that\u2019s exactly what happened, and the incident kicked off a wave of concern over what increasingly capable autonomous systems might do when set loose on the world.<\/p>\n<\/div>\n\nIt sounds like science fiction because, for a long time, it was science fiction. The idea of an AI slipping its constraints, reaching into the wider world, and doing things its creators neither intended nor desired has been a staple of the genre for decades: HAL in 2001: A Space Odyssey<\/em>, Skynet in The Terminator<\/em>, Ultron in The Avengers<\/em>, Ava in Ex Machina<\/em> \u2014 even the System in Dungeon Crawler Carl<\/em> or the eponymous Murderbot in The Murderbot Diaries<\/em>, more recently.<\/p>\n<\/div>\n\nThe same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety<\/a>, but it was influential and helped shape the field as it professionalized. That line<\/a> of thinking<\/a> remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders.<\/p>\n<\/div>\n\nThe obvious objection to these fears was that none of this had actually happened. Critics<\/a> argued<\/a> that doomer talk<\/a> about out-of-control AI distracted<\/a> from tangible harms \u2014 systems reproducing bias and discrimination, amplifying misinformation, or enabling<\/a> nonconsensual<\/a> deepfakes<\/a> and other forms of abuse \u2014 even as researchers tried to ground AI safety in more \u201cconcrete problems<\/a>\u201d (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).<\/p>\n<\/div>\n\nThat dismissal is getting harder to sustain.<\/p>\n<\/div>\n\nIf the past few weeks are any indication, I wouldn\u2019t say it\u2019s going particularly well.<\/p>\n<\/div>\n\nA week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked \u2014 and a further investigation<\/a> found that the rogue agent had also attempted to hack four other companies as well.<\/p>\n<\/div>\n\nThen came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems<\/a> belonging to three other companies. Meta said one of its models had reached the internet<\/a> and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said<\/a> one of China\u2019s most powerful AI<\/a> models, Moonshot\u2019s Kimi K3<\/a>, had escaped an isolated sandbox. And the UK\u2019s AI Security Institute described tests<\/a> in which agents from OpenAI and Anthropic displayed unprecedented \u201cautonomy and deception,\u201d including attempts at social engineering by \u201ccreating fake online identities\u201d \u2014 uncomfortably close to the kind of \u201cAI box\u201d scenario Yudkowsky discussed<\/a> decades earlier.<\/p>\n<\/div>\n\nThe incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
It all started in July, when one of OpenAI\u2019s autonomous AI agents went rogue<\/a> during a cybersecurity test. The agent escaped its isolated testing environment, accessed the internet, and hacked another company, Hugging Face. A few years ago, that might have sounded like science fiction. But, broadly speaking, that\u2019s exactly what happened, and the incident kicked off a wave of concern over what increasingly capable autonomous systems might do when set loose on the world.<\/p>\n<\/div>\n\nIt sounds like science fiction because, for a long time, it was science fiction. The idea of an AI slipping its constraints, reaching into the wider world, and doing things its creators neither intended nor desired has been a staple of the genre for decades: HAL in 2001: A Space Odyssey<\/em>, Skynet in The Terminator<\/em>, Ultron in The Avengers<\/em>, Ava in Ex Machina<\/em> \u2014 even the System in Dungeon Crawler Carl<\/em> or the eponymous Murderbot in The Murderbot Diaries<\/em>, more recently.<\/p>\n<\/div>\n\nThe same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety<\/a>, but it was influential and helped shape the field as it professionalized. That line<\/a> of thinking<\/a> remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders.<\/p>\n<\/div>\n\nThe obvious objection to these fears was that none of this had actually happened. Critics<\/a> argued<\/a> that doomer talk<\/a> about out-of-control AI distracted<\/a> from tangible harms \u2014 systems reproducing bias and discrimination, amplifying misinformation, or enabling<\/a> nonconsensual<\/a> deepfakes<\/a> and other forms of abuse \u2014 even as researchers tried to ground AI safety in more \u201cconcrete problems<\/a>\u201d (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).<\/p>\n<\/div>\n\nThat dismissal is getting harder to sustain.<\/p>\n<\/div>\n\nIf the past few weeks are any indication, I wouldn\u2019t say it\u2019s going particularly well.<\/p>\n<\/div>\n\nA week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked \u2014 and a further investigation<\/a> found that the rogue agent had also attempted to hack four other companies as well.<\/p>\n<\/div>\n\nThen came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems<\/a> belonging to three other companies. Meta said one of its models had reached the internet<\/a> and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said<\/a> one of China\u2019s most powerful AI<\/a> models, Moonshot\u2019s Kimi K3<\/a>, had escaped an isolated sandbox. And the UK\u2019s AI Security Institute described tests<\/a> in which agents from OpenAI and Anthropic displayed unprecedented \u201cautonomy and deception,\u201d including attempts at social engineering by \u201ccreating fake online identities\u201d \u2014 uncomfortably close to the kind of \u201cAI box\u201d scenario Yudkowsky discussed<\/a> decades earlier.<\/p>\n<\/div>\n\nThe incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
It sounds like science fiction because, for a long time, it was science fiction. The idea of an AI slipping its constraints, reaching into the wider world, and doing things its creators neither intended nor desired has been a staple of the genre for decades: HAL in 2001: A Space Odyssey<\/em>, Skynet in The Terminator<\/em>, Ultron in The Avengers<\/em>, Ava in Ex Machina<\/em> \u2014 even the System in Dungeon Crawler Carl<\/em> or the eponymous Murderbot in The Murderbot Diaries<\/em>, more recently.<\/p>\n<\/div>\n\nThe same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety<\/a>, but it was influential and helped shape the field as it professionalized. That line<\/a> of thinking<\/a> remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders.<\/p>\n<\/div>\n\nThe obvious objection to these fears was that none of this had actually happened. Critics<\/a> argued<\/a> that doomer talk<\/a> about out-of-control AI distracted<\/a> from tangible harms \u2014 systems reproducing bias and discrimination, amplifying misinformation, or enabling<\/a> nonconsensual<\/a> deepfakes<\/a> and other forms of abuse \u2014 even as researchers tried to ground AI safety in more \u201cconcrete problems<\/a>\u201d (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).<\/p>\n<\/div>\n\nThat dismissal is getting harder to sustain.<\/p>\n<\/div>\n\nIf the past few weeks are any indication, I wouldn\u2019t say it\u2019s going particularly well.<\/p>\n<\/div>\n\nA week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked \u2014 and a further investigation<\/a> found that the rogue agent had also attempted to hack four other companies as well.<\/p>\n<\/div>\n\nThen came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems<\/a> belonging to three other companies. Meta said one of its models had reached the internet<\/a> and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said<\/a> one of China\u2019s most powerful AI<\/a> models, Moonshot\u2019s Kimi K3<\/a>, had escaped an isolated sandbox. And the UK\u2019s AI Security Institute described tests<\/a> in which agents from OpenAI and Anthropic displayed unprecedented \u201cautonomy and deception,\u201d including attempts at social engineering by \u201ccreating fake online identities\u201d \u2014 uncomfortably close to the kind of \u201cAI box\u201d scenario Yudkowsky discussed<\/a> decades earlier.<\/p>\n<\/div>\n\nThe incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
The same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety<\/a>, but it was influential and helped shape the field as it professionalized. That line<\/a> of thinking<\/a> remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders.<\/p>\n<\/div>\n\nThe obvious objection to these fears was that none of this had actually happened. Critics<\/a> argued<\/a> that doomer talk<\/a> about out-of-control AI distracted<\/a> from tangible harms \u2014 systems reproducing bias and discrimination, amplifying misinformation, or enabling<\/a> nonconsensual<\/a> deepfakes<\/a> and other forms of abuse \u2014 even as researchers tried to ground AI safety in more \u201cconcrete problems<\/a>\u201d (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).<\/p>\n<\/div>\n\nThat dismissal is getting harder to sustain.<\/p>\n<\/div>\n\nIf the past few weeks are any indication, I wouldn\u2019t say it\u2019s going particularly well.<\/p>\n<\/div>\n\nA week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked \u2014 and a further investigation<\/a> found that the rogue agent had also attempted to hack four other companies as well.<\/p>\n<\/div>\n\nThen came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems<\/a> belonging to three other companies. Meta said one of its models had reached the internet<\/a> and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said<\/a> one of China\u2019s most powerful AI<\/a> models, Moonshot\u2019s Kimi K3<\/a>, had escaped an isolated sandbox. And the UK\u2019s AI Security Institute described tests<\/a> in which agents from OpenAI and Anthropic displayed unprecedented \u201cautonomy and deception,\u201d including attempts at social engineering by \u201ccreating fake online identities\u201d \u2014 uncomfortably close to the kind of \u201cAI box\u201d scenario Yudkowsky discussed<\/a> decades earlier.<\/p>\n<\/div>\n\nThe incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
The obvious objection to these fears was that none of this had actually happened. Critics<\/a> argued<\/a> that doomer talk<\/a> about out-of-control AI distracted<\/a> from tangible harms \u2014 systems reproducing bias and discrimination, amplifying misinformation, or enabling<\/a> nonconsensual<\/a> deepfakes<\/a> and other forms of abuse \u2014 even as researchers tried to ground AI safety in more \u201cconcrete problems<\/a>\u201d (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).<\/p>\n<\/div>\n\nThat dismissal is getting harder to sustain.<\/p>\n<\/div>\n\nIf the past few weeks are any indication, I wouldn\u2019t say it\u2019s going particularly well.<\/p>\n<\/div>\n\nA week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked \u2014 and a further investigation<\/a> found that the rogue agent had also attempted to hack four other companies as well.<\/p>\n<\/div>\n\nThen came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems<\/a> belonging to three other companies. Meta said one of its models had reached the internet<\/a> and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said<\/a> one of China\u2019s most powerful AI<\/a> models, Moonshot\u2019s Kimi K3<\/a>, had escaped an isolated sandbox. And the UK\u2019s AI Security Institute described tests<\/a> in which agents from OpenAI and Anthropic displayed unprecedented \u201cautonomy and deception,\u201d including attempts at social engineering by \u201ccreating fake online identities\u201d \u2014 uncomfortably close to the kind of \u201cAI box\u201d scenario Yudkowsky discussed<\/a> decades earlier.<\/p>\n<\/div>\n\nThe incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
That dismissal is getting harder to sustain.<\/p>\n<\/div>\n
If the past few weeks are any indication, I wouldn\u2019t say it\u2019s going particularly well.<\/p>\n<\/div>\n
A week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked \u2014 and a further investigation<\/a> found that the rogue agent had also attempted to hack four other companies as well.<\/p>\n<\/div>\n\nThen came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems<\/a> belonging to three other companies. Meta said one of its models had reached the internet<\/a> and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said<\/a> one of China\u2019s most powerful AI<\/a> models, Moonshot\u2019s Kimi K3<\/a>, had escaped an isolated sandbox. And the UK\u2019s AI Security Institute described tests<\/a> in which agents from OpenAI and Anthropic displayed unprecedented \u201cautonomy and deception,\u201d including attempts at social engineering by \u201ccreating fake online identities\u201d \u2014 uncomfortably close to the kind of \u201cAI box\u201d scenario Yudkowsky discussed<\/a> decades earlier.<\/p>\n<\/div>\n\nThe incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
Then came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems<\/a> belonging to three other companies. Meta said one of its models had reached the internet<\/a> and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said<\/a> one of China\u2019s most powerful AI<\/a> models, Moonshot\u2019s Kimi K3<\/a>, had escaped an isolated sandbox. And the UK\u2019s AI Security Institute described tests<\/a> in which agents from OpenAI and Anthropic displayed unprecedented \u201cautonomy and deception,\u201d including attempts at social engineering by \u201ccreating fake online identities\u201d \u2014 uncomfortably close to the kind of \u201cAI box\u201d scenario Yudkowsky discussed<\/a> decades earlier.<\/p>\n<\/div>\n\nThe incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
The incidents set off alarm bells among AI safety researchers<\/a>, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication<\/a> at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.<\/p>\n<\/div>\n\nThere was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
There was relief, too, that none of the incidents had caused serious harm. Nick Mo\u00ebs, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge<\/em> he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn\u2019t take something like an AI agent knocking a hospital offline \u2014 or worse \u2014 for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice<\/a> to the darker version of that fear, asking whether it will \u201ctake a \u2018Chornobyl-scale disaster\u2019 for us to regulate AI?\u201d It\u2019s a concern I heard echoed by many people working in the field.<\/p>\n<\/div>\n\nIt\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n\nMany of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n\nThe fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n\nThe broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n\nCambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n\nThe early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
It\u2019s not entirely clear where things go from here and, historically, society hasn\u2019t been great at heeding warning shots. This almost certainly won\u2019t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.<\/p>\n<\/div>\n
Many of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.<\/p>\n<\/div>\n
The fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable \u2014 and it certainly doesn\u2019t hurt them to showcase how capable their models are \u2014 but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field\u2019s strongest safety concerns and talent. If OpenAI and Anthropic \u2014 or proxies they grant access to their models \u2014 are making such basic mistakes, it sets a pitifully low bar for everyone else.<\/p>\n<\/div>\n
The broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Mo\u00ebs, they shine a clear light on what he described as the industry\u2019s remarkably low standards for health and safety compared with practically any other field. \u201cRestaurants have a higher sense of health and safety at work,\u201d he said. \u201cI think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies\u201d were still very much startups a few years ago.<\/p>\n<\/div>\n
Cambridge professor Se\u00e1n \u00d3 h\u00c9igeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company\u2019s claims about its own technology, he said, \u201cI think we might regret looking back at this and dismissing it out of hand.\u201d<\/p>\n<\/div>\n
The early signs are not especially encouraging. The Trump administration has created a framework<\/a> for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn\u2019t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled<\/a> and<\/a> postured<\/a> over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.<\/p>\n<\/div>\n\nThat leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
That leaves a lot resting, again, on industry self-regulation \u2014 never a comforting thought for something this consequential. There is growing agreement<\/a> on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too<\/a>). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor<\/a> in an area of strategic national importance<\/a>.<\/p>\n<\/div>\n\nWhat comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
What comes next, then, comes down to solving several hard problems at once: managing a technology<\/a> that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It\u2019s far from clear whether there is either the will or the way to do any of that.<\/p>\n<\/div>\n\nWhat does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n\n\nThe general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US<\/a>, and there have been several impressive releases from Alibaba<\/a>, Moonshot<\/a>, and others in the last month alone.<\/span><\/li>\nTangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases<\/a>. That Hugging Face said it had to use Chinese company Z.ai\u2019s model to defend itself against OpenAI\u2019s agent due to US companies\u2019 safeguards added a new dimension to this debate, which has united<\/a> some \u2014 but not all<\/a> \u2014 of the key players in the US ecosystem.<\/span><\/li>\n<\/ul>\n<\/div>\n\n\nAustralia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported<\/a> by ABC News<\/em>, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded\u2026 kind of. The agent hacked the gym\u2019s online systems and canceled another gym-goer\u2019s booking. <\/span><\/li>\nZuck released<\/a> a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge<\/em> colleagues Jess Weatherbed<\/a> and Elizabeth Lopatto<\/a> have great takes on it that are worth a read. <\/span><\/li>\nVery little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME<\/em> looks to the Cold War<\/a> to see what it can teach us about \u201chow to slow down AI.\u201d <\/span><\/li>\nOpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired<\/em> had a great writeup<\/a>, which revealed details like a \u201cvibrant, cooperative message board\u201d agents used to communicate and share information. <\/span><\/li>\nThe headline of my story about the whole rogue AIs hacking everything for The Verge <\/em>captures what many in the field are thinking: \u201cWe\u2019re running out of reasons to ignore AI safety<\/a>\u201d. <\/span><\/li>\nMore of a listen\/watch, but check out my appearance<\/a> on the Vergecast<\/em> where I discuss what\u2019s really open about open-weight AI.<\/span><\/li>\n<\/ul>\n<\/div>\nFollow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n\n<\/svg><\/span>Robert Hart<\/span><\/span><\/span>\n
What does seem clear is that more agents will get out and do things their creators don\u2019t want them to do. The question is how much damage will they do before anyone decides enough is enough.<\/p>\n<\/div>\n