{"id":7017,"date":"2026-09-14T23:33:25","date_gmt":"2026-09-14T23:33:25","guid":{"rendered":"https:\/\/silvybrand.com\/?p=7017"},"modified":"2026-09-14T23:33:25","modified_gmt":"2026-09-14T23:33:25","slug":"is-big-techs-ai-slowdown-a-safety-pact-or-a-cartel","status":"publish","type":"post","link":"https:\/\/silvybrand.com\/?p=7017","title":{"rendered":"Is Big Tech\u2019s AI slowdown a safety pact or a cartel?"},"content":{"rendered":"


\n<\/p>\n

\n
\n

When OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind cofounder Demis Hassabis, and SpaceX head Elon Musk loosely agreed over the weekend to slow down AI development, skeptics spotted an ulterior motive immediately. The AI titans had declared that their aim was to \u201cpace the frontier,\u201d signing on at least partially to a proposal for embedding third-party auditors, regulating domestic labs, and reaching a global slowdown agreement. Their critics, however, argued they simply wanted to stop would-be competitors, kneecap the open-source movement, and avoid real legal safeguards \u2014 some<\/a> dubbed it an outright \u201ccartel.\u201d<\/p>\n<\/div>\n

\n

The truth is more complicated, according to sources across the industry. The three-step proposal, laid out in an essay<\/a> by Amodei, is calling for changes long espoused by AI safety advocates. While it could become a substitute for meaningful regulation, under Trump, regulation is unlikely anyway. But experts say that on an issue that\u2019s only likely to grow in importance, AI leaders aren\u2019t the best people to lead the charge.<\/p>\n<\/div>\n

\n

\u201cThe industry as a whole needs new champions,\u201d said Nick Reese, an adjunct professor at New York University and the Department of Homeland Security\u2019s former director of emerging tech policy. \u201cWe\u2019ve looked at people like Dario Amodei, Sam Altman, and Elon Musk as these people who are closest to the problem and working on it every day, and they\u2019re the ones who know best. But the truth is there\u2019s never been a realistic vision for what we\u2019re building toward.\u201d<\/p>\n<\/div>\n

\n
\n

\u201cThe industry as a whole needs new champions.\u201d<\/p>\n<\/div>\n<\/div>\n

\n

Concerns about AI have been escalating for months, sparked largely by revelations that agents were behind rogue hacks going on right under the noses of leading frontier labs Anthropic and OpenAI. Reports from both companies fueled the fire, as did the resignation of Anthropic researcher Jacob Coxon, who posted a public letter<\/a> explaining his choice. \u201cThe people building AI earnestly believe that it could kill us all by the end of the decade,\u201d he wrote, adding that neither OpenAI nor Anthropic is \u201cacting responsibly\u201d and rather \u201cracing straight to self-improving superintelligence and gambling with our lives.\u201d<\/p>\n<\/div>\n

\n

Dire warnings about AI aren\u2019t anything new inside the industry, but Coxon\u2019s letter seemed to break containment. It\u2019s been viewed more than 170 million times on X alone, while Coxon and his story have made the rounds in newspapers and on TV.<\/p>\n<\/div>\n

\n

To a significant chunk of AI safety researchers and AI nonprofit workers, however, this is simply drawing more attention to an already widely discussed issue. \u201cA lot of people are thinking of this as Dario\u2019s idea, or it\u2019s coming from the CEOs, but that\u2019s false,\u201d said Daniel Kokotajlo, an ex-OpenAI employee who now leads the AI Futures Project, an AI research nonprofit. \u201cPeople outside the companies have been calling for this for years \u2026 There\u2019s been this growing chorus of voices saying, \u2018Please don\u2019t build superintelligence soon. We are not ready. You need to slow down.\u2019\u201d<\/p>\n<\/div>\n

\n

Kokotajlo said that after years of people \u201cyelling at them to do this\u201d \u2014 including more than 1,000 AI lab employees signing a public letter<\/a> from July, which called for a slowdown in AI development after the OpenAI-Hugging Face incident \u2014 the CEOs are now \u201cnow bowing to that pressure and also claiming credit for it, [although] not rightfully.\u201d<\/p>\n<\/div>\n

\n

Overall, a handful of sources told The Verge<\/em> they believe that the verbal agreement made by the AI leaders is a step in the right direction. NYU\u2019s Reese said he \u201cdo[es] not think it\u2019s all hot air.\u201d Apollo Research CEO Marius Hobbhahn called it \u201ca good idea,\u201d saying \u201cit\u2019s one of the best things for safety in a long time if it actually happens.\u201d The Midas Project\u2019s Tyler Johnston said it was a \u201cgood sign.\u201d Redwood Research CEO Buck Shlegeris said it was \u201csome great news \u2014 obviously it\u2019s hard to know whether this is going to convert into anything real, but I\u2019m feeling cautiously optimistic.\u201d All agreed, though, that there was still a lot of work to do to make this verbal commitment a reality, particularly when it comes to making it an ironclad agreement.<\/p>\n<\/div>\n

\n
\n

\u201cIt\u2019s one of the best things for safety in a long time if it actually happens.\u201d<\/p>\n<\/div>\n<\/div>\n

\n

Many still question the motivations of AI\u2019s leaders. The larger tech industry has spent years getting ahead of regulation by lobbying for its own preferred rules or promising self-regulation. Big platforms have proposed policies that could hit smaller competitors harder, using altruistic language to justify self-serving goals. They\u2019ve been accused of safety-washing, or making meaningless changes that give the false impression of actual safeguards. It\u2019s no surprise people are concerned this will happen in the AI industry as well, particularly since AI labs\u2019 voluntary safety frameworks have been criticized for years.<\/p>\n<\/div>\n

\n

Several sources believe that concerns of safety-washing are valid. \u201cThere\u2019s a serious concern that they\u2019re not actually going to slow down,\u201d Kokotajlo says, adding that the fear is that \u201cthey\u2019ll just bring in some external auditors, do a bunch of safety paperwork \u2014 some of which will be genuinely good \u2014 but at the end of the day, it actually won\u2019t slow them down very much at all.\u201d<\/p>\n<\/div>\n

\n

NYU\u2019s Reese compared this gambit to the social media platforms\u2019 playbook a decade ago, when companies began calling for regulatory action to get ahead of impending, less favorable laws. For the AI industry, Reese said, \u201cthe hammer may not come in this administration, but I think if there were a Democratic administration after the next election, there would be a really good possibility.\u201d<\/p>\n<\/div>\n

\n

That kind of regulation would be vital, said Sacha Haworth, executive director of the Tech Oversight Project \u2014 any voluntary framework is essentially regulatory capture and that \u201cwe should not be letting the foxes run the henhouse. This is not an opportunity for Congress to once again outsource responsibility to industry.\u201d Daniel Lobo-Lewis, co-founder of the Political Integrity Project, said that voluntary regulation will likely go the way of Meta\u2019s largely toothless Oversight Board.<\/p>\n<\/div>\n

\n

Most people also, however, believe companies are in no imminent danger. President Trump posted<\/a> on Monday that \u201cthe only control or \u2018guardrails\u2019 that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!\u201d He also called Nvidia CEO Jensen Huang while Huang was onstage at a conference \u2014 Trump told the crowd on speakerphone<\/a> that recent concerns about AI were a \u201choax\u201d and that \u201cthe robots will not be taking over.\u201d<\/p>\n<\/div>\n

\n

As for concerns that regulation could hold back other companies or open source developers, calls for slowdowns have been almost exclusively aimed at big frontier AI labs defined by exact size metrics, many industry experts told The Verge<\/em>.<\/p>\n<\/div>\n

\n

With no government AI regulation incoming, the best option may be to push companies into an immediate, measurable, and enforceable agreement. Kokotajlo\u2019s AI Futures Project, for instance, proposed<\/a> AI labs giving auditors access to their compute budget. Amodei\u2019s essay already called for AI labs to allow external third-party auditors \u2014 like METR, Apollo, and Redwood Research \u2014 to embed within their organizations to some extent and potentially be able to flag, and blow the whistle on, problematic findings. Kokotajlo says companies would ideally need to cut down their own compute budgets for research in the process, slowing down development and allowing other AI labs to catch up.<\/p>\n<\/div>\n

\n
\n

\u201cIf I am going to die at the hands of killer AI, I want it to be American, not Chinese.\u201d<\/p>\n<\/div>\n<\/div>\n

\n

Arguably the single biggest challenge to an AI slowdown can be summarized in one word: China.<\/p>\n<\/div>\n

\n

AI leaders and politicians alike have long positioned China as the reason why US AI progress can\u2019t slow down \u2014 because no matter how dangerous the technology may become, they\u2019d rather it be in US hands than Chinese ones, and China won\u2019t pump the brakes. Lobo-Lewis compared China fears to the Cold War missile gap. One X user wrote<\/a>, \u201cIf I am going to die at the hands of killer AI, I want it to be American, not Chinese.\u201d<\/p>\n<\/div>\n

\n

Redwood Research\u2019s Shlegeris said it wasn\u2019t in the best interest of neither the US nor the Chinese government to pursue AI development in a \u201creckless\u201d way, adding that \u201cit would not be unprecedented levels of international coordination.\u201d<\/p>\n<\/div>\n

\n

\u201c[People are] taking for granted that China won\u2019t cooperate,\u201d Johnston said, adding, \u201cThis is an issue so serious and so widespread that it seems like it\u2019s in everyone\u2019s interest to coordinate on it, in the same way it was in both the US and Russia\u2019s interest to coordinate on nuclear de-proliferation.\u201d<\/p>\n<\/div>\n

\n

On Monday, Chinese Foreign Ministry spokesperson Guo Jiakun did push back<\/a> on the calls for a slowdown, calling it \u201cfearmongering.\u201d<\/p>\n<\/div>\n

\n

The Midas Project\u2019s Johnston and Redwood Research\u2019s Shlegeris both said that even in the absence of Chinese cooperation, it\u2019s still vital to encourage US coordination. And for the Tech Oversight Project\u2019s Haworth, the slowdown presents an opportunity to \u201cposition ourselves as the compass for how AI technology can be developed and used.\u201d She sees arguments to the contrary as part of a familiar playbook. \u201cChina gets brought up as a bogeyman every time that an industry wants to escape oversight.\u201d<\/p>\n<\/div>\n

\n

Kokotajlo simply compared the situation to a cartoon he\u2019d seen, with people in a car driving off a cliff. He recalled the speech bubble saying something like, \u201cHooray, we\u2019re ahead of China.\u201d<\/p>\n<\/div>\n

\n
\n

\u201cWe really do earnestly believe AI could kill all humans!\u201d<\/p>\n<\/div>\n<\/div>\n

\n

The issue underlying all the recent panic is a milestone known as recursive self-improvement, and based on the industry\u2019s current trajectory, we\u2019re only hearing the beginning of the alarm bells.<\/p>\n<\/div>\n

\n

RSI refers to a hypothetical point where AI models can train, advance, and create new versions of themselves, all without human involvement. Anthropic has said<\/a> this point could come as soon as early 2027, and OpenAI\u2019s chief scientist wrote<\/a> earlier this month that OpenAI is directing a lot of resources towards reaching this milestone. Increasingly, engineers and researchers in the AI industry worry that RSI precedes a whole host of new and more serious AI concerns, including more severe cybersecurity incidents that more deeply impact society at large.<\/p>\n<\/div>\n

\n

Amodei cited RSI as a major factor in his decision to call for a slowdown: \u201csince roughly this summer, AI has been advancing drastically faster\u201d and RSI is \u201cstarting to happen across the industry, including at Anthropic \u2026 Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,\u201d he wrote in his recent essay. He proposed implementing \u201csome kind of \u2018speed limit\u2019\u201d on the rate of RSI and compared it to caps on missile numbers.<\/p>\n<\/div>\n

\n

In Jacob Coxon\u2019s resignation post from Anthropic, he warned of \u201csuperhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.\u201d Many other researchers at leading AI labs echoed his concerns. It\u2019s \u201chard to overstate how dangerous\u201d it is to speed towards RSI, wrote<\/a> Jasmine Wang, an OpenAI researcher. Vishal Maini, an ex-Google DeepMind employee, said RSI is \u201cnow so imminent that no other option makes sense.\u201d<\/p>\n<\/div>\n

\n

Fears about RSI can easily turn apocalyptic. \u201cAI developers believe their technology could cause human extinction (or similarly bad outcomes),\u201d Samuel Marks, an Anthropic researcher, wrote<\/a> on X \u201cThis could happen in the next few years. In general, the more senior the employee, the more concerned they are.\u201d Another Anthropic researcher and team lead, Evan Hubinger, wrote<\/a> on X that \u201cJacob is correct here\u2014we really do earnestly believe AI could kill all humans!\u201d (He put the chance at greater than 10 percent over the next decade.) Alex Turner, a former Google DeepMind employee, wrote<\/a> that \u201cmany researchers believe they are building something that could kill everyone on the planet. It was literally my day job to think about how to stop that.\u201d<\/p>\n<\/div>\n

\n

Micah Carroll, an OpenAI researcher, wrote<\/a> that Coxon\u2019s belief is a \u201ccross-partisan position\u201d with research teams across \u201call frontier AI companies,\u201d and that they all believe that \u201cbusiness-as-usual AI development poses unacceptable catastrophic risk.\u201d<\/p>\n<\/div>\n

\n

\u201cBut,\u201d Carroll added, \u201cwe should also not hyperstition catastrophic risks into existence \u2013 they can be greatly reduced via safety requirements with teeth, international coordination, and a consensus to not build [artificial superintelligence] unless there are sufficient safety advances to make us collectively confident to do so.\u201d<\/p>\n<\/div>\n

\n

That kind of \u201cinternational coordination\u201d is exactly what AI leaders, and the public at large, are now calling for.<\/p>\n<\/div>\n

\n

\u201cWe have never been in a situation where we actually understood what we were driving towards with regard to AI,\u201d NYU\u2019s Reese said. \u201cWe\u2019ve always had this amorphous undefined end state that we don\u2019t really understand but we have to beat China to get to. It\u2019s really hard to race when we don\u2019t even understand the path or even understand what the finish line is \u2014 or even if there is one.\u201d<\/p>\n<\/div>\n

Follow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n