{"id":6414,"date":"2026-09-03T00:55:35","date_gmt":"2026-09-03T00:55:35","guid":{"rendered":"https:\/\/silvybrand.com\/?p=6414"},"modified":"2026-09-03T00:55:35","modified_gmt":"2026-09-03T00:55:35","slug":"openai-astra-ai-monitoring-safety","status":"publish","type":"post","link":"https:\/\/silvybrand.com\/?p=6414","title":{"rendered":"Researchers fear safety disaster ahead of OpenAI\u2019s Astra release"},"content":{"rendered":"


\n<\/p>\n

\n
\n

OpenAI is on the cusp of releasing its most powerful AI model yet, Astra, following weeks of delays<\/a> to shore up safety protocols<\/a> after its agents attacked real targets<\/a> during testing. As details about the model trickle out, researchers are warning<\/a> it \u201cmay be the single worst development for AI security\/safety to date.\u201d<\/p>\n<\/div>\n

\n

Shortly after OpenAI said<\/a> on Tuesday that it had delayed Astra\u2019s release to work on safety issues, The Information<\/em> reported<\/a> that Astra shows far less of its \u201cthinking\u201d than other frontier AI models, sparking concern it could be dangerously hard to monitor.<\/p>\n<\/div>\n

\n

Most top AI systems today are built using a technology known as a transformer, which processes some types of information linearly through layers before producing an answer. Models can be made to show their reasoning as they go, essentially \u201cthinking out loud.\u201d This \u201cchain of thought\u201d allows researchers and automated safety systems to monitor what AI models are doing and potentially spot undesirable behavior, such as lying or plans to circumvent safety guardrails, before they act.<\/p>\n<\/div>\n

\n

According to The Information<\/em>, citing an unnamed person familiar with the unreleased model\u2019s development, Astra uses a more opaque technique known as a recurrent depth or looped transformer, which cycles information through internal layers before producing an output. This would mean much more of the model\u2019s \u201cthinking\u201d happens inside the system, and in a form that looks a lot less like natural human language, rather than being expressed in a way that researchers can easily monitor. This can boost model performance, but makes potential threats and unwanted behavior harder to detect.<\/p>\n<\/div>\n

\n

OpenAI has limited its use of the looped transformer \/ recurrent depth technique with Astra so researchers can continue to monitor the model\u2019s reasoning, according to The Information\u2019s<\/em> unnamed source.<\/p>\n<\/div>\n

\n

In a blog post<\/a> published Tuesday, OpenAI said it is \u201cdeploying Astra with additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions.\u201d It did not mention if the model has a different technical foundation.<\/p>\n<\/div>\n

\n

The Information<\/em>\u2019s report sparked widespread concern among AI safety researchers on social media. It was Redwood Research\u2019s chief scientist Ryan Greenblatt, one of three outsiders OpenAI permitted to research<\/a> the Hugging Face hack, who said<\/a> a decision to use a more opaque architecture for Astra \u201cmay be the single worst development for AI security\/safety to date.\u201d<\/p>\n<\/div>\n

\n

Greenblatt said the investigation into the Hugging Face incident relied heavily on the models\u2019 chain-of-thought, warning that less visible reasoning could allow AI systems to devise and execute strategies that would be far harder for researchers to detect.<\/p>\n<\/div>\n

\n

Greenblatt\u2019s primary concern, echoed<\/a> by other<\/a> safety experts, is that competition to develop more advanced AI systems could lead to \u201ca race to the bottom on architectures that could be catastrophic for our ability to oversee\/monitor AIs\u201d \u2014 with developers adopting increasingly opaque systems to gain an edge until models become difficult, or even impossible, to monitor. He added that OpenAI\u2019s communications left him concerned that the company \u201cplans on being extremely reliant on chain-of-thought monitoring for safety.\u201d<\/p>\n<\/div>\n

\n

OpenAI bigwigs responded to the criticism in a series of social media posts that do not explicitly deny the company\u2019s use of the technique. Several expressed concerns about the possibility of unmonitorable AI or a race to the bottom in terms of transparency, including OpenAI safety researchers Micah Carroll<\/a> and Tomek Korbak<\/a>, head of strategic futures Dean Ball<\/a>, and chief scientist Jakub Pachocki<\/a>, who voiced fears of \u201ca race into unmonitorability kicked off by confused reporting.\u201d He said the depth of Astra\u2019s computation \u2014 a measure of how many steps it can perform internally \u2014 \u201cis within a factor of two of GPT-4,\u201d indicating that if the technique was used, the increased opacity is less dramatic than some reactions imply. OpenAI did not respond to The Verge<\/em>\u2019s request to confirm or deny whether looped transformers were used for Astra and directed us to Pachocki\u2019s X post<\/a>.<\/p>\n<\/div>\n

\n

\u201cOpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models,\u201d Pachocki wrote, adding that such monitoring \u201cis fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon.\u201d<\/p>\n<\/div>\n

Follow topics and authors<\/strong> from this story to see more like this in your personalized homepage feed and to receive email updates.<\/span><\/p>\n