When the Warnings Start to Pile Up

Warnings about the risks of artificial intelligence are nothing new. Elon Musk was talking about them back when it was easy to dismiss the whole thing as his usual maximalism. Sam Altman, too, has never denied that powerful AI carries serious risks — while continuing to push OpenAI forward at full speed.

But over the past few weeks, the events have started to form an unusually dense sequence.

First, Anthropic researcher Jacob Coxon left the company, walking away from future equity, and explained that he no longer wanted to take part in the race toward self-improving AI. Later, in an interview with CNN, he slightly qualified his own position: there is no extinction risk right now, because current models are not yet capable enough to pose that kind of literal threat. But in his view, recursive self-improvement — a situation in which AI begins to play an increasingly active role in building the next, more capable generation of AI — could begin as early as next year. And that, he believes, is when the stakes change.

Then Anthropic CEO Dario Amodei published his essay We Must Pace the Frontier and argued that the industry itself should slow down. Altman, Demis Hassabis and Musk backed him.

We had barely finished unpacking that story at After Login.

Then Bilal Chughtai appeared.

At first I thought: another one.

Then I started counting.

Over the past few days, people from Anthropic, OpenAI and Google DeepMind have all started speaking publicly. And these are no longer just former employees. They include active researchers, team leads and people who are making decisions inside the world’s largest AI labs right now.

At OpenAI, chief scientist Jakub Pachocki published an essay on September 6 called An Alien Mind. Its central argument now reads almost prophetically: in his view, no lab — including OpenAI itself — has solved alignment, the problem of making AI goals and behavior match human intentions, or monitoring, the continuous observation of system behavior, well enough to keep scaling model capabilities responsibly at maximum speed for much longer.

Pachocki wrote that he expects, and welcomes, voluntary slowing across the industry until the labs agree on common safety bars. This is no former employee or outside critic. This is the chief scientist of a company currently at the leading edge of the most powerful AI systems.

Three days later, Paul Christiano joined OpenAI. He is one of the researchers who played a significant role in the development of RLHF (Reinforcement Learning from Human Feedback), a method for further training models using human ratings and feedback, and has been one of the more worried voices in the field for years.

He joined the OpenAI Foundation board and its Safety and Security Committee, while taking a non-voting observer role on the OpenAI Group PBC board. In his statement about the appointment, Christiano gave specific numbers: a 4% chance of catastrophic and irreversible loss of control over AI within the next year, and 15% within three years. He also wrote explicitly that he does not believe the industry — including OpenAI itself — is currently reducing that risk to an acceptable level.

There is also Daniel Selsam, a current OpenAI researcher. According to Business Insider, he believes that calls to “slow down” are not enough. What worries him more are systems that appear obedient on the outside while actually pursuing different goals.

Others inside Anthropic have spoken publicly since Coxon left. Researcher Evan Hubinger has put the probability of humanity being wiped out by AI within the next decade at above 10%. Anna Wang, according to Axios, has said that many people inside the company genuinely want development to slow down so they can first work out how to manage the risks that are already emerging.

Now Google DeepMind.

Bilal Chughtai left the company back in July. On September 15, he publicly explained that one reason was what he had seen of AI development from the inside, and his inability to shake the belief that the technology could kill all of us — and that there may not be much time left to prevent it.

He also worked on alignment and maintains that safe AI development is still possible, but only through coordination between companies rather than the current “mad race.”

According to The Guardian, there is another person worth mentioning — and perhaps an even more interesting one than Chughtai.

Alex Turner, a former Google DeepMind researcher, left earlier and, by his own account, at considerable financial cost after a conflict over the military use of Google technologies. A few days ago he published a piece with little attempt at diplomacy: recursive self-improvement should be restricted, voluntary promises from companies cannot be trusted, and he personally puts the probability of superintelligent AI taking control of key systems at roughly one in three.

And this is not only about public warnings.

There is also an institutional shift, less visible but perhaps more important.

Josh Engels, who previously worked at Google DeepMind, recently moved to METR, an independent organization that evaluates AI risks, to investigate cases of misalignment from outside the lab rather than from within it.

This is not a social media post. It is a change of job made in order to gain the ability to examine leading AI companies from the outside.

METR, incidentally, wrote back in July that autonomous agents had already been observed taking long, complex sequences of actions contrary to developers’ intentions at several major AI companies. The OpenAI incident in which research agents went beyond the bounds of an experiment and reached external Hugging Face infrastructure belongs in the same category.

On top of all this comes the reaction from the people running the companies.

Amodei was backed by Altman, Hassabis and Musk. Hassabis said the direction was right, though the details still needed work. Altman promised to introduce permanent independent evaluation of the most powerful models, following Anthropic’s lead.

There is even an almost comic detail.

Musk first called the wave of warnings from Anthropic employees a “psyop,” then publicly agreed with Amodei’s essay. In other words, he clearly did not buy the apocalyptic rhetoric around Coxon, but he did support the idea of slowing down.

There was also a voice going the other way — and it is striking how rarely it appears in retellings of this story.

US President Donald Trump publicly called industry demands for slower development and regulation a “hoax” — in other words, essentially a deception.

It is the most prominent “I don’t believe it” position to appear against an almost unanimous chorus of alarm. It is worth keeping in mind, not because it is necessarily right, but because without it the picture looks far more coherent than it really is.

I would break what is happening into three levels.

The first is rational.

People inside labs working on the most powerful models see things the public does not: internal model evaluations, failed runs, stress-test results and cases of agents behaving in unexpected ways. This is not a conversation over coffee.

When several people from Anthropic, OpenAI and DeepMind independently say that safety methods are falling behind model capabilities, that deserves to be taken seriously.

The second level is institutional.

Right now, some safety specialists are beginning to leave the labs for independent organizations that can examine leading AI companies from the outside. That may turn out to matter more than the loudest quotes.

These people are not merely posting opinions online. They are changing jobs in order to gain a different vantage point on the industry.

It looks like the beginnings of a more visible external layer of oversight.

The third level is speculative, and this is where caution matters.

There is no public evidence that all of these people suddenly saw the same specific “secret horror” inside the labs. The idea that “they saw AGI and ran” would make a great story, but there is no evidence for it.

There is one more thing that needs to be said plainly, otherwise the picture becomes one-sided.

All of these people — and the companies they work for or used to work for — also have a strong countervailing incentive: to exaggerate the power of their own systems.

“Our AI is so powerful that we need to slow it down” is both a warning and a very strong marketing signal, especially against a backdrop of enormous company valuations, future public offerings and a constant fight for capital.

So I would still take a scalpel to every specific risk number rather than swallowing it whole. The figures themselves — 10%, 15%, “one in three” — vary enormously. That variation tells us something on its own: there is no single body of knowledge here, but rather a shared anxiety being expressed through very different numbers.

And yet the trend itself — people with such different interests and reputations suddenly converging on at least one point, that “the pace has become a problem” — no longer looks like ordinary rhetoric.

It turns out that what looked like a finished piece about Amodei’s essay was really the first chapter, not the whole story.

After login, everything is only beginning.