OpenAI, Anthropic researchers ramp up calls for AI slowdown as warnings of catastrophic risk intensify

OpenAI and Anthropic researchers are ramping up calls for an AI slowdown and warning of existential risks to humanity after the at Anthropic fueled fresh scrutiny.

The concerns started after Anthropic researcher Jacob Coxon said Tuesday that he was quitting the company as he accused Anthropic and rival OpenAI of "gambling with our lives. " He added that those building AI believed that it could "kill us all by the end of the decade. " Evan Hubinger, Anthropic's alignment lead, responded that he expects there is a more than 10% chance of that happening.

Since then, several employees at both AI labs have come out in support of calls to slow the pace of AI development as they stressed the risks of the technology. The public warnings are the culmination of growing concern globally about the capability of AI, following numerous cyberattacks and security incidents in recent months by rogue models developed by both OpenAI and Anthropic.

"In my personal capacity, I also think we need to slow down," Julie Steele, a member of OpenAI's technical staff who works on the safety team, said late on Wednesday in a post on X in response to Coxon's warnings.

Anthropic researcher Samuel Marks said that "AI developers believe their technology could cause human extinction (or similarly bad outcomes)," in an X post on Wednesday. "This could happen in the next few years. In general, the more senior the employee, the more concerned they are. "

Anthropic was the first lab to publish a framework dedicated to mitigating "catastrophic risks from AI models," a spokesperson told CNBC when asked about the comments from employees on social media.

"We have always been transparent that AI will bring both enormous benefits and unprecedented risks," an Anthropic spokesperson said, adding that the company was building models with "some of the strongest safeguards in the industry. "

OpenAI declined to comment when approached by CNBC, noting recent blog posts on its site.

Many of the biggest AI safety fears revolve around advanced models getting increasingly capable at improving their own performance, a technique known as recursive self-improvement, or RSI.

"It's hard to overstate how dangerous speeding towards RSI is,"

"If AI development continues along its current path, the systems we'll see in the next few years are likely to represent further capability jumps of equal or larger magnitude, and to increasingly drive their own development," he said in a company blog post.

"This is a time that calls for extreme caution," Pachocki added. "I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence. "

Paul Christiano, who was formerly head of safety at the U.S. Commerce Department's Center for AI Standards and Innovation (CAISI),

AI safety warnings reach Washington

In July, OpenAI said its models were incident on another company, while Anthropic's Claude models were also responsible for including in one case where Mythos to fool humans.

Roughly 1,400 AI researchers, from companies including OpenAI, Anthropic, and DeepMind, published an open letter in July urging the U.S. government to develop the tools necessary to support an effort to "deliberately pace the frontier of automated AI development. "

While chiefs of AI labs have increasingly publicly around the development of models, huge competition between companies developing the tech is spurring rapid advances.

OpenAI and Anthropic are both racing towards public listings. Anthropic is expected to begin marketing its initial public offering in mid-October at the earliest and complete the listing days before the U.S. midterm elections in November, Reuters on Friday, citing people familiar with the matter.

Former AI czar of U.S. President Donald Trump, David Sacks appeared to suggest Anthropic's plans for an IPO should be halted. "Surely Anthropic's IPO must be paused until the claims of this "whistleblower" can be investigated," he