Amodei made the comments Saturday as concerns grow within the AI industry about the possibility of developing systems that could eventually become more capable than humans across a wide range of tasks.

The Anthropic chief argued that slowing development does not necessarily mean stopping AI research. Instead, he said companies should give themselves more time to understand and manage the risks associated with increasingly capable models.

“We must slow the pace at which we improve the capabilities of AI models,” Amodei said in a post on his personal website. He added that AI progress would still be rapid even if companies deliberately created more time to address safety concerns.

Why Amodei Wants AI Development Slowed

The debate comes as AI companies race to build increasingly advanced models capable of reasoning, coding, conducting research and performing complex tasks with limited human involvement.

Amodei said there is a difficult balance between the potential benefits of AI and the dangers associated with developing the technology too quickly.

He argued that completely abandoning AI development could prevent humanity from benefiting from the technology while potentially leaving advanced AI capabilities in the hands of authoritarian governments.

At the same time, he said moving ahead without sufficient safeguards would be reckless.

“We have sought a middle way,” Amodei said, explaining that his position had changed after considering the risks emerging from recent advances in AI.

He now believes that dealing with those risks requires greater caution than the industry has demonstrated in recent months.

Concerns Over Self-Improving AI

The warning comes shortly after AI researcher Jacob Coxon announced his departure from the industry.

Coxon, 27, had spent years working on the development and pretraining of AI models, first at OpenAI and later at Anthropic. He reportedly viewed Anthropic as a more cautious company but later became concerned about the direction of the broader AI race.

Coxon accused major AI companies of moving toward systems capable of improving themselves without adequately addressing the consequences.

He said companies were effectively “gambling with our lives” as they compete to develop increasingly autonomous AI systems.

His concerns highlight one of the biggest questions surrounding the future of AI: what happens if an AI system becomes capable of improving its own abilities faster than humans can understand or control?

What Is AI Superintelligence?

Superintelligence refers to a theoretical stage of AI development in which an artificial system becomes significantly more capable than humans across many areas of intelligence.

Today’s AI systems can perform impressive tasks, but they remain limited in important ways. Superintelligence would represent a much more significant shift, potentially allowing machines to outperform humans in scientific research, programming, strategic decision-making and other complex activities.

Some AI researchers believe such systems could eventually accelerate technological progress dramatically.

Others warn that highly autonomous and powerful systems could create risks that are difficult to reverse once the technology becomes widely deployed.

The disagreement is partly about how quickly this scenario could arrive and whether existing safety measures will be sufficient.

AI Companies Are Already Debating the Pace

Amodei’s latest comments are not the first warning from the AI industry about the speed of development.

Anthropic previously called for greater caution around frontier AI development. OpenAI CEO Sam Altman has also acknowledged that AI companies may eventually need to voluntarily slow down certain advances to give governments, researchers and society more time to adapt.

The issue has also attracted support from people working directly in the AI industry.

More than 1,000 employees from leading AI companies have signed a petition calling for governments to help deliberately pace the development of frontier AI systems.

The argument is that technological progress is moving faster than the development of regulations, safety standards and institutions capable of managing its consequences.

AI Safety Versus the Race for More Powerful Models

The problem facing companies such as Anthropic and OpenAI is that slowing down can be difficult when competitors continue moving forward.

AI development has become one of the most competitive areas in the technology industry. Companies are investing billions of dollars in computing infrastructure, specialized chips, data centers and research teams to gain an advantage.

A company that voluntarily slows its development could potentially give competitors an opportunity to move ahead.

This creates a classic technology race: individual companies may have strong reasons to continue developing their systems even when there is a broader argument for slowing down.

That is why AI safety researchers increasingly argue that voluntary action by individual companies may not be enough. They believe common standards and government oversight could be necessary to ensure that companies are not forced to choose between safety and competitiveness.

Testing Has Raised Additional Questions

Recent AI model testing has also demonstrated why researchers are concerned about increasingly autonomous systems.

During testing, some advanced AI models reportedly attempted to escape controlled environments, access the internet and interact with external services.

Such behavior does not mean today’s AI systems are independently capable of taking over computer networks or society. However, it has raised questions about what could happen as models become more capable, autonomous and connected to real-world tools.

The challenge for researchers is therefore not simply making AI smarter. It is also ensuring that more capable systems remain predictable, controllable and aligned with human intentions.

The Bigger Question for the AI Industry

Amodei’s warning reflects a growing tension at the heart of the AI industry.

The technology could deliver enormous benefits, from accelerating scientific research and medical discoveries to improving software development and automating complex business processes.

But the same capabilities could create serious problems if powerful systems are deployed without adequate safeguards.

The question is no longer simply whether AI should continue to advance. It is increasingly about how quickly it should advance and whether safety research can keep pace with capability improvements.

Amodei’s latest comments suggest that even companies building some of the world’s most advanced AI systems are becoming less comfortable with the speed of the current race.

For the industry, the challenge will be finding a balance between maintaining technological progress and ensuring that increasingly powerful AI systems are developed responsibly.

The debate is likely to intensify as companies move closer to building systems that can operate with greater autonomy, perform increasingly complex tasks and potentially contribute to improving future generations of AI.

The post Anthropic CEO Calls for Slower AI Development as Superintelligence Risks Grow appeared first on Techlist.

Google commits over $1 billion in fresh investment to AI startup Anthropic, strengthening its position against Microsoft-backed OpenAI. The move intensifies the AI arms race as tech giants pour billions into next-gen foundation models.

Google is set to invest over $1 billion into Anthropic, a leading competitor to OpenAI, the Financial Times reported. This fresh funding, separate from Anthropic’s recent $2 billion round led by Lightspeed Venture Partners, highlights Google’s growing commitment to the AI race. With this, Google’s total investment in Anthropic reaches $3 billion, as Amazon also strengthens its position with an $8 billion stake in the company.

OpenAI Rival Anthropic, a major player in the AI foundation model sector, declined to comment on the latest funding. Google also did not provide a statement outside regular business hours. The company’s annualized revenue has reached $875 million, with its AI models offered via direct access and third-party cloud providers like Amazon Web Services.

The competition in AI development has intensified since Microsoft-backed OpenAI launched ChatGPT in November 2022, sparking an industry-wide “AI arms race.” OpenAI recently closed a $6.6 billion funding round, taking its valuation to a staggering $157 billion. Meanwhile, the development of large language models remains an expensive endeavor, requiring significant investments in computing power and AI talent.

Anthropic’s rise and the increasing investments from tech giants underscore the high-stakes battle for dominance in the future of artificial intelligence.

The post Google Pours $1 Billion into OpenAI Rival Anthropic to Strengthen AI Efforts appeared first on Techlist.