Anthropic calls for AI pause button to let humans take stock

[NEW YORK] Anthropic called for the creation of a system in which governments and artificial intelligence developers collectively decide when to slow work on the technology to stave off the risks it may pose.

“It would be good for the world to have the option to show or temporarily pause” AI work that may be dangerous, the company said in a lengthy blog post published on Thursday (Jun 4) by co-founder Jack Clark and Anthropic Institute lead Marina Favaro.

AI is advancing to the point where the technology can make human work thousands of times more efficient or even replace it, creating a new set of risks, they said.

Anthropic has acknowledged how difficult it would be to get companies, much less entire countries that have been locked for years in a race to advance their own technologies, to agree to any collective pause. The company has made similar calls for governments and rivals to adopt thresholds.

In 2023, the startup established its own limits and promised to halt work that might be dangerous. But the firm loosened that pledge earlier this year, saying the company would no longer pause if it lacks a significant lead over a competitor.

In walking back that commitment, Anthropic said that the policy environment had shifted towards prioritising AI competitiveness and economic growth as opposed to safety. On Thursday, the company said that it would meet with policymakers and other AI companies in the coming months to talk about better coordination and share the outcome of those discussions.

A protocol for deciding on a collective pause is growing more important as AI may soon be capable of improving itself and building successors, diminishing the role of humans, Clark and Favaro said. “That collision, where recursive intelligence building itself ever faster meets the world of humans, relationships, and governance, is another part of this future we can’t predict,” they wrote.

The blog post compared the idea to the international regulation of nuclear weapons. Any pause mechanism would also need to ensure less scrupulous AI labs don’t secretly keep working to advance the technology during a break, likely through checks by peer labs.

AI researchers have previously urged governments to temporarily halt work. In 2023, for example, the nonprofit Future of Life Institute called for a pause of at least six months to put guardrails on AI, warning about potentially catastrophic effects. Billionaire Elon Musk signed it along with more than 1,000 other researchers and executives. Critics at the time said stopping development would crimp innovation and hand an advantage to others who don’t halt work.

SEE ALSO

As AI reshapes entry-level work, fresh grads must prepare for the ‘third-year job’: panel

Only 15% of workers in Singapore feel their jobs are safe amid AI-driven anxieties: survey

AI unlikely to lead to ‘jobs apocalypse’, OpenAI CEO Altman says

AI needs to ‘work in support of humans’: Josephine Teo

Meta launches enterprise-focused AI ‘business agent’ to automate daily operations

The returnees: Inside China’s AI talent reversal

JD.com founder vows to protect Chinese jobs from AI and robots

More

That’s a challenge that Anthropic itself acknowledged in the blog post: “Training runs are far easier to conceal than missile silos, their inputs are general-purpose, and the incentive to defect quietly is enormous, because whoever continues while others pause could inherit the lead.”

Anthropic has continued to put out advanced models and tools such as its popular assistant Claude, and the new Mythos model that it says can detect and exploit cybersecurity flaws at remarkable speed. The company is also preparing for an initial public offering. BLOOMBERG