Skip to main content

Anthropic CEO's AI Regulation Idea: Who Watches the Watchmen?

Anthropic CEO's AI Regulation Idea: Who Watches the Watchmen?

Anthropic CEO Dario Amodei has a plan to keep AI safe: put independent safety assessors inside leading AI companies, just like bank regulators. In a recent CNBC report, Amodei proposed giving these embedded assessors access to internal risk teams and allowing them to publish findings with limited restrictions. He promised Anthropic would grant them system access equal to its own risk team.

But Amodei also issued a stark warning. In a lengthy weekend essay, he predicted that within 6 to 12 months, a misaligned group of AI agents could take over most of the internet, causing hundreds of billions in losses. His warning echoes concerns from former Anthropic researcher Jacob Coxon, who resigned and cautioned that top labs are racing to build potentially uncontrollable systems.

Bank Examiners Can Shut Banks Down. AI Assessors? Not So Much.

Several banking regulation experts and AI assessment professionals say Amodei's proposal falls short. The key difference: bank regulators have real teeth. Julie Andersen Hill, Dean of the University of Wyoming Law School, explained that government examiners stationed at large banks can order them to stop activities, limit growth, force management changes, or even shut down the bank in extreme cases.

Amodei's assessors, by contrast, can only investigate and report. They have no power to halt model training or release. "This is a fundamental difference," Hill said, "because the power of bank regulators is much greater."

Albert Ziegler, AI lead at cybersecurity firm XBOW, added that black-box testing can reveal whether a model can perform dangerous tasks, but major risks may only surface in condition combinations assessors have never triggered. "We really don't have veto power," he said. Assessors can force companies to make informed decisions before releasing, but the final call stays with the company.

The Independence Question: METR and 'Audit Washing'

Amodei named the nonprofit METR as a potential embedded assessor. Another former Anthropic researcher, Joe Benton, recently left to join METR. But Deborah Raji, a researcher at UC Berkeley, sees conflicts of financial, ideological, and personal interest. She noted that METR has mainly focused on Anthropic and OpenAI in recent years, "allowing all sorts of strange things to happen."

Raji argues that merely gaining access does not equal independence. An institution independent from the company should decide who is qualified to assess, what can be checked, and where results are reported. Otherwise, "it's like the company hiring a regular friend to check its homework."

She said the ultimate standard is whether adverse findings lead to consequences. "If you do an audit and nothing happens, it's audit washing." Hill agreed: "If you truly believe AI has the power to destroy society, then you must have an independent supervisor capable of pulling the plug."

Anthropic's Response: We Can't Check Our Own Homework

Sarah Heck, public policy director at Anthropic, responded on Wednesday that AI companies cannot manage oversight and safety through "honorable codes." "We can't check our own homework," she said, "and that's very clear."

So the debate boils down to a simple question: can AI companies be trusted to regulate themselves? Amodei's proposal is a step toward external oversight, but without enforcement power, critics say it may just be window dressing. As AI grows more powerful, the pressure for real regulation—with real teeth—will only intensify.

Key Points:

  • Anthropic CEO Dario Amodei proposes embedding independent safety assessors in AI companies, similar to bank regulators.
  • Experts note these assessors would lack enforcement power—unlike bank regulators who can shut banks down.
  • Concerns about independence and "audit washing" arise, as assessors may have conflicts of interest.
  • Anthropic acknowledges the need for external oversight, but critics say real consequences are essential.
  • The debate highlights the challenge of regulating fast-moving AI development.