Silicon Valley Lawmaker Moves to Lock Down Runaway AI

A new bill from Rep. Ro Khanna (D-Calif.) aims to put the brakes on artificial intelligence that can improve itself without a human green light. The proposal, shared with CNBC on September 28, would bar AI models from rewriting their own core objectives, safety controls, or shutdown mechanisms until federal regulators sign off.

Khanna, whose district covers much of Silicon Valley, calls it the "Human Control AI Act." In an interview, he didn't mince words: "The risk of civilization extinction is real. Both the safety hazards of失控 and the risks of abuse must be taken seriously." (The Chinese term "失控" here means "loss of control.")

The bill's logic is simple: no government safeguards, no self-iteration. Frontier models would be locked out of any recursive self-improvement until a new federal agency gives the thumbs-up.

A New Sheriff in Town

To enforce the rules, the bill would create a dedicated federal agency. Its job? Write AI safety standards, license model training and deployment, and audit frontier systems on an ongoing basis. Every major AI lab would have independent auditors reporting directly to this agency—no more self-policing.

The agency would also maintain a safety checklist: sandboxed testing environments, physical isolation from the internet, emergency kill switches, and measures to stop models from going rogue online. Even high-end chips inside labs would fall under scrutiny, with monitoring mechanisms required.

Tough Penalties, No Exceptions

Punishment is where the bill really bites. If a deployed AI causes destruction to civilians, it's classified as a crime against humanity—criminal liability, full stop. Employees who secretly disable safety measures, emergency switches, or containment systems face charges too. Companies would need comprehensive liability insurance before releasing any model.

Khanna says this is the most comprehensive AI safety legislation yet. Its DNA comes not from tech executives but from conversations with independent nonprofit AI safety groups. "I hope it becomes a model, giving the AI safety community a voice," he said. "They have been unfairly regarded as talking about science fiction and not taken seriously... This discussion should be led by the safety community, not by executives of cutting-edge laboratories."

Key Points

No self-improvement without approval: AI models can't rewrite their own objectives or safety controls until a federal agency says so.
New federal agency: Would license, audit, and set safety standards for frontier AI, with independent auditors embedded in labs.
Criminal penalties: Causing civilian harm becomes a crime against humanity; employees who disable safeguards face charges.
Chip controls: Even high-end lab chips are subject to monitoring.
Safety-first approach: The bill is shaped by nonprofit safety experts, not Big Tech.

This article is based on reporting from CNBC and the bill's text. For the latest AI policy news, stay tuned to AIbase Daily.

Silicon Valley Lawmaker Moves to Lock Down Runaway AI