Silicon Valley Lawmaker Moves to Lock Down Runaway AI

What if an AI decided to upgrade itself overnight—and no one could pull the plug?

That's the scenario Democratic Rep. Ro Khanna is trying to prevent. The Silicon Valley congressman is preparing to introduce what he calls the Human Control AI Act, a bill that would put strict limits on frontier AI models. According to CNBC, the legislation would bar AI systems from recursively self-improving or rewriting their own core objectives, suppression methods, or shutdown controls—unless the government first sets up safeguards and a regulatory body gives the thumbs-up.

Khanna isn't mincing words. "The risk of civilization extinction is real," he said in an interview. "Both the safety hazards of loss of control and the risks of abuse must be taken seriously."

A New Federal AI Watchdog

To make sure companies actually follow the rules, the bill would create a new federal agency dedicated to AI safety. This agency would run a licensing system for training and deploying models, audit frontier systems, and set safety standards to keep them under human control. Every frontier lab would also need independent auditors who report directly to the agency—no more self-policing.

The agency would enforce a specific safety checklist:

Sandbox testing environments
Physical isolation of testing environments from the internet
Emergency shutdown switches
Measures to prevent models from running independently online if they escape the lab

Even the high-end chips inside labs would be subject to regulation, with mechanisms to monitor or control their use.

Criminal Charges for Bypassing Safeguards

The penalties are tough. If a deployed AI model causes destruction to a civilian population, it would be classified as a crime against humanity, with criminal liability. Employees who disable protective measures, emergency switches, logging systems, or containment systems—or who knowingly deploy systems without authorization—would also face criminal charges. And companies hoping to release a model would first need comprehensive liability insurance.

Khanna says this is the most comprehensive AI safety legislation proposed so far. Its approach comes from discussions with independent non-profit AI safety organizations, not from big tech executives. "I hope it becomes a model, giving the AI safety community a voice," he said. "They have been unfairly regarded as talking about science fiction and not taken seriously... This discussion should be led by the safety community, not by executives of cutting-edge laboratories."

Key Points

No self-improvement without approval: AI models cannot recursively self-improve or rewrite their core objectives, suppression methods, or shutdown controls until a government safety mechanism is in place and a regulator approves.
New federal agency: A dedicated body would license, audit, and set safety standards for frontier AI models.
Independent auditors: Every frontier lab must have auditors reporting directly to the new agency.
Safety checklist: Sandbox testing, internet isolation, emergency shutdown switches, and controls to prevent online escapes.
Chip regulation: High-end chips in labs would be monitored or controlled.
Criminal liability: Deploying an AI that harms civilians could be a crime against humanity; employees who bypass safeguards face charges.
Insurance requirement: Companies must hold comprehensive liability insurance before releasing models.

Silicon Valley Lawmaker Moves to Lock Down Runaway AI