Silicon Valley Lawmaker Moves to Lock Down AI Self-Improvement

A new bill from Democratic Rep. Ro Khanna, who represents California's Silicon Valley district, aims to put hard limits on how far artificial intelligence can go on its own. Dubbed the Human Control AI Act, the proposal would forbid AI models from performing recursive self-improvement or rewriting their own core objectives, suppression methods, or shutdown controls—unless the government first sets up a safety mechanism and a regulatory body gives its stamp of approval.

In an interview, Khanna didn't mince words: "The risk of civilization extinction is real. Both the safety hazards of loss of control and the risks of abuse must be taken seriously." Under the bill, companies would have to wait for federal rules and regulatory approval before they can let frontier models iterate on their own. Otherwise, those self-improvement channels get locked shut.

A New Agency with Teeth

To make sure the rules stick, the bill calls for a brand-new federal agency dedicated to AI safety. This agency would run a licensing system for training and deploying models, audit frontier systems, and set safety standards to keep them under continuous testing and human control. Every frontier lab would also need independent auditors who report directly to the agency—no more self-policing.

The agency would enforce a specific safety checklist: sandboxed testing environments, physical isolation from the internet during tests, emergency shutdown switches, and measures to prevent models from running rogue online if they escape the lab. Even the high-end chips inside labs would fall under scrutiny, with mechanisms to monitor or control their use.

Tough Penalties, Criminal Liability

The bill doesn't pull punches on punishment. If a deployed AI model causes destruction to a civilian population, it would be classified as a crime against humanity, with criminal liability attached. Employees who secretly disable protective measures, emergency switches, logging systems, or containment systems—or who knowingly deploy systems without authorization—would also face criminal charges. And any company wanting to release a model would first need to carry comprehensive liability insurance.

Khanna calls this the most comprehensive AI safety legislation proposed so far. Its approach comes mainly from discussions with independent nonprofit AI safety organizations, not from the demands of big tech executives. "I hope it becomes a model, giving the AI safety community a voice," he said. "They have been unfairly regarded as talking about science fiction and not taken seriously... This discussion should be led by the safety community, not by executives of cutting-edge laboratories."

Key Points

No self-improvement without approval: AI models cannot rewrite their own objectives or shutdown controls until a government safety mechanism and regulatory body sign off.
New federal agency: A dedicated agency would license, audit, and set safety standards for frontier AI models, with independent auditors reporting directly to it.
Strict safety checklist: Sandboxed testing, physical isolation, emergency shutdown switches, and controls to prevent models from running online after escaping the lab.
Criminal penalties: Deploying an AI that harms civilians could be a crime against humanity; employees who bypass safeguards face charges.
Insurance requirement: Companies must hold comprehensive liability insurance before releasing models.

Silicon Valley Lawmaker Moves to Lock Down AI Self-Improvement