Silicon Valley Lawmaker Moves to Lock Down AI Self-Improvement

What if a machine could rewrite its own code—and no one could stop it? That's the scenario Democratic Rep. Ro Khanna is trying to prevent. On September 28, the congressman from California's Silicon Valley district announced he's preparing to introduce the Human Control AI Act, a bill that would put strict limits on frontier AI models.

The core rule is simple: AI models cannot self-improve or rewrite their own objectives, suppression methods, or shutdown controls until the government sets up a safety framework and a regulatory body gives the green light. Until then, the self-iteration channels for frontier models would be locked shut.

A New Federal AI Watchdog

To enforce these rules, the bill calls for a new federal agency dedicated to AI safety. This agency would develop regulations, run a licensing system for training and deploying models, and audit frontier models to ensure they stay under human control. Every frontier lab would also need independent auditors reporting directly to the agency.

The agency would maintain a safety checklist: sandbox testing environments, physical isolation from the internet, emergency shutdown switches, and measures to prevent models from running loose online. Even high-end chips in labs would be monitored and controlled.

Tough Penalties and Insurance Requirements

The bill doesn't go easy on violators. If a deployed AI model causes destruction to civilians, it would be classified as a crime against humanity, with criminal liability. Employees who disable safeguards, emergency switches, logging systems, or containment measures—or who knowingly deploy unauthorized systems—would also face criminal charges. Companies would need comprehensive liability insurance before releasing any model.

A Voice for the Safety Community

Khanna says this is the most comprehensive AI safety legislation proposed so far. Its approach comes from discussions with independent nonprofit AI safety organizations, not from big tech executives. "I hope it becomes a model, giving the AI safety community a voice," he said. "They have been unfairly regarded as talking about science fiction and not taken seriously... This discussion should be led by the safety community, not by executives of cutting-edge laboratories."

Key Points

No self-improvement without approval: AI models can't rewrite their own code or objectives until a government safety mechanism is in place and approved by a regulator.
New federal agency: A dedicated body would oversee AI safety, licensing, audits, and safety standards.
Strict safety measures: Sandbox testing, internet isolation, emergency shutdowns, and chip monitoring are all on the checklist.
Criminal penalties: Deploying unsafe AI could be a crime against humanity; employees who bypass safeguards face charges.
Insurance mandate: Companies must hold liability insurance before releasing models.
Safety community first: The bill aims to give AI safety experts a leading voice over industry executives.

Silicon Valley Lawmaker Moves to Lock Down AI Self-Improvement