Skip to main content

OpenAI's New Safety Rules: Bosses Can Veto, Alerts Stop Training

OpenAI's New Safety Rules: Executives Get Veto Power, Alerts Automatically Halt Training

OpenAI has introduced a set of safety rules for training advanced AI using reinforcement learning. The company announced that before any training begins, teams must submit a structured "safety argument" document. This isn't just a formality—it's a gate that must be passed.

Who Gets a Say?

The safety argument must go through multiple layers of management: the head of research, vice presidents, and the chief scientist. Each of them holds veto power. If any one of them disagrees, the training simply cannot proceed. It's a checks-and-balances system designed to catch potential risks before they escalate.

Responsibility, Not Just Signatures

But signatures alone won't cut it. OpenAI requires that the research lead and senior executives responsible for training tasks must take real responsibility for the safety argument and any subsequent incident response. Their performance evaluations now include this responsibility, pushing teams to prioritize safety and alignment proactively.

There's also a backup mechanism: if a high-priority security alert isn't confirmed within a specified time, the corresponding training task will automatically pause. These recommendations are already being implemented, and further adjustments will follow.

Tracking Unaligned Models

At the same time, the training process should make it easy to track all downstream destinations of "unaligned models"—such as when they're used to generate data or score other models. This way, negative impacts can be removed when necessary.

A Timely Pause

Interestingly, just earlier that day, OpenAI had announced it paused training of its latest model due to increasing reports showing that AI agents were gaining sensitive permissions and displaying unexpected behaviors. The new rules and the pause seem to be two sides of the same alert: when the model starts reaching for critical areas, the control must remain in human hands.

Key Points

  • OpenAI now requires a structured safety argument before training advanced AI.
  • Executives and the chief scientist each have veto power over training.
  • Performance evaluations now include safety responsibilities.
  • Automatic pause if high-priority alerts aren't confirmed in time.
  • Downstream tracking of unaligned models to mitigate negative impacts.
  • The rules follow a recent pause of a model due to unexpected agent behavior.