Skip to main content

OpenAI Calls Out Flawed AI Benchmark: Nearly a Third of Questions Have Issues

OpenAI has taken aim at one of the industry's most respected AI evaluation benchmarks, SWE-Bench Pro, arguing that nearly a third of its test questions are flawed. In a blog post released today, the company said that about 30% of the 731 public test tasks suffer from evaluation issues, making the benchmark unreliable for measuring the true programming capabilities of large language models.

SWE-Bench Pro, developed by Scale AI, was designed to test how well AI agents can handle real-world software engineering tasks. It quickly became a go-to standard in the field because of its high similarity to actual enterprise development and strict anti-cheating measures. But OpenAI's analysis suggests that the benchmark's rapid improvement in scores—from 23.3% to 80.3% in just eight months—is too good to be true.

"This kind of progress is suspicious," an OpenAI spokesperson said. "It's more likely that the evaluation itself has systematic problems rather than the models suddenly becoming four times better at coding."

To verify their suspicions, OpenAI ran two parallel reviews. The first, a data-driven analysis, flagged 200 failed tasks—27.4% of the total. The second, a manual annotation effort, identified 249 flawed tasks, or 34.1%. Cross-referencing both methods, OpenAI estimates that roughly 30% of SWE-Bench Pro tasks have defects. These include overly strict testing, insufficient prompts, narrow testing scope, and misleading instructions.

One example OpenAI shared: a task asked the model to add a single space at the beginning of a line when converting content to Markdown. But the hidden test expected two spaces. So even if the model followed the problem statement perfectly, it would be marked wrong. "This kind of hidden requirement that contradicts the explicit instruction directly distorts the assessment of the model's true ability," the blog post explains.

Image

Based on these findings, OpenAI has officially withdrawn its previous recommendation for SWE-Bench Pro. The company argues that future benchmarks should be designed by experienced software developers specifically for AI evaluation, rather than repurposing testing logic meant for human developers.

"When the industry's benchmark itself may have nearly 30% defects, the credibility of the entire AI evaluation system is called into question," the post states. "It's time to move away from score-chasing competitions and focus on real engineering capability assessments."

The criticism comes at a time when AI evaluation standards are under increasing scrutiny. Many researchers worry that benchmarks are becoming outdated or gamed, leading to inflated claims about AI performance. OpenAI's challenge to SWE-Bench Pro could push the industry to rethink how it measures progress in AI software engineering.

Key Points

  • OpenAI claims about 30% of SWE-Bench Pro tasks have evaluation flaws.
  • Pass rates jumped from 23.3% to 80.3% in eight months, signaling benchmark issues.
  • Two review paths found 27.4% and 34.1% of tasks defective, respectively.
  • Issues include overly strict tests, misleading prompts, and hidden requirements.
  • OpenAI withdraws its recommendation and calls for new benchmarks designed by developers.