Skip to main content

OpenAI Calls Out Flawed AI Benchmark: Nearly a Third of Questions Have Issues

OpenAI has taken aim at one of the industry's go-to benchmarks for measuring AI programming prowess, SWE-Bench Pro, alleging that nearly a third of its test questions are flawed. In a blog post, the company argued that the benchmark, which was launched by Scale AI and is widely used to assess large language models and AI agents, has become ineffective at gauging real-world software engineering skills.

The Problem with Rapid Progress

OpenAI highlighted a suspicious trend: the pass rate of cutting-edge models on SWE-Bench Pro jumped from 23.3% to 80.3% in just eight months. That kind of meteoric rise, the company says, is too good to be true. Instead of reflecting genuine improvements in AI capabilities, it likely points to systematic issues within the evaluation itself.

To test this theory, OpenAI ran two parallel reviews. The first, a data-driven analysis, flagged 200 out of 731 public tasks as problematic—about 27.4%. The second, a manual review, identified 249 flawed tasks, or 34.1%. Cross-referencing both methods, OpenAI estimates that roughly 30% of SWE-Bench Pro tasks have defects. These fall into four categories: overly strict testing, insufficient prompts, narrow testing scope, and misleading prompts.

A Case in Point

OpenAI shared a telling example: one task required adding a single space at the start of a line when converting content to Markdown. But the hidden test expected two spaces. So even if a model followed the instructions perfectly, it would still be marked wrong. This kind of mismatch between explicit instructions and hidden requirements artificially inflates failure rates—and, conversely, makes any model that happens to guess right look better than it is.

Call for a New Approach

Based on these findings, OpenAI has officially withdrawn its previous endorsement of SWE-Bench Pro. The company argues that future benchmarks should be designed by experienced software developers specifically for AI evaluation, rather than repurposing testing logic meant for human developers. When a benchmark itself may have a 30% defect rate, the credibility of the entire AI evaluation ecosystem is called into question.

Image

Key Points

  • OpenAI claims about 30% of SWE-Bench Pro tasks have evaluation flaws.
  • Pass rates surged from 23% to 80% in eight months, suggesting benchmark issues.
  • Flaws include overly strict tests, insufficient prompts, narrow scope, and misleading prompts.
  • OpenAI has withdrawn its recommendation of the benchmark.
  • The company calls for new benchmarks designed by experienced developers for AI.