OpenAI Retracts Recommendation for SWE-Bench Pro Coding Benchmark

OpenAI announced it no longer recommends SWE-Bench Pro as a reliable measure of AI coding ability. The decision follows an internal audit which discovered that nearly one-third of the benchmark's tasks contain significant design flaws, leading the company to retract its previous endorsement for the research community.

The audit examined 731 tasks from the benchmark's public dataset. Using a combination of AI-based analysis and review from five experienced software engineers, OpenAI's investigation flagged 27.4% of the tasks as broken, while human reviewers identified issues in 34.1%. The company noted that top AI models' performance on the test had rapidly increased from a 23.3% to an 80.3% pass rate in eight months, suggesting the benchmark had reached a saturation point near a "70% noise ceiling."

According to OpenAI, the identified problems fall into four main categories. These include overly strict tests that reject functionally correct solutions, underspecified prompts that omit key requirements enforced by hidden tests, low-coverage tests that allow incomplete code to pass, and misleading prompts that guide models toward incorrect solutions.

Ad

The company stated that these evaluation flaws can create a misunderstanding of an AI model's true capabilities, noting that correct solutions sometimes fail due to hidden requirements or incomplete grading criteria. OpenAI expressed concern that such misrepresentations could affect safety assessments and research priorities under its Preparedness Framework.

SWE-Bench Pro tasks are sourced programmatically from code repositories, where problem descriptions and tests do not always align perfectly for evaluation purposes. OpenAI had previously recommended SWE-Bench Pro as an alternative to another benchmark, SWE-bench Verified, after also finding it to have fundamental design issues.