Understanding the Importance of Accurate Model Evaluation
Accurately measuring the capabilities of our models is crucial for informed deployment and security decisions, including those made by OpenAI. The Readiness framework guides these decisions, ensuring that each model release includes results from various external and internal benchmarks to track progress effectively. However, errors in assessments can lead to misunderstandings about capabilities, misrepresent safety evidence, and skew research priorities.
Challenges with Coding Benchmarks
Recently, we scrutinized one of the most widely used coding benchmarks, SWE-bench Verified, discovering fundamental design and contamination issues. These problems rendered the assessment ineffective in providing meaningful signals about software development capabilities. Consequently, we encouraged a shift to SWE-Bench Pro.
The Evolution to SWE-Bench Pro
SWE-Bench Pro was designed as an improvement over SWE-Bench Verified. It tests models over extended periods and more realistic coding tasks to better evaluate coding capabilities. Tasks are sourced programmatically from feature changes in public and private repositories. Models must implement solutions that pass new tests without affecting existing functionality. In a public split with 731 tasks, boundary models improved from a success rate of 23.3% to 80.3% within eight months.
Audit and Findings
We conducted a similar audit on SWE-Bench Pro using a data point analysis pipeline. This pipeline reviewed model experiments, task metadata, and error traces to flag likely scoring errors. Each flagged task underwent multiple investigator-agent runs and was independently reviewed by five senior software engineers, with disagreements escalated for further examination.
Our findings revealed issues in a significant portion of the dataset. The data point analysis pipeline flagged 200 (27.4%) erroneous tasks, while a human annotation campaign identified 249 (34.1%). The problems fell mainly into four categories:
- Too strict tests: Force certain implementation details not specified in the prompt, invalidating many functionally correct submissions.
- Underspecified prompts: Omit requirements enforced by hidden tests, which cannot be reasonably derived.
- Low coverage testing: Review the requested feature so that incomplete fixes can be made.
- Misleading requests: Indicate incorrect behavior to models or contradict test requirements.
Quality Assurance and Human Review
Our results underscore the challenge of creating hard yet fair benchmarks and highlight the growing utility of agents for scalable data quality checks. Given these results, we estimate that approximately 30% of SWE-Bench Pro tasks are defective and recommend that model developers thoroughly review the results.
The Human Annotation Campaign
In parallel, we ran a human annotation campaign on the labeled subset. We collaborated with experienced software developers trained on the benchmark objectives, problem taxonomy, and edge cases before reviewing the tasks. Each task was reviewed by five engineers, forming independent judgments based on the visible problem statement, test cases, and ground truth reference solution, before using the pipeline analysis or transcript as supporting context. Reviewers then assigned a designation and severity based on concrete evidence and escalated disagreements or low-confidence cases for further review.
Key Findings and Recommendations
Human reviewers marked tasks as defective more often than investigators. There was some disagreement about the categories between the two review paths, but none of the labeled tasks used the most common human label “not broken.” For the categories identified by the agent pipeline, assessors’ judgments overlapped in 74% of cases.
Compared to the agent pipeline, the human reviewers were more likely to select multiple labels for a task, suggesting that tasks were flawed in various ways or did not clearly fit into a single category. This indicates that the agent-plus-reviewer pipeline resulted in conservative labeling, capturing the same broad failure modes identified by humans, while undercounting cases where reviewers saw additional or overlapping issues.
Conclusion
The challenges and findings from SWE Bench Pro underline the necessity of rigorous benchmark reviews. Open source repositories, initially created for human collaboration, often involve intricate interactions between maintainers and contributors. Problem statements, merged code, and unit tests may not always translate into clean, isolated tasks for reliable model evaluation. Therefore, benchmarks should provide meaningful signals, be difficult to manipulate, easy to trust, and accurately reflect model capabilities or alignment. As these evaluations inform OpenAI’s deployment and security decisions, they must be valid and informative.
For further details on this analysis, visit the full report here.
“`
