Why We No Longer Evaluate on SWE-bench Verified

Scale AI has stopped evaluating models on SWE-bench Verified, arguing the benchmark is increasingly contaminated and no longer measures frontier coding progress.

The company's analysis identified two core problems. First, the benchmark's tests contain flaws. Second, training data leakage means models may have effectively seen the evaluation tasks before being tested on them — a failure mode that inflates scores without reflecting genuine capability.

The stakes are considerable. SWE-bench Verified has become one of the most widely cited measures of AI coding ability, used to compare frontier models from major labs. If its results are compromised by contamination, leaderboard positions that shape buyer decisions and research priorities rest on unreliable numbers.

Scale AI recommends SWE-bench Pro as the replacement. The newer benchmark is designed to resist the leakage that undermines its predecessor while testing harder, more realistic software engineering tasks.

The move signals a broader industry problem: as models train on ever-larger corpora, public benchmarks degrade as measurement instruments. Organizations that rely on them to track progress — or to market it — will need to migrate to harder, better-isolated evaluations, or accept that headline scores say less than they used to.