SWE-bench Pro: the code benchmark that humbles AI agents

In this article you will see:

• What SWE-bench Pro is and why it is harder than Verified.
• The official numbers: 1,865 tasks, 41 repositories, three subsets.
• The current scoreboard: Muse Spark 1.1 leads with 61.5% on the public set.
• Why performance drops on tasks that require more lines and files.
• The benchmark limits and how to use it in practice.

What is SWE-bench Pro

SWE-bench Pro is a benchmark from Scale Labs, Scale AI, to evaluate AI agents on software engineering tasks. It was created to fix limitations of previous tests: data contamination, short tasks and public repositories already seen in training.

Each task asks the agent to solve a real problem in an existing repository. The agent needs to read code, edit files, run tests and deliver a fix that passes the hidden test suite. Evaluation is binary: the task is solved or not.

The benchmark has 1,865 tasks distributed across 41 professional repositories. The tasks range from bug fixes to feature requests, optimizations, security updates and UI changes.

Three subsets with different purposes

SWE-bench Pro is split into three sets.

The public set has 731 tasks from open source repositories with copyleft licenses, such as GPL. The license works as a legal barrier against contamination: it is unlikely that this code entered proprietary training corpora.

The private set has 276 tasks from 18 proprietary codebases from startups, obtained through partnership. It is the definitive test of generalization on industrial code the model has never seen.

The held out set has 858 tasks from another group of public repositories with copyleft licenses. It stays out of public leaderboards and serves for future analyses and internal evaluations.

Why it is harder than SWE-bench Verified

On SWE-bench Verified, the best models solve more than 70% of tasks. On SWE-bench Pro, the same models drop to the 20% range. GPT-5 and Claude Opus 4.1, for example, score 23.3% and 23.1% on the public set.

The difference comes from the design. Pro tasks require average changes of 107 lines in 4.1 files. Some reach more than 100 lines across several files. Older benchmarks had small, localized fixes.

The copyleft license also reduces contamination. If the code was not in training, the agent cannot rely on memorization.

Current public set scoreboard

The current public scoreboard has 25 models. The leader is Muse Spark 1.1 from Meta, with 61.5%. Next are GPT-5.4, xHigh, with 59.1%, Muse Spark with 55%, Claude Opus 4.6, thinking, with 51.9% and Gemini 3.1 Pro, thinking, with 46.1%.

Older models lag far behind. GPT-4o scores 1.51% and Qwen3 32B scores 3.4%. The gap between generations is large and consistent.

The private set is even tougher. Muse Spark 1.1 leads with 51.5%, followed by Claude Opus 4.6, thinking, with 47.1%. GPT-5 drops from 23.1% on public to 14.9% on private.

What brings models down

Three patterns show up in the official results.

First, performance drops as the solution requires more lines and files. Small changes are solved more often; large changes, almost never.

Second, language matters. Tasks in Go and Python have higher resolution rates, with some models above 30%. In JavaScript and TypeScript, rates vary from almost 0% to over 30%, depending on the model.

Third, the specific repository matters. Some repositories are consistently hard for all models, with rates below 10%. In others, certain models pass 50%. Factors like codebase complexity, problem type and documentation quality affect the outcome.

Top models are also more consistent. Claude Opus 4.1 and GPT-5 not only score higher, but maintain stable performance across languages and repositories. Smaller models have erratic performance.

How to test today

The public dataset and evaluation code are available on GitHub (scaleapi/SWE-bench_Pro-os). The public leaderboard is at labs.scale.com/leaderboard/swe_bench_pro. The private set has a separate leaderboard.

If you evaluate code agents, always compare the same harness (the program that reads, writes and executes code) across models. The benchmark measures the model plus environment pair, not the model in isolation.

For those building agent pipelines, SWE-bench Pro is a good maturity thermometer. An agent that solves less than a quarter of real tasks still needs human review for almost everything.

Benchmark limits

SWE-bench Pro has limits. It measures only whether the fix passes the hidden tests. It does not measure code quality, readability or maintainability.

Evaluation uses the SWE-Agent scaffold. Other harnesses can produce different results for the same model.

The private set is not reproducible outside Scale, because it depends on proprietary codebases. This makes the private leaderboard informative, but not verifiable by third parties.

Results reflect the state of models on the date of each measurement. New models enter the leaderboard over time.