SWE-Bench Pro Explained: Scores and Known Flaws
SWE-Bench Pro is Scale AI's coding-agent benchmark. Scale built it to be harder than SWE-bench Verified and less exposed to training data. When it launched in September 2025, the best model solved 23.3% of its public tasks. By October 2026 the top public score had passed 60%. Audits published in 2026 also found that a large share of its tasks grade answers incorrectly. If you use SWE-Bench Pro numbers to pick a coding agent, you need to know what it measures and which results to discount.
What is SWE-Bench Pro, in one sentence?
SWE-Bench Pro is a set of 1,865 human-verified coding tasks from 41 maintained repositories, where an agent must write a patch that makes failing tests pass.
That definition comes from the paper. It was submitted on September 21, 2025 and revised on November 14, 2025. Of the 41 repositories, 11 are public and 12 are held out. The other 18 are commercial. Of the tasks, 731 form the public set. Another 276 come from private startup codebases, and Scale publishes only results for them.
The tasks are bigger than a typical bug fix. The paper says reference solutions average 107.4 lines of code across 4.1 files. Human reviewers checked every task and added context to it. The paper says each task has enough detail to be resolvable, with a problem statement, requirements and an interface.
How does SWE-Bench Pro differ from SWE-bench Verified?
SWE-bench Verified is a human-validated subset of 500 SWE-bench tasks, created with OpenAI. Pro was designed to be harder and to resist contamination, meaning tasks that models may have seen in training.
| SWE-bench Verified | SWE-Bench Pro | |
|---|---|---|
| Tasks | 500 | 1,865 (731 public) |
| Source repositories | Projects from the original SWE-bench | 41 repositories, public ones under GPL-style copyleft licenses |
| Private tasks | None | 276 commercial tasks from startup codebases |
| Typical patch size | Not stated on the Verified page | 107.4 lines across 4.1 files on average |
| Top score at Pro's launch | Over 70% for frontier models | 23.3% public (GPT-5) |
Sources: the SWE-Bench Pro paper and its first version. The Verified top score is as the Pro paper reports it.
The contamination defense is the licensing. Scale picked public repositories under strong copyleft licenses, betting that model trainers avoid GPL code. The commercial set goes further: its code has never been public.
What are the top SWE-Bench Pro scores today?
When we checked the public leaderboard on October 28, 2026, the top five were:
| Rank | Model | Resolve rate |
|---|---|---|
| 1 | Muse Spark 1.1 | 61.50% ± 3.10 |
| 2 | gpt-5.4 (xHigh) | 59.10% ± 3.56 |
| 3 | Muse Spark | 55.00% ± 3.60 |
| 4 | claude-opus-4-6 (thinking) | 51.90% ± 3.61 |
| 5 | gemini-3.1-pro (thinking) | 46.10% ± 3.60 |
Model names are Scale's. All five ran in the mini-swe-agent harness. The page says listed models get a 250-turn limit and uncapped cost, except grayed-out entries. The confidence intervals are around 3.5 points, so a gap of two or three points between neighbors is noise.
Part of the rise came from looser run settings. The first paper version reported GPT-5 at 23.3% on the public set. The revised paper reports GPT-5 (high) at 41.8% under longer runs. In its capped run (50 turns, $2 per task), GPT-5 (high) scored 25.9%.
Why do coding agents fail SWE-Bench Pro tasks?
The paper used GPT-5 as a judge to classify failed runs. For Claude Opus 4.1, wrong solutions made up 35.9% of failures and syntax errors 24.2%. The authors say GPT-5 made fewer wrong solutions but may differ in how well it used tools.
Our reading is that these tasks punish missing context. A 4-file change in an unfamiliar repository needs the agent to find conventions and interfaces before it writes code. Every task starts the agent cold, with no notes from earlier work in that codebase. That is fair for ranking models. A team's agent works in the same repository for months.
What did audits find wrong with SWE-Bench Pro?
Several groups audited the benchmark in 2026. Epoch AI collected them in a benchmark review and rated SWE-Bench Pro "Flawed":
- Jonathan Gabor reviewed 100 random problems on February 24, 2026 and found issues in 83. The most common was requirements that the tests never checked.
- Datacurve's LLM-assisted audit on May 26, 2026 estimated 24% false negatives and 8.5% false positives.
- OpenAI estimated on July 8, 2026 that 30% of tasks were broken, after five experienced engineers reviewed flagged tasks.
Epoch concludes that more than 20% of questions very likely have scoring defects. It did not review contamination.
A second problem is leakage. On April 29, 2026, an evaluator at Poolside opened a GitHub issue on the public repository. The issue reports that the Docker images keep future commits in branches and tags, and those commits can contain the fix. An agent can find the answer with a git command. The reporter found some runs that did this, though not many.
Scale's answer is SWE-Bench Pro V2, released September 22, 2026. It has 642 validated tasks across 11 public repositories, plus a 51-task hard subset. Its images ship with sanitized history that drops the fixing commit and stray refs. The locked protocol runs the agent offline with a 50-minute budget per task. It then re-grades the agent's diff on a fresh image. Scores from the original 731-task set and from V2 are different tests, so do not mix them.
Read the fine print
Scale's leaderboard is a vendor-run benchmark, and model makers report their own runs with their own harnesses. The audits above come from other labs and researchers, and some used LLMs to grade. Treat the exact error rates as estimates. The direction is consistent across all three audits: a meaningful share of tasks grade correct patches as failures or accept wrong ones.
The bottom line
Use SWE-Bench Pro to compare models run in the same harness with the same turn and cost limits, and ignore gaps smaller than the confidence interval. Do not treat a score as the share of your own tickets an agent will close. Test candidate agents on tasks from your own repositories, where you know which answers are right.
How do you stop a coding agent from starting cold in your repo?
Benchmark tasks are cold starts by design. In your repository, the same agent returns every day. It should not have to rediscover the build command or the interface you agreed on last week. Without memory, every session pays that discovery cost again, and it pays it in each tool you use.
memU records what agents learn as a shared wiki of readable Markdown. A scheduled background task slices new session logs into jobs. The agent decides what is worth keeping and writes memory and skill files. memU itself makes no LLM calls. It stores and retrieves the Markdown the agent prepared. A standing instruction in CLAUDE.md or AGENTS.md tells the agent to retrieve before answering. All hosts share one backend, so a fix Codex found is there for Claude Code next time. We have not measured memU on SWE-Bench Pro, and memory would not help on a benchmark that resets every task.
To try the hosted version, which is free and syncs across devices, get an API key from memu.so and send your agent one message:
Read https://memu.pro/SKILL.md, follow its instructions to install and configure memU, API Key is
YOUR_MEMU_API_KEY.
Self-hosting instructions are in the memU GitHub repository.
FAQ
Who made SWE-Bench Pro?
Scale AI. The paper was submitted to arXiv on September 21, 2025, and Scale runs the public and commercial leaderboards.
Is SWE-Bench Pro harder than SWE-bench Verified?
Yes, by design. Its tasks average 107.4 changed lines across 4.1 files, and the top launch score was 23.3% against over 70% on Verified.
What is the highest SWE-Bench Pro score?
On October 28, 2026, Scale's public leaderboard listed Muse Spark 1.1 first at 61.50%, followed by gpt-5.4 (xHigh) at 59.10%. The leaderboard changes often, so check the live page.
Is SWE-Bench Pro reliable?
Partly. Epoch AI rates it "Flawed" because audits estimate that more than 20% of tasks have scoring defects, and an open issue reports that git history can leak fixes. Use it for rough comparisons only.
Can I run SWE-Bench Pro myself?
Yes, for the public set. Scale publishes the open-source tasks and Docker images on GitHub. The commercial set is private, and only Scale's results for it are published.