SWE-bench is a benchmark for evaluating AI models and agents on real software engineering. Instead of made-up problems, it uses real issues from popular open-source repositories written in Python.
How it works
Each task has an issue, which describes a bug or improvement, and the project code in the state before the fix. The agent needs to read the repository, understand the problem, and generate a patch.
Then the project tests run with the patch applied. If the tests that previously failed now pass and the ones that worked continue working, the task counts as resolved.
The result is usually the percentage of tasks resolved. Variations exist, such as human-reviewed subsets to ensure tasks are fair.
Because the result depends on both the model and the agent around it, the score blends the two. Code search tools, test execution, and number of attempts change the scoreboard.
Practical example
An issue says a date function returns the wrong month in leap years. The agent opens the repository, locates the function, writes the fix, and runs the tests.
pytest tests/test_datas.pyIf everything passes and nothing else breaks, that task scores a point. Repeated across hundreds of tasks, this produces the final score.
What it's for
SWE-bench serves to compare code agents in a scenario closer to daily work. When a company announces a new model, it's common to cite its score on this test.
For those choosing tools, it indicates which models handle existing projects better. But the score doesn't guarantee the same result on your code, so test with your own tasks.
When comparing scores, check which variant was used, which agent ran, and the date, because rankings change fast.
Difference between SWE-bench and HumanEval
HumanEval asks for a small, isolated function. SWE-bench asks for a fix inside a real project, with multiple files and existing tests. The second is more demanding and more like a dev's day-to-day.