Co-Design Bench: code agents get 26% of real world tasks right

In this article you will see:

• What the Co-Design Bench is and why it uses real user requests.
• The official numbers: 10 models, 26.4% total completion, 70% of requirements met.
• Why tasks with multiple types of change bring down all models.
• What the harness effect changes for those choosing code agents.
• The limits of the benchmark and how it compares to METR and SWE-bench.

What is the Co-Design Bench

The Co-Design Bench is a new test for code agents. Instead of synthetic tasks, agents need to rework existing web applications from real user requests. Requests can include images, audio recordings, videos or documents as input.

Researchers preferred cases where users had to reformulate the request several times or revert changes. This ensures the test contains mainly tasks that already gave work to agents in practice. Tasks range from small adjustments, like extracting a value from a receipt and updating all dependent totals, to complete project redesigns.

Evaluation is rigorous. An AI judge tests the final application in the browser. A second judge rechecks each positive verdict and looks for mockups or hardcoded results. Points are given only for features that actually work in the application.

Why old benchmarks no longer suffice

The authors justify the new test by pointing to the saturation of existing benchmarks. When models can execute, verify and fix their own code, they reach scores above 99% on these tests, and differences between models can barely be measured.

In old benchmarks like WebCompass-edit, code agents reach almost perfect scores, while the Co-Design Bench leaves a gap of more than 35 percentage points. The Co-Design Bench uses real requests from real coding sessions, which makes the test closer to the day to day of those who use these tools.

The official Co-Design Bench numbers

GPT-6 Astra, the best of the ten models tested, meets about 70% of the requirements of a task on average. But the result is fully correct in only 26.4% of tasks. In most cases the agent does the bulk of the work but leaves at least one detail unfinished, so those who use these tools almost always need to clean up by hand.

Behind GPT-6 Astra, Claude Opus 5 and Qwen3.8-Max occupy the next positions, while Grok 4.6 finishes last. In larger redesigns, Qwen3.8-27B, much smaller, rises to second, but the authors do not derive a general advantage for the model from this result.

Multiple types of change bring down all models

Performance drops in all models when a task combines more types of change: layout, design, interaction and content. But these intertwined requirements are the norm in real projects.

Each of the ten models scores worse when the task requires three or four different types of change. Extra compute helps little: GLM-5.2 consumes almost six times more tokens than GPT-6 Astra on small changes and still scores significantly worse.

GPT-6 Astra leads both tracks but fully solves only 41% of multimodal tasks and a little less than 18% of repository tasks.

The harness matters as much as the model

In a separate test, researchers ran the same models on different harnesses, the programs through which an agent reads, writes and executes code.

Qwen3.8-27B scored 75 or just 56 percent on major rewrites depending on the environment. Which of the two models came out ahead also depended on the environment.

Leaderboards that list only the model tell only part of the story. The authors recommend always reporting model and work environment together. For companies choosing a code agent, this means testing both components as a pair, not separately.

Limits of the benchmark

The benchmark has limits. It evaluates a fixed request per task; follow-up questions or mid-task corrections are not part of the test. AI judges can also be wrong.

A study by research organization METR already showed how much benchmark results and real-world usability can diverge.

The effect of the work environment is not clear-cut. A recent study on code agents found little measurable benefit from more elaborate harnesses, even while consuming up to three times more tokens.

How to test today

If you evaluate code agents, Co-Design Bench is a good starting point to understand current limitations. Access the original article on arXiv (2610.35138) to see the full details, including the score charts by change type and the task examples.

Frequently asked questions about Co-Design Bench

What problem does Co-Design Bench try to solve? Old code agent benchmarks are saturated: models reach over 99% and differences between them can barely be measured. Co-Design Bench uses real user requests to measure performance in real situations.

How many models were tested? Ten models, including GPT-6 Astra, Claude Opus 5, Qwen3.8-Max, Qwen3.8-27B and Grok 4.6, on tasks involving layout, design, interaction and content.

What was the best result? GPT-6 Astra meets about 70% of a task's requirements on average, but fully completes only 26.4% of tasks.

Why are tasks with multiple change types harder? Because they combine layout, design, interaction and content, which is what happens in real projects. All ten models score worse when the task requires three or four types of change.

Does the harness really matter? Yes. The same model scored 75 or 56 percent on major rewrites depending on the environment. Leaderboards that list only the model tell only part of the story.

Do high scores on old benchmarks mean agents solve real tasks? No. Co-Design Bench leaves a gap of more than 35 percentage points compared to old benchmarks, and the real code acceptance rate by project leads was about 24 points below SWE-bench Verified.