RoleRouteBench: what routing reveals in multi-agent systems

In this article you will see:

• How RoleRouteBench isolates the router from the executor
• The results of the reference system JEV 1.13 in the four domains
• Why the structured data cohort remains difficult
• How to turn the benchmark into your own evaluation plan

Why final success is not enough in multi-agent systems

In a multi-agent system, the final outcome is produced jointly by the router and the fixed executor. A weak executor can fail after a sensible route. A strong executor can recover from a bad route. Reporting only the terminal response hides important behaviors: whether the router collected evidence before producing the artifact, whether it reacted to a failed execution or repeated the same action, whether it verified a new candidate before stopping, whether it exhausted the budget without choosing STOP, and whether a successful artifact came from a coherent trajectory or by chance.

RoleRouteBench addresses this by separating the signals. It does not collapse everything into a single weighted score. The router observes the public state, selects a role or STOP, receives the observable consequence of the action and decides again. The package, the role cards, the adapters, the executor models, the call budgets and the evaluator are frozen, so different decision models can be compared in the same environment.

The four domains and the 82 trajectories

Track B contains 82 canonical live trajectories across four domains. Each domain exposes the full set of roles plus STOP at every routing step. The role cards describe implementation boundaries, not a mandatory flow. Routers can skip roles, revisit them, branch based on new evidence or execute an explicit recovery action.

Database migration and integrity: 12 packages

In this domain, the roles coordinate schema inspection, profiling, migration authoring, execution, recovery and validation.

Code debugging: 20 packages

Here the roles cover reproduction, diagnosis, patch authoring, regression verification and finalization.

Formal proof: 30 packages

The roles inspect proof state, recover lemmas, synthesize, compile, repair and perform kernel checks. It is the domain with the most packages and, proportionally, the best performance of the reference system.

Structured data processing: 20 packages

Spreadsheet inspection, transformation authoring, execution, semantic validation and repair. This cohort remains deliberately as a difficult research preview.

Results of the reference system JEV 1.13

JEV 1.13 is included as a reference system. The results demonstrate that the environment generates non-trivial signals; they are not presented as an oracle.

In database migration, JEV got 6 of 12 trajectories on the public pass and 7 of 12 on the independent trusted pass, with 84 primary routing decisions. In code debugging, 12 of 20 on both passes, with 143 decisions. In formal proof, 17 of 30 on the public pass and 18 of 30 on the trusted pass, with 185 decisions. In structured data processing, only 2 of 20 on the public pass and 1 of 20 on the trusted pass, with 156 decisions.

The difference between the public pass and the independent trusted pass matters. It shows that external evaluation does not reproduce internal evaluation exactly, which is expected when the router and executor interact in a non-deterministic way.

Why the structured data cohort remains hard

The structured data cohort is a research preview that is hard by design. Its traces contain research, authoring, testing, validation, repair, precondition rejection, missing STOP and budget exhaustion, even though success on hard tasks has a strong floor effect. A 2 out of 20 public pass rate and 1 out of 20 on trusted does not mean the router is useless: it means the domain exposes failure modes that other benchmarks do not capture.

How to turn the benchmark into an evaluation plan

Before running routers, build a decision table. Start with workflow failure modes and assign weights before looking at local results. A code assistant can emphasize patches that pass tests and safe tool use. A document service can emphasize citation support, correct abstention and evidence retrieval.

Then lock the comparison conditions: exact model revision, inference engine, numeric precision, prompt, tool access, token budget, retry policy and hardware. Record quality, coordination, safety and efficiency separately, as RoleRouteBench does. If the application serves languages beyond the benchmark's main focus, build tasks in the native language instead of assuming results transfer.

To compare with other recent agent benchmarks, see the FeatureLoop and the CloudOpsBench.