Not every task needs the most expensive model. Asking for the time of a meeting and refactoring an entire system are very different jobs. Model routing solves this: a component decides, for each request, which model will handle it.
How it works
The router sits between the application and the models. It receives the request, analyzes it and chooses the destination. This analysis can follow three paths:
- Fixed rules: short requests go to a small model, requests with code go to a code model.
- Classifier: a lightweight model reads the request and estimates the difficulty.
- Business criteria: maximum cost, response time and provider availability.
After the choice, the router sends the request to the model and returns the response. Whoever uses the system does not even notice the switch.
A good router also measures the result. If the cheap model makes many mistakes on a certain type of task, the rule changes and that type starts going to a stronger model.
Practical example
Imagine an online store with a customer service assistant. Questions like "what is the delivery time?" go to a small, fast model. A long complaint, with product exchange and refund, goes to a more capable reasoning model.
In a simple configuration, the rule can be something like this:
{
"regras": [
{ "se": "tokens < 200", "modelo": "modelo-pequeno" },
{ "se": "contem_codigo", "modelo": "modelo-de-codigo" },
{ "padrao": "modelo-grande" }
]
}The names above are just illustrative. You put the models you actually use.
What it's for
The main use is reduce cost without losing quality where it matters. Since price per token varies a lot between models, sending simple work to a cheap model saves a lot at high volume.
It also serves to gain speed, since smaller models respond faster, and to provide resilience: if a provider goes down, the router sends the request to another. This last case connects to model fallback.
In teams that use several agents, routing allows giving each role the most suitable model: a strong model to plan, a more economical one for repetitive tasks. Orchestration platforms like bridgecode follow this logic by letting each agent use its own model.
Difference between routing and fallback
Routing chooses the best model before the call. Fallback acts after a failure, trying a backup model. The two usually work together in an AI gateway.
As a point of attention, the router can also be wrong. If it misclassifies a difficult task as easy, the response gets worse. Therefore it is worth tracking the quality of responses per route and adjusting the rules gradually.