When you hear 'model routing' it sounds simple: you pick the best model for the task and that’s it. Sounds familiar, right? What if I tell you that rarely happens in real agentic systems? What looks like a classification problem quickly becomes a systems optimization problem, with three dimensions that complicate things in unexpected ways.
Why classifying fails
The common intuition is to estimate a request's difficulty and send the hard ones to powerful models. Makes sense, doesn’t it? In practice this runs into two clear limits.
First, difficulty is often invisible at routing time. A seemingly simple prompt—like summarizing a contract—can trigger document retrieval, compliance checks, tool usage and several rounds of refinement. Only when you run the task do you learn how costly it will be.
Second, even with a perfect difficulty estimate, that’s only one factor. In production you must balance cost, quality, latency and reliability at the same time. Then add business constraints: compliance rules, data residency, privacy and approved-model lists. The router has to respect all that without breaking the experience.
In practice, routers aren’t solving a single problem. They’re constantly balancing cost, quality, latency, compliance and reliability.
Costs, cache and the surprise of pricing
A concrete finding that breaks token-price logic: Claude Sonnet 4.6 turned out cheaper than GPT-4.1 in a real experiment. Over 417 tasks using the same CodeAct agent, Sonnet cost $79 total ($0.19 per task) while GPT-4.1 cost $155 ($0.37 per task). On paper, GPT-4.1 had better price-per-token and lower in/out fees, so it should have won easily.
The explanation was cache. Agents tend to reuse large chunks of context between steps. When cache hit rates are high, the effective input cost drops dramatically. Sonnet had a lower cache read price, which gave it a disproportionate edge despite using more reasoning steps.
The lesson: real cost comes from the interaction between model, workload and serving infrastructure. A router that only looks at price sheets is optimizing against the wrong numbers.
Latency and granularity: it’s not just model size
It’s tempting to think a bigger model is always slower and a smaller one faster. But user experience depends on many factors: routing overhead itself, the hardware running the model, whether cache is hot and endpoint load. A theoretically faster model can feel slower if service conditions aren’t favorable.
Granularity also matters. Routing once per task adds little overhead. But routing at each step of a flow lets you adapt on the fly and improve efficiency, although every decision point adds latency and operational complexity.
A router that ignores serving state is optimizing against the wrong reality.
Practical approach: treat routing as optimization
The authors changed the question. Instead of asking ‘which model is best for this task’, they framed an optimization problem: maximize quality and latency while minimizing cost, without the router itself becoming a bottleneck. The result is a mapping of operational points you can pick from depending on priorities.
In the AppWorld Test Challenge with a CodeAct agent, the router traces a cost-accuracy frontier where each configuration is an operating point. For example, Configuration 1, optimized for latency, reached 84% accuracy for $93 and 83 seconds latency. That reflects a 21% cost reduction and 9% latency reduction compared to running Opus alone, with only a 4% drop in accuracy. Other configurations push cost down further.
A router based only on difficulty (the teal point in the original figure) lands in a region of similar accuracy but higher cost. The advantage of the optimization approach is exploring the tradeoff space instead of sticking to a single criterion.
Technically, the designed router is very lightweight: around 6 ms and 2 kB of memory per task, light enough to avoid becoming the bottleneck teams fear.
Practical recommendations for builders
- Measure cache hit rate and compute effective input cost, not just price per token.
- Watch the infrastructure: hardware, request queues and endpoint health can change latency more than model size.
- Consider routing granularity: per-step routing gives flexibility but adds latency and operational complexity.
- Integrate compliance and data-residency policies into routing logic, not as a post-filter.
- Treat the router as a lightweight optimizer that exposes different operating points to choose from based on priority (cost vs latency vs quality).
Final reflection
The core message is simple but powerful: routing isn’t just choosing models, it’s designing the best operating point for the whole system. Models, cache, infrastructure and governance are interdependent variables. Solving this is harder than classifying, but it’s also where the biggest efficiencies appear.
If you’re adding routing to agentic systems, start by measuring real interactions between cache, load and endpoints before trusting fixed rules. The real win isn’t finding the ‘best model’ per task, it’s finding the best configuration for your system.
Original source
https://huggingface.co/blog/ibm-research/model-routing-is-simple-until-it-isnt