Loading
90 days of the enterprise, pro and team plans free, no card needed. See the plans
Benchmark
150 real prompts, every one answered through the router and, for the baseline, by the frontier model; two frontier judges scored each pair blind, 1 to 10. Real provider calls at list price, 2026-09-10, OpenSmartRoute 1.1.0. The files behind every number are in the repository and the commands to re-run them are at the end of this page.
89.3%
less spent
$0.0263 routed against $0.2457 always-frontier, for 150 prompts
9.53 vs 9.15
judged quality, routed vs baseline
Mean of two blind judges, 1 to 10; routed answers won 68, tied 52 and lost 30
1.17 s vs 2.01 s
median latency, routed vs baseline
A smaller model answers most prompts faster
94%
of routed answers within one point of the baseline
The two judges agreed on the winner on 72% of prompts
Reference and judges: azure-gpt-4.1 answered every prompt as the baseline; azure-gpt-4.1 and gcp-gemini-2.5-pro judged each pair blind. One direct model scored higher than the router (9.66 for $0.0467); the router reached 9.53 for $0.0263 - on this catalogue it chose between exactly those two models, request by request. Routing changes which model bills the tokens, not how many are sent; reasoning models bill their thinking as output, so a cheaper row can show more tokens.
Classification and FAQ answers judged lower when routed to the small model - the cases a quality rule or a reported outcome would move to the larger one. The run was a cold start: no learned state, no feedback. The router learns from outcomes, so these are its worst numbers, not its best.
What the router buys is the cost axis. At the default objective it spends $0.34 per thousand calls against $2.98 for the table and $3.31 for always-frontier, for eleven points of accuracy. Told that quality is all that matters, it reaches 0.810 with a higher realised quality than the table.
Where it loses. With real task labels a lookup table fitted on the training half is a strong, cheap competitor: 0.830. Between cost weights 0.1 and 0.3 the frontier drops off a cliff, because on this catalogue the cheap models are much worse - the operator chooses the side of the cliff; the router does not pretend there is a middle.
Calibration. Raw confidence with eleven candidates was badly under-confident (ECE 0.55); a calibrator fitted on the even rows brings the odd rows to ECE 0.01. The platform ships one with every catalogue this wide.