# Published runs - opensmartroute 1.1.0 Produced with the recipe in [../README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/leaderboard.md) on 2026-09-10, `osr 1.1.0`, seed 0, `--limit 1000` per suite (the 0.5.0 run of 2026-09-06 is quoted in the comparison column below). One JSON per suite: `.json` is the learned router (`--slm slm.json --baselines --calibration --robustness`), `.declarative.json` is the same catalogue without the SLM (`--baselines` only). The per-row arrays `calibration.conformal_pairs` and `calibration.temperature_samples` were removed from the committed files; every aggregate they feed (`reliability`, `held_out`) is kept. ## Data | Role | Cache name | Hugging Face dataset | Rows after `--tier` relabelling | |---|---|---|---| | train | `arena-55k-tiers` | `lmarena-ai/arena-human-preference-55k` | 2 084 | | train | `routellm-battles-tiers` | `routellm/gpt4_judge_battles` | 4 000 | | held out | `mt-bench-human-tiers` | `lmsys/mt_bench_human_judgments` (`human`) | 2 228 | | held out | `ppe-human-tiers` | `lmarena-ai/PPE-Human-Preference-V1` (`test`) | 2 360 | | held out | `webdev-arena-tiers` | `lmarena-ai/webdev-arena-preference-10k` (`test`) | 1 070 | Catalogue: ../targets.yaml (`examples/leaderboard/targets.yaml`) (`llm-small` / `llm-mid` / `llm-frontier`). SLM bundle: `osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0` -> 6 051 rows, 10 % holdout (605 rows) accuracy 0.724, ECE 0.051, Brier 0.196; the constant-predictor baseline on the same holdout is 0.727, so the SLM's measured `skill` on pairwise tier labels is 0 - the report says so rather than hiding it. `sha256 d1a101088b179811c72bef873ebace926ef855f52b03573bfe983d2470469fde` (4.0 MB, not committed; the command is deterministic for the same cache). The 0.5.0 bundle was `8faf8466...cbbb1e` (5 956 rows, 0.721). ## Results | Suite (n=1000) | Router (learned) | 0.5.0 | Declarative | Static task table | Random | Cheapest | Best prior | Paraphrase | Repeat flips | ECE raw | ECE held-out temp. | ECE held-out isotonic | |---|---|---|---|---|---|---|---|---|---|---|---|---| | MT-Bench human | **0.514** | 0.523 | 0.421 | 0.541 | 0.441 | 0.372 | 0.343 | 1.000 | 0.000 | 0.198 | 0.085 | 0.050 | | PPE human | **0.563** | 0.552 | 0.418 | 0.651 | 0.445 | 0.174 | 0.651 | 0.894 | 0.000 | 0.244 | 0.169 | 0.116 | | WebDev Arena | **0.573** | 0.573 | 0.192 | 0.670 | 0.391 | 0.000 | 0.670 | 0.812 | 0.000 | 0.160 | 0.058 | 0.071 | Cost and latency per 1k tokens for the learned router: MT-Bench $0.00370 / 978 ms, PPE $0.00460 / 1096 ms, WebDev $0.00331 / 933 ms (static task table: $0.003 / 900 ms on MT-Bench, $0.015 / 2500 ms on PPE and WebDev, since it degenerates to "always the best single tier"). Reading: the learned router beats the declarative router by +9 to +38 pp and random / cheapest everywhere, at a fifth to a third of the cost of "always frontier" on PPE and WebDev. Against 0.5.0 the accuracies moved by -0.9 / +1.1 / 0.0 pp - noise - while **raw** ECE fell from 0.28-0.33 to 0.16-0.24: the 1.1.0 SLM trains with a validation split, early stopping and a calibrator fitted on the validation rows (G-1 in the gap plan), so the shipped confidence is closer to honest before any post-hoc fitting. It still does **not** beat the best single tier on any suite (-3 to -10 pp): a pairwise battle between two named models is a weak label for a three-tier decision, and the label-noise floor cannot be measured on these rows (one sample per prompt). A calibrator fitted on the even rows and scored on the odd rows brings held-out ECE to 0.05-0.12 (isotonic). These are the numbers a RouterArena / LLMRouterBench / xRouteBench entry should quote; the gated multi-sample suites (step 5 of the recipe) are what would let the router be scored against the noise floor. ## RouterBench (0-shot) - 11 real models, real task labels routerbench.json (`examples/leaderboard/results/routerbench.json`), produced by ../routerbench.py (`examples/leaderboard/routerbench.py`) on 2026-09-10 with `osr 1.1.0`: `withmartian/routerbench` `routerbench_0shot.pkl` (36,497 prompts from ~85 named evaluations, 11 models with per-response correctness and cost), split 50 / 50 by a hash of `sample_id`. The catalogue (quality prior = mean correctness, cost = mean $ per response) and the SLM are built from the **train** half only (18,258 rows; the SLM's 10 % report holdout: accuracy 0.785 against a constant-predictor baseline of 0.804); everything below is scored on the **test** half (18,239 rows). The static task table is fitted on the train half; its in-sample variant (fitted on the rows it is scored on, which is what `osr eval --baselines` did by default when this run was made - it now splits by `--seed` and fits on the train half) is shown for reference. | Policy (test half, n=18,239) | Accuracy | Realised quality | $ per 1k calls | |---|---|---|---| | Oracle (per-prompt best model) | 1.000 | 0.912 | 0.22 | | Static task table, fitted on train | 0.830 | 0.780 | 2.98 | | Static task table, in-sample | 0.835 | 0.784 | 2.98 | | Always gpt-4-1106-preview (= best prior = most expensive) | 0.819 | 0.780 | 3.31 | | **Learned router, cost weight 0** | 0.810 | **0.786** | 3.25 | | **Learned router, cost weight 0.1** | 0.805 | - | 2.08 | | **Learned router, default objective (cost weight 0.15)** | 0.716 | 0.682 | **0.34** | | Learned router, cost weight 0.3 | 0.603 | - | 0.14 | | Random | 0.552 | 0.521 | 0.84 | | Cheapest (mistral-7b) | 0.317 | 0.310 | 0.05 | | Declarative catalogue, no SLM | 0.313 | 0.315 | 0.05 | Reading, in order of what matters: - **The router does not beat the static task table on RouterBench.** With real task labels the table is a strong, cheap-to-fit competitor: 0.830 on the train-fitted variant. The router reaches 0.810 when told that quality is all that matters (cost weight 0), with a *higher* realised quality than the table (0.786 vs 0.780 - it picks a better-than-gpt-4 model on some prompts) but 9 % higher spend. Per task it loses to the table on 78 of 85 evaluations at the default objective. The v0.4 criterion stays **not met**, now measured on the suite it was written for. - **What the router buys is the cost axis.** At the default objective it spends **9x less** than the table ($0.34 vs $2.98 per 1k calls) for -11 pp; at cost weight 0.1 it spends 30 % less for -2.5 pp. Between weights 0.1 and 0.3 the frontier drops off a cliff (0.805 -> 0.603): on this catalogue the cheap models are much worse, so "a little cheaper" is not available - the operator has to choose the side of the cliff. - **The label-noise floor cannot be measured here either.** RouterBench 0-shot carries one score per (prompt, model); `noise_floor()` needs repeated samples and reports 0 by construction. The recipe's earlier claim that these rows carry several samples was wrong and has been removed. - **Calibration.** Raw confidence with 11 candidates is badly under-confident (mean 0.17-0.26 at 0.72-0.81 accuracy, ECE 0.55); an isotonic calibrator fitted on the even rows brings the odd rows to **ECE 0.010** (temperature scaling alone: 0.28). Ship a fitted calibrator with any catalogue this wide. - The SLM's own skill is again nil against the constant predictor on its holdout (0.785 vs 0.804): the learned gain over the declarative catalogue (0.313 -> 0.716 / 0.810) comes from the priors, the cost term and the bandit / similarity ensemble, not from the SLM as a stand-alone classifier. To reproduce: `hf download withmartian/routerbench --repo-type dataset --local-dir ` (the dataset is no longer gated), then `python examples/leaderboard/routerbench.py --pkl /routerbench_0shot.pkl --work ` (needs `pandas` to read the pickle; about 15 minutes, no network after the download). ## Listings Where this run has been submitted. A submission links the exact commit, the SLM SHA-256 above and this page. | Leaderboard | Entry | Status | Notes | |---|---|---|---| | RouterArena | - | not submitted | submission = a PR to the RouterArena repository with `runs/*.json` | | LLMRouterBench | - | not submitted | needs the gated multi-sample suites (step 5) for the noise-floor comparison | | xRouteBench | - | not submitted | component ablation report from `osr eval --ablation` |