# Reproducible leaderboard runs Everything the published public-suite numbers are made of, as commands you can re-run. The same recipe is what an OpenSmartRoute entry on RouterArena, LLMRouterBench or xRouteBench is built from; the runs themselves are recorded, with their submission status, in [results/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/leaderboard-results.md#listings). Requirements: `pip install "opensmartroute[yaml]"`, network access to the Hugging Face datasets-server (public suites; RouterBench is downloaded with `hf download`, see step 5). ## 1. Catalogue targets.yaml (`examples/leaderboard/targets.yaml`): three tiers, `llm-small` / `llm-mid` / `llm-frontier`. Battle datasets name 2023-24 models; `osr collect --tier` maps each model to the tier it played in (`eval.collect.ARENA_TIERS`), so every row's label is "which tier was good enough for this prompt" - the RouteLLM framing. ## 2. Collect the public suites ```bash osr collect --cache-dir data --limit 4000 \ --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small ``` Writes one `-tiers.jsonl` (+ `.meta.json`) per public source: `arena-55k`, `arena-100k`, `arena-140k`, `routellm-battles`, `ppe-human`, `webdev-arena`, `mt-bench-human`, `reward-bench`, `ultrafeedback`, `routellm-gpt4`. `--source NAME` restricts the run; `--refresh` ignores the cache. ## 3. Train the routing SLM on the training suites only ```bash osr -t examples/leaderboard/targets.yaml slm train --out slm.json --cache-dir data \ --source arena-55k-tiers --source routellm-battles-tiers --report --seed 0 ``` `--report` holds out 10 % and prints its accuracy. The suites in step 4 are **not** in `--source`, so they are held out at the suite level, not just the row level. ## 4. Evaluate on the held-out suites ```bash for suite in mt-bench-human ppe-human webdev-arena; do osr -t examples/leaderboard/targets.yaml --slm slm.json eval data/$suite-tiers.jsonl \ --baselines --calibration --robustness --limit 1000 --json > results/$suite.json done # declarative router (no learned signals) for the ablation column osr -t examples/leaderboard/targets.yaml eval data/mt-bench-human-tiers.jsonl --baselines --limit 1000 ``` What each flag reports and where it lands in the table: | Flag | Reports | Table column | |---|---|---| | (default) | accuracy, mean quality, cost, latency, mean confidence, raw ECE / Brier | Router, ECE raw | | `--baselines` | random / cheapest / best-prior / static task table / oracle, gap to oracle, noise floor | Static task table, Random, Cheapest | | `--robustness` | repeat flip rate, paraphrase stability, profile-swap dependence (first 200 rows) | Paraphrase | | `--calibration` | ECE, Brier, reliability bins, and `held_out`: temperature and isotonic calibrators fitted on the even rows, scored on the odd rows | ECE held-out | The published table used `--limit 1000` per suite and seed 0 throughout; the SLM bundle and the `Router` are deterministic, so re-running gives the same numbers for the same cache. ## 5. RouterBench and the other named benchmarks `withmartian/routerbench` is no longer gated: `hf download withmartian/routerbench --repo-type dataset --local-dir `. routerbench.py (`examples/leaderboard/routerbench.py`) runs the honest protocol on it - a 50 / 50 split by `sample_id`, catalogue and SLM from the train half, router vs the baselines (task table fitted on the train half) on the test half, frontier and calibration - and writes results/routerbench.json (`examples/leaderboard/results/routerbench.json`); the reading is in [results/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/leaderboard-results.md#routerbench-0-shot---11-real-models-real-task-labels). Other benchmark files in the same shape evaluate directly through the layout presets: ```bash osr -t examples/leaderboard/targets.yaml --slm slm.json eval routerbench_0shot.jsonl \ --preset routerbench --baselines --calibration --limit 5000 ``` `noise_floor()` needs rows with *repeated* samples per model (`samples`); RouterBench 0-shot carries one score per (prompt, model), so the label-noise floor is not measurable from it - only suites that publish several draws per model can be scored against it. ## 6. Publish Commit `results/*.json`, the exact `osr --version`, the dataset ids, splits and row counts, and the SHA-256 of `slm.json` together; that triple is the reproducible config a leaderboard entry links to. The run published with 0.5.0 is in [results/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/leaderboard-results.md) (one JSON per suite, learned and declarative); it is the entry submitted to RouterArena, LLMRouterBench and xRouteBench. ## 7. Live multi-cloud run (real models, real prices) ../multicloud_benchmark.py (`examples/multicloud_benchmark.py`) routes ../multicloud_prompts.jsonl (`examples/multicloud_prompts.jsonl`) across three price tiers on Azure OpenAI, Amazon Bedrock and Google Vertex AI, answers every prompt for real, has a judge grade the routed answer against an always-frontier baseline blind, and prices both from the providers' token usage. Reports land in ../results/multicloud/ (`examples/results/multicloud`): the cold-start run (150 prompts, 8 models, one judge) measured 56 % lower cost than all-gpt-4.1 at 9.22 vs 9.35; with the routing rules in ../multicloud_rules.yaml (`examples/multicloud_rules.yaml`), a warm start from that run's judged outcomes and two judge families (gpt-4.1 and gemini-2.5-pro, 73 % agreement) the second run measured 64 % lower cost at 9.44 vs 9.39. `--matrix` then had every model answer every prompt directly, judged the same way (matrix-20260910-093639.md (`examples/results/multicloud/matrix-20260910-093639.md`)): a single cheap model, gpt-oss-120b on Bedrock, scored 9.66 for $0.047 and beat that router (9.44, $0.087) - the hand-set quality priors were wrong for this traffic and the default `Objective(cost=0.15)` kept 18 % of prompts on gpt-4.1. `--replay` scores router configurations offline against the matrix cells (no calls, replay-20260910-152410.md (`examples/results/multicloud/replay-20260910-152410.md`)): the objective weight is the lever, warm-starting the learners is not (it spread traffic and raised cost at the default weight). The predicted best setting, `Objective(cost=1.0)` with the rules and no learned state, was then run live (20260910-100323.md (`examples/results/multicloud/20260910-100323.md`), final table matrix-20260910-100351.md (`examples/results/multicloud/matrix-20260910-100351.md`)): **89 % lower cost than all-gpt-4.1 ($0.026 vs $0.246) at 9.53 vs 9.27**, 68 wins / 52 ties / 30 losses, p50 1.2 s vs 2.0 s - second only to gpt-oss-120b direct (9.66, $0.047) and 44 % cheaper than it. Routing does not reduce tokens (reasoning models bill their thinking) and prompt compaction saved nothing on these short prompts; the savings come from which model bills the tokens. Total spend of all runs about $6.60. Credentials are read from the environment (see the module docstring); `--probe` checks every model with one tiny call, `--cap-usd` bounds the spend and `--rejudge` re-scores a run's stored answers without re-running the models.