# OpenSmartRoute documentation
> OpenSmartRoute is an open-source router that sits between your team, your apps and the AI models they use. For every request it picks the most cost-effective model, agent or tool that meets your quality rules, keeps personal data and secrets out of the prompt before it is sent, and shows why it chose what it did. Upload last month's bill and see what you would have saved.
Every published page of https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs as Markdown, in reading order. The short index is https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/llms.txt.
---
# OpenSmartRoute
> **An open, intelligent route to the right decision, solution, or destination.**
OpenSmartRoute is an open-source **AI decision control plane**. For each request it picks the best
**LLM, agent harness, skill, persona, tool, workflow or human**, honours hard constraints (privacy,
region, budget, latency), explains the choice, executes the resulting plan and **learns from every
outcome**. The core is pure Python with zero runtime dependencies.
```mermaid
flowchart LR
R([request]) --> G["guard` ([deploy/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md)).
From a checkout: `pip install -e '.[dev]'`.
Then sign in. The hosted platform (community or enterprise edition) uses a browser hand-shake; a
self-hosted `osr serve` accepts a token you generate yourself:
```bash
osr login # opens https://opensmartroute.ai/cli/authorize
osr login --url https://osr.example.com # your own platform deployment
osr login --url http://router:8000 --token osr_local_... # self-hosted server (see `osr serve --generate-token`)
osr whoami # workspace, plan, edition, key
```
Credentials live in `~/.config/opensmartroute/credentials.json` (`%APPDATA%\opensmartroute` on
Windows), one profile per `--profile`; `OSR_API_URL` / `OSR_API_KEY` override them in CI.
## Quick start
Three lines, nothing configured - a router in your process over the built-in starter catalogue (the one
`osr route` uses without `--targets`):
```python
from opensmartroute import Router
router = Router.starter()
print(router.route("Prove that sqrt(2) is irrational, step by step.").target.id) # llm-frontier
```
Or the hosted platform (or your own `osr serve`) over HTTP, with the key `osr login` saved or `OSR_API_KEY`:
```python
from opensmartroute import OpenSmartRoute
osr = OpenSmartRoute()
d = osr.route("Refund order #123, the customer is upset", objective={"cost": 0.4, "quality": 0.6})
print(d.target_id, d.confidence, d.explanation)
osr.feedback(d, success=True, quality=0.9) # the learners improve on your traffic
print(osr.ask("What is the capital of France?")) # OpenAI-compatible chat, the router picks the model
```
Timeouts, retries with `Retry-After`, typed errors that never carry the key, and no key over plain `http://`
to a public host - see [docs/SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md). Your own catalogue, in full:
```python
from opensmartroute import (Router, TargetRegistry, RouteTarget, TargetKind, Capabilities,
TargetConstraints, Outcome)
registry = TargetRegistry([
RouteTarget("llm-small", TargetKind.LLM,
capabilities=Capabilities(max_complexity=0.45),
cost={"usd_per_1k_tokens": 0.0002}, latency_ms=300, quality_prior=0.55,
examples=["Hi, how are you?", "What is the capital of France?"]),
RouteTarget("llm-frontier", TargetKind.LLM,
capabilities=Capabilities(min_complexity=0.5, domains=["math", "coding"]),
cost={"usd_per_1k_tokens": 0.015}, latency_ms=2500, quality_prior=0.93,
examples=["Prove the theorem step by step."]),
RouteTarget("llm-onprem", TargetKind.LLM, # the only target allowed to see private data
constraints=TargetConstraints(regions=["eu"], data_boundary="private", pii_allowed=True),
cost={"usd_per_1k_tokens": 0.004}, latency_ms=900, quality_prior=0.7),
RouteTarget("human", TargetKind.HUMAN,
capabilities=Capabilities(actions=["escalate"], tags=["safety"]),
cost={"usd_per_1k_tokens": 0.5}, latency_ms=300_000),
])
router = Router(registry)
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, f"{d.confidence:.2f}") # llm-frontier, with the calibrated confidence
print(d.trace.explain()) # per-strategy scores and rationales
router.learn(Outcome(request_id=d.request_id, target_id=d.target.id, success=True,
quality=0.9, cost_usd=0.002, latency_ms=1800, domains=d.trace.signals.domains))
```
Hard constraints and a per-request objective - the policy filters before anything is scored, and the
trace says who was rejected and why:
```python
from opensmartroute import RouteRequest, RequestConstraints, Objective
req = RouteRequest("Summarize this patient intake note.",
constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005),
objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6))
d = router.route(req)
print(d.target.id) # llm-onprem
print(d.trace.policy_rejections) # {'llm-small': 'data boundary public < required private', ...}
```
When no target is admissible the router raises `NoRouteError` (`details["rejections"]` carries the same
map) instead of guessing.
Catalogue and rules from YAML, evaluated and served from the command line:
```bash
osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan
osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier
export OSR_LLM_BASE_URL=https://api.openai.com/v1 OSR_LLM_API_KEY=sk-... # any OpenAI-compatible endpoint
osr -t examples/targets.yaml -r examples/rules.yaml serve # http://127.0.0.1:8000/docs
```
`serve` routes without a provider; `OSR_LLM_BASE_URL` (OpenAI, Azure, vLLM, Ollama, OpenRouter, ...)
is what lets `/v1/chat/completions` and `router.run()` actually run the chosen model.
Wiring real providers, executing plans with `router.run()`, agent harnesses, SKILL.md and MCP
catalogues, the decorator SDK and the enterprise builder are covered step by step in
[docs/GUIDE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md).
## What it routes
| Kind | Examples | Executed by |
|---|---|---|
| `llm` | model endpoints, per reasoning effort or token budget | OpenAI-compatible client or any callable |
| `agent` | coding, research and support harnesses with tools and memory | callable, HTTP or subprocess harness |
| `skill` | deterministic capabilities, SKILL.md packages | your function; instructions disclosed to the model |
| `persona` | system-prompt layers composed on top of the primary target | `run()` prompt composition |
| `tool` | MCP or function tools, with schema-aware matching | tool call - as the routed target, or offered to the plan's model as a function it may call |
| `workflow` | fixed multi-step pipelines | workflow engine |
| `human` | queues and experts with Erlang-C capacity maths | ticketing or hand-off; also the abstention target |
Catalogues load from YAML or JSON, MCP `tools/list` payloads, A2A agent cards, SKILL.md and persona
directories and vLLM semantic-router configurations; the router plugs into LangGraph and Agent
Framework graphs as a node and into any chat loop as an OpenAI tool.
## How it decides
1. **Signals** (under 1 ms): task type, domains, complexity, reasoning need, PII, language, modality,
history; optional verbalised difficulty, cheap-draft features and hidden-state probes.
2. **Policy**: hard constraints filter targets before any scoring.
3. **Strategies**: rules, capability fit, example similarity, task table, Thompson and LinUCB bandits,
IRT, Bradley-Terry, Markov lookahead, multi-turn history embeddings, learning-to-defer, edge/cloud
tiers, token budgets, auctions, user adaptation and an LLM judge consulted only below a confidence
threshold; optionally a routing SLM (`RouterSLM`) distilled from the whole ensemble that keeps
improving on its own from outcomes, public routing datasets, a live model catalogue and the article
feeds you point it at - a trained one ships inside the package (`--slm bundled`,
`RouterBuilder.with_slm()`), so an on-prem install starts warm
([guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#the-routing-slm-and-the-self-improvement-loop)).
4. **Utility**: confidence-weighted ensemble, then
`w_q * quality - w_c * norm(cost) - w_l * norm(latency)` with a hard quality floor; temperature
scaling and conformal candidate sets calibrate the confidence; the router abstains when nothing is
safe enough.
5. **After the answer**: cascades and self-escalation stop or reroute on response-side uncertainty
(semantic entropy, P(True), streaming competence posterior); mixture-of-agents aggregation and
permanent hand-off policies cover agentic trajectories.
Formulas and citations: [docs/MATH.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MATH.md), [docs/RESEARCH.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/RESEARCH.md).
## Production
```python
from opensmartroute import RouteRequest, RequestConstraints
from opensmartroute.enterprise import RouterBuilder, MetricsTelemetry, FileAuditSink, TenantMiddleware
from opensmartroute.security import GuardMiddleware
app = (RouterBuilder(registry)
.with_defaults().with_auto_learning(state_dir=".osr-state")
.with_health(latency_slo_ms=3000)
.with_middleware(GuardMiddleware(redact=True),
TenantMiddleware({"acme": {"deny_targets": ["llm-frontier"]},
"globex": {"data_boundary": "on_prem"}}))
.with_telemetry(MetricsTelemetry()).with_audit(FileAuditSink("audit.jsonl"))
.build())
req = RouteRequest("Prove that sqrt(2) is irrational.", constraints=RequestConstraints(tenant="acme"))
d = app.route(req) # guard -> tenant -> policy -> strategies; audited and measured
print(d.target.id, app.health_snapshot())
# app.run(req) executes the plan and records the Outcome once targets carry handlers
# (`attach_chat_handlers`, agent harnesses, MCP tools - docs/GUIDE.md); without one it raises ExecutionError.
```
Routing overhead is 4.6 ms p50 with the default strategies and 7.4 ms with every learner enabled on the
27-target `examples/targets.yaml` with its rules (the repository's benchmark, 300 requests, CPython 3.11 on a
Windows laptop, re-measured 2026-10-09; the continuous build fails when p95 passes 50 ms); retrieve-then-rank
keeps it flat for thousands of tools
([measurements](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#10-routing-latency)). `osr serve` (or `create_app(app)` with the
router above) exposes a FastAPI app and an OpenAI-compatible proxy (`/v1/chat/completions` with streaming
and tool calling, `/v1/responses`, `/v1/embeddings`, `/v1/models`), the Anthropic Messages API (`/v1/messages`)
and an MCP endpoint (`/mcp`): point the official OpenAI or Anthropic
SDK at it with `model="osr/auto"` and keep the rest of the code; guard and tenant violations return 400, no
admissible target 422, an unreachable provider 503 and a failing target falls back to the next candidate.
Circuit breakers open on provider failures and recover on their own;
learner state survives restarts from `state_dir` or a Redis / SQL store. Deployment references:
[deploy/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md), [docs/ENTERPRISE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ENTERPRISE.md).
## Hosted platform
platform/ (`platform/README.md`) packages the router as a service from two container images: `osr-platform-api`
(FastAPI: self-serve signup with hashed API keys, plans and quotas, usage metering, the metered
`/api/v1/route` family with the full trace, per-tenant constraints and workspace policy, the audit
trail, per-request traces and a live event stream, and the OpenAI-compatible `/v1/chat/completions`
proxy - one process, or one container per domain behind a gateway) and `osr-platform-web` (Next.js: landing page, the rendered documentation, pricing, a live
playground and the account dashboard with activity, events, governance and health pages; it proxies
`/api` and `/v1` so the browser only talks to one origin). `OSR_PLATFORM_EDITION=community` runs the core
`Router`; `enterprise` runs the builder above with auto-learning, health, guard, metrics, a
hash-chained audit log and per-tenant constraints. The Azure solution template deploys everything to Azure
Container Apps with Azure OpenAI ([docs/INSTALL.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/INSTALL.md)); the reference deployment is
https://opensmartroute.ai. The end-user guide is
[docs/PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md); the REST API reference is generated from
the platform's OpenAPI document.
```bash
curl -s -XPOST $OSR/api/v1/signup -H 'content-type: application/json' -d '{"email":"you@example.com"}'
curl -s $OSR/api/v1/route -H "authorization: Bearer $KEY" -H 'content-type: application/json' \
-d '{"text":"Prove that sqrt(2) is irrational."}'
```
From TypeScript or JavaScript, the client on npm - `npm install @opensmartroute/sdk`
([platform/sdk-ts](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/sdk-typescript.md)) - wraps the same endpoints and the OpenAI-compatible `/v1`
surface with streaming, zero runtime dependencies, Node 20+ / Deno / Bun / edge / browser:
```ts
import { OpenSmartRoute } from "@opensmartroute/sdk";
const osr = new OpenSmartRoute({ baseUrl: "https://opensmartroute.ai", apiKey: process.env.OSR_API_KEY! });
const decision = await osr.route({ text: "Prove that sqrt(2) is irrational.", top_k: 3 });
```
## Security
Routing is a control plane; its integrity is a security property. `InputGuard` combines a heuristic
and a learned detector for confounder gadgets that reroute queries, an injection-risk scorer, PII
redaction and prompt sanitisation for the LLM judge. Per-task resource limits, an origin policy for
sensitive tool parameters, signed MCP manifests, AES-GCM state, content-free logs and hash-chained
audit complete the model. CI runs `ruff -S`, `mypy`, `bandit`, `pip-audit` and the `osr safety` red-team
suite. Threat model: [docs/SECURITY.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SECURITY.md). Reporting: [SECURITY.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/security-policy.md).
## Documentation
The user documentation is a searchable, versioned site at
https://opensmartroute.ai/docs (built from these files
at release time, with a REST API reference generated from the platform's OpenAPI document). It covers
the hosted platform, the Python SDK and the pip package; the internal planning, go-to-market, brand
and sales documents below stay in the repository only.
| Document | Contents |
|---|---|
| [docs/QUICKSTART_PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_PLATFORM.md) | Platform quickstart: workspace and key, first `/api/v1/route` call, the OpenAI-compatible endpoint, feedback |
| [docs/QUICKSTART_SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_SDK.md) | Python SDK quickstart: install, describe targets, route, constraints and objectives, learning, the CLI |
| [docs/INSTALL.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/INSTALL.md) | Install the platform yourself: Azure Marketplace offer, Docker Compose on your own servers (on-premises, air-gapped), first sign-in, custom domain, upgrade and backup |
| [docs/USE_CASES.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/USE_CASES.md) | Real-world use cases by industry: nine complete, working catalogues (healthcare, banking, legal, retail support, engineering, HR, marketing, travel, analytics) with the requests, the routed target and the reason - every decision verified on every change |
| [docs/GUIDE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md) | User guide: targets, routing, execution, providers, learning, SDK, enterprise builder, research-track modules, CLI, latency, layout |
| [docs/PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md) | Platform guide: authentication, `/api/v1/route`, execution, the OpenAI-compatible endpoint, feedback, plans and quotas, organizations, tenants, governance, observability (traces, events, readiness), the MCP server, dashboard |
| [docs/PLATFORM_JOURNEYS.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM_JOURNEYS.md) | Platform journeys: who uses the platform, what each plan unlocks (feature-to-page map), the website, dashboard and console navigation, the new user step by step, existing-user tasks by role, plan changes, and offboarding checklists for a person leaving a team, a person leaving the platform and a team shutting a workspace down |
| [docs/MODELS.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MODELS.md) | Model server: host open-weight models on your own hardware (CPU or CUDA) behind the Ollama and OpenAI APIs - pulls from ollama.com and Hugging Face, Ollama's on-disk store, always-warm residency with a fit check, admission queue, metrics, settings, use with the platform |
| [docs/MARKETPLACE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MARKETPLACE.md) | Marketplace: find, install, buy and publish agents, skills, personas, prompts and stack templates; ratings, review lifecycle, `osr stack` |
| [docs/MCP.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MCP.md) | Cost estimates (`estimate`, `POST /api/v1/estimate`), recommended models per use case and the MCP server for VS Code, Cursor, Claude, Windsurf and agents |
| [docs/PUBLISHING.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PUBLISHING.md) | Content publishing for your website: your feeds and briefs become posts written through the router as your workspace's requests, delivered to your site (signed webhook, WordPress, Ghost) or pulled as RSS; `/api/v1/content`, the `content_*` MCP tools and the dashboard page |
| [docs/EXTENSION.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/EXTENSION.md) | Browser extension (Chrome, Edge, Firefox): the pre-send advisor on ChatGPT, Claude, Gemini, Copilot and Perplexity - task and tokens, PII redaction, model hints, on-device or platform quotes, analysis policy, device sign-in |
| [docs/DESKTOP.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/DESKTOP.md) | Desktop app (Windows, macOS, Linux): the routed command bar over any application - shortcut, the selection read and pasted back, on-device checks, connecting a workspace, installing the extension into every browser, updates, privacy, troubleshooting |
| [docs/VSCODE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/VSCODE.md) | VS Code extension: the routing panel for the selection - task, language, tokens, personal data, the recommended model and its cost - and Ask / Explain / Rewrite in place through the router; device sign-in, what is sent, settings |
| [docs/OFFICE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/OFFICE.md) | Word add-in: the pane beside the document - task, language, size, personal data, the recommended model and its cost for the selected passage - and Ask / Explain / Summarise / Rewrite / Proofread through the router; the manifest, sideloading and admin-centre deployment, sign-in, what is sent |
| [docs/SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md) | API reference for the decorator SDK, components and settings |
| [docs/REFERENCE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/REFERENCE.md) | Generated API reference: every module and exported name |
| [docs/ARCHITECTURE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ARCHITECTURE.md) | Request path, module boundaries, performance envelope |
| [docs/ENTERPRISE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ENTERPRISE.md) | Ports, stores, middleware, shadow and A/B, multi-replica operation |
| [docs/COMPATIBILITY.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/COMPATIBILITY.md) | Compatibility matrix: every model provider, client, agent framework and adapter the router connects to, how, and how each connection was verified (continuous tests, a dated live run, or the OpenAI-compatible protocol alone) |
| [docs/OBSERVABILITY.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/OBSERVABILITY.md) | Tracing and observability: spans and events for every stage, sinks (memory, metrics, log, file, OpenTelemetry), `/events`, `/trace`, `/metrics`, `OSR_OBSERVABILITY_*`; the platform's per-workspace `/api/v1/trace`, `/api/v1/events`, `/api/v1/status` and dashboard pages |
| [docs/MATH.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MATH.md) | Every formula the router uses, with derivations |
| [docs/AGENT_ENGINEERING.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/AGENT_ENGINEERING.md) | Prompt, harness, context, memory and agent engineering, integrations, guardrails and evals: where each lives in the SDK and the platform; the prompt registry; trace export to Langfuse and OTLP/OpenInference (Arize Phoenix, TruLens); datasets in Ragas / DeepEval / Langfuse shape |
| [docs/RESEARCH.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/RESEARCH.md) | Literature survey and the idea-to-module map |
| [docs/SECURITY.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SECURITY.md) | Threat model and hardening checklist |
| [docs/SECURITY_REVIEW.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SECURITY_REVIEW.md) | External security review pack: scope, trust boundaries, evidence, reviewer questions, review log |
| docs/ROADMAP.md (`docs/ROADMAP.md`) | Per-version exit criteria and status |
| [docs/RELEASES.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/RELEASES.md) | Versions, channels and releases: the one version every artifact carries, stable releases vs developer previews from `main`, how to read the build a deployment runs, unreleased changes, how a release is cut |
| docs/PLATFORM_PLAN.md (`docs/PLATFORM_PLAN.md`) | Platform strategy: agentic routing, Open Capability Manifest, marketplace, editions and pricing |
| docs/MODEL_SERVING_PLAN.md (`docs/MODEL_SERVING_PLAN.md`) | Plan for OSR Models: our own model hosting server (store, registry, engine lifecycle, Ollama- and OpenAI-compatible API) and the scalable inference gateway and fleet controller over llama.cpp and vLLM engines; research summary, architecture, phases, verification |
| docs/GO_TO_MARKET.md (`docs/GO_TO_MARKET.md`) | Go-to-market plan; the sales enablement kit lives in docs/sales/ (`docs/sales/README.md`) |
| [spec/ocm/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ocm.md) | Open Capability Manifest specification and JSON Schema |
| [deploy/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md) | Container image, Helm chart, reference deployments |
| platform/README.md (`platform/README.md`) | Hosted platform: `api/` (FastAPI) and `web/` (Next.js), playground, API keys, plans, editions, `OSR_PLATFORM_*` settings, Azure deployment |
| docs/BRAND.md (`docs/BRAND.md`) | Logo system and naming conventions |
.claude/skills/ (`.claude/skills`) ships Agent-Skills packages that teach coding assistants how to use
and extend the project; they are also valid routing targets (`osr skills` validates them) and each one is
published on the documentation site under `/docs/skills/`. Coding agents working in this repository
start from AGENTS.md (`AGENTS.md`).
## Roadmap
| Release | Theme | Status |
|---|---|---|
| 0.3 | Real integrations: OpenAI-compatible client, adapters, config loaders | Shipped |
| 0.4 | Learned signals and honest evaluation | Shipped |
| 0.5 | Target representations, effort and personalisation | Shipped |
| 0.6 | Multi-step and agentic routing | Shipped |
| 0.7 | Catalogue interop and discovery at scale | Shipped |
| 0.8 | Operations, risk control and economics | Shipped |
| 0.9 | Security hardening of the control plane | Shipped |
| 1.0 | Stable API, reference deployments, security review pack | Shipped |
| 1.1, 1.2 | Hosted platform: organizations, tenants, billing, marketplace, support, e-mail, operator console | Shipped |
Every item in the research track is implemented on `main` with tests; a version is **Complete** when
its measured exit criterion is published (8/8 met) and **Shipped**
when it is in a tagged release - the current release is 2.1.0 ([CHANGELOG.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/changelog.md)). 1.0 means what the repository can prove: frozen public API with a
deprecation path, published performance envelope, reference deployments, security controls with a
red-team suite in CI and a published review pack ([docs/SECURITY_REVIEW.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SECURITY_REVIEW.md)),
the public leaderboard run ([examples/leaderboard/results](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/leaderboard-results.md),
produced by the recipe in [examples/leaderboard](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/leaderboard.md)) and consecutive
releases without a breaking change (0.4.0 through 2.1.0, additions only). Adoption evidence - an independent security report, accepted
leaderboard listings, two production users (ADOPTERS.md (`ADOPTERS.md`)) - is tracked, never claimed,
and the release tooling refuses a release while a readiness row is open.
Details in docs/ROADMAP.md (`docs/ROADMAP.md`) and [CHANGELOG.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/changelog.md).
## Contributing
Issues and pull requests are welcome; see CONTRIBUTING.md (`CONTRIBUTING.md`). CI runs `ruff`, `mypy`,
`bandit` and `pytest` on CPython 3.10 to 3.13 on Linux and 3.12 on Windows, a routing-accuracy gate and the
safety suite on the example catalogue, and builds the distribution and the container image; a weekly
security run (also on every push) runs `bandit` over the SDK and the platform, `pip-audit` and
`npm audit` over the dependencies and the `osr safety` red-team suite. The test suite has no mocks of the router itself: the end-to-end suite
drives the shipped example catalogue through an in-process OpenAI-compatible provider fleet over
HTTP, a real MCP server subprocess over stdio, the CLI, the FastAPI service, provider outages with
breaker recovery, agentic multi-round loops and concurrent traffic. Releases are automated: a release
pull request bumps the version and rolls the changelog; merging it and running the release process
tags, publishes to PyPI (`twine check --strict`, project-scoped token) and pushes the image.
## License
Apache-2.0
---
# Platform quickstart
Route your first request through the hosted platform in about five minutes: create a workspace, get
an API key, ask the router for a decision, let it execute the answer, and report how it went. Every
call shown here is documented in full in the [platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md) and in the REST API
reference at `/docs/api`.
```mermaid
flowchart LR
APP["Your app (OpenAI client, model=auto)"] --> API["POST /v1/chat/completions"]
API --> AUTH["API key, plan, quota and budget"]
AUTH --> RT["Router.route()"]
RT --> EXE["Chosen provider or target executes"]
EXE --> RESP["Answer + X-OSR-Target + token usage"]
RESP --> LOOP["Metered, traced, learners updated"]
```
## 1. Create a workspace and an API key
Sign up at [opensmartroute.ai/platform/signup](https://opensmartroute.ai/platform/signup) with
GitHub, Google, Microsoft or an email address. Signup creates a personal workspace on the **free**
plan and issues an API key.
:::warning[The API key is shown once]
Copy it into a secret manager immediately - it cannot be retrieved again later. You can create,
rename, rotate and revoke more keys on the keys page of the dashboard at any time.
:::
Authenticate with either a copied key or the CLI browser hand-shake:
::::tabs
:::tab[API key header]
Send the key with every request in either header:
```http
Authorization: Bearer osr_...
X-API-Key: osr_...
```
:::
:::tab[CLI sign-in]
```bash
curl -LsSf https://opensmartroute.ai/install.sh | sh # Windows: irm https://opensmartroute.ai/install.ps1 | iex
osr login # opens the browser, mints a key for this machine
osr whoami # workspace, plan, edition behind the stored credential
```
:::
::::
## 2. Route a request
`POST /api/v1/route` returns a decision: the chosen target, the confidence, ranked alternatives, the
signals the router extracted and a human-readable explanation. Nothing is executed yet. The examples use the
hosted API origin `https://api.opensmartroute.ai`; the site origin (`https://opensmartroute.ai/api/v1/...`, the form
the dashboard's quick-start card shows) answers the same requests, and a self-hosted deployment uses its own URL.
::::tabs
:::tab[curl]
```bash
curl -s https://api.opensmartroute.ai/api/v1/route \
-H "Authorization: Bearer $OSR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"text": "Prove that sqrt(2) is irrational.", "top_k": 3}'
```
:::
:::tab[TypeScript]
```bash
npm install @opensmartroute/sdk
```
```ts
import { OpenSmartRoute } from "@opensmartroute/sdk";
const osr = new OpenSmartRoute({ baseUrl: "https://opensmartroute.ai", apiKey: process.env.OSR_API_KEY! });
const decision = await osr.route({ text: "Prove that sqrt(2) is irrational.", top_k: 3 });
console.log(decision.target.id, decision.confidence, decision.explanation);
```
:::
:::tab[Python]
```bash
pip install opensmartroute
```
```python
import os, httpx
r = httpx.post(
"https://api.opensmartroute.ai/api/v1/route",
headers={"Authorization": f"Bearer {os.environ['OSR_API_KEY']}"},
json={"text": "Prove that sqrt(2) is irrational.", "top_k": 3},
)
print(r.json()["target"]["id"], r.json()["confidence"])
```
:::
::::
```json
{
"request_id": "7f3c...",
"target": {"id": "llm-frontier", "kind": "llm", "name": "Frontier model"},
"confidence": 0.81,
"alternatives": [{"id": "llm-mid", "kind": "llm", "utility": 0.62}],
"signals": {"complexity": 0.74, "domains": ["math"], "reasoning_need": 0.9, "contains_pii": false},
"explanation": "llm-frontier: high reasoning need, math domain, quality weight dominates ..."
}
```
Two request fields cover most needs:
- `objective` trades quality, cost and latency per request: `{"objective": {"quality": 1.0, "cost": 0.5}}`
prefers cheaper targets whenever quality allows.
- `constraints` are hard limits, never traded off: `{"constraints": {"max_cost_per_1k": 0.005,
"data_boundary": "private"}}` removes every target that stores data less strictly or costs more,
and the response's `policy_rejections` says which target was dropped and why.
The full field tables - history, context, tenant, kinds, plans - are in the
[platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md#2-route-a-request).
## 3. Get the answer, not just the decision
The simplest way is the OpenAI-compatible endpoint: point any OpenAI SDK at `/v1` and set `model` to
`auto`. The router chooses the target per request, runs it and returns a standard `chat.completion`
whose `model` field is the target that answered; set `stream: true` and each provider token is
relayed as it arrives.
::::tabs
:::tab[Python]
```python
from openai import OpenAI
client = OpenAI(base_url="https://api.opensmartroute.ai/v1", api_key=OSR_API_KEY)
reply = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Write a haiku about routing."}],
)
print(reply.model) # the target the router chose, e.g. "llm-small"
print(reply.choices[0].message.content)
```
:::
:::tab[Node.js]
```javascript
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://api.opensmartroute.ai/v1", apiKey: process.env.OSR_API_KEY });
const reply = await client.chat.completions.create({
model: "auto",
messages: [{ role: "user", content: "Write a haiku about routing." }],
});
console.log(reply.model); // the target the router chose
console.log(reply.choices[0].message.content);
```
:::
:::tab[curl]
```bash
curl -s https://api.opensmartroute.ai/v1/chat/completions \
-H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"model": "auto", "messages": [{"role": "user", "content": "Write a haiku about routing."}]}'
```
:::
::::
Decision metadata (request id, confidence, alternatives, cost) rides along in an `opensmartroute`
object that other clients ignore. A `models` list makes the router choose among your candidates and
fall back down the list when a target fails. On the **pro** plan and above, `POST /api/v1/route` with
`"execute": true` does the same for any target kind - agents, skills, tools - and returns the output
in `result`.
### Name a model instead of `auto`
Every model the platform sells by the token is on the [storefront](/models?view=offers), with the price your
workspace pays per million tokens, and in `GET /v1/models` under its catalogue id (`openai/gpt-4.1-mini`).
| `model` | The router |
|---|---|
| `"auto"` | chooses across the whole catalogue. |
| `"openai/gpt-4.1-mini"` | chooses among the providers serving that model, by price, health and learned quality. |
| `"openai/gpt-4.1-mini@azure"` | uses that one endpoint. |
```python
reply = client.chat.completions.create(
model="openai/gpt-4.1-mini",
messages=[{"role": "user", "content": "Write a haiku about routing."}],
extra_body={"provider": {"data_collection": "deny", "sort": "price"}}, # optional: steer the endpoint
)
print(reply.model) # the endpoint that answered: "openai/gpt-4.1-mini@azure"
print(
reply.model_extra["opensmartroute"]["charge_usd"]
) # what the call cost you; the same figure lands on the usage row
```
The `provider` object takes `order`, `only`, `ignore`, `data_collection`, `sort` and `max_price`, as on
OpenRouter. The [platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md#models-by-the-token-offers-prices-credits-and-provider-preferences)
covers the full set, the pay-as-you-go plan, per-key spend limits and how operators list models.
## 4. Report the outcome
The learners (Bradley-Terry, IRT, LinUCB) improve on your own traffic when you tell the platform how
an answer turned out (every plan, the free one included):
```bash
curl -s https://api.opensmartroute.ai/api/v1/feedback \
-H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"request_id": "7f3c...", "target_id": "llm-frontier", "success": true, "quality": 0.9}'
```
The dashboard has the same form without curl: expand a row on the activity page and report good or
poor, a quality score and, optionally, the target that would have done better.
## FAQ
::::accordions
:::accordion[What does the free plan include?]
Routing decisions (`POST /api/v1/route`) with the full trace, the OpenAI-compatible endpoint, outcome
feedback and your own provider keys, metered against a monthly quota of 15,000 requests (30 per minute,
2 API keys, 5 seats). Plans (`plan` / `execute` for agents, skills and tools), a bigger quota with overage
instead of a stop, tenants, statistics, audit and SSO come with the paid tiers - see the pricing page or
`GET /api/v1/info`.
:::
:::accordion[Do I bring my own provider keys?]
Yes, on every plan. The platform routes and meters; you connect your own provider credentials under
*Model providers* so the answer is billed to you by the provider and never leaves your account boundary.
:::
:::accordion[Can I pay for models by the token instead?]
Yes. The **pay as you go** plan has no monthly fee and no request quota: buy credits on the billing page and
call any model on the [storefront](/models?view=offers) at the price shown - `model: "openai/gpt-4.1"`
routes among the providers serving it, `"openai/gpt-4.1@azure"` pins one. Requests stop with `402` when the
credits run out; per-key spend limits cap what one integration may charge.
:::
:::accordion[How do I see why a target was chosen?]
Every `POST /api/v1/route` response carries `signals`, `alternatives` and an `explanation`, and
`policy_rejections` lists any target a hard constraint removed. The activity page shows the same
trace per request.
:::
:::accordion[Can I self-host instead?]
Yes. The same router ships as a zero-dependency library and as `osr serve`. Start with the
[Python SDK quickstart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_SDK.md) or [Deploy with Docker and Helm](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md).
:::
::::
## Where next
- [Platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md) - authentication, organizations, tenants, budgets, traces, errors.
- [MCP server and cost estimates](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MCP.md) - put the router inside VS Code, Cursor or Claude.
- [Marketplace](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MARKETPLACE.md) - shared agents, skills, personas, prompts and stack templates.
- [Python SDK quickstart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_SDK.md) - the same router as a zero-dependency library.
---
# Python SDK quickstart
Route between models, agents, skills, tools and humans inside your own process in about five
minutes. The core package has zero runtime dependencies; everything here runs offline. The
[user guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md) continues where this page stops - real providers, plans, learning, the
enterprise builder.
:::tip[Everything here runs offline]
No API keys and no network calls are needed. Sections 2 to 5 build on each other - paste them into
one file in order and run it - the router is deterministic, so you get the same decisions shown here.
:::
## 1. Install
::::tabs
:::tab[pip]
```bash
pip install opensmartroute
```
The core package has **zero runtime dependencies**.
:::
:::tab[Extras]
```bash
pip install 'opensmartroute[yaml]' # + YAML catalogues and rules
pip install 'opensmartroute[server]' # + FastAPI server and OpenAI-compatible proxy
pip install 'opensmartroute[fast]' # + numpy-accelerated SLM training
pip install 'opensmartroute[all]' # every extra (yaml, server, fast, embeddings, tiktoken, otel, crypto, dev)
```
`osr doctor` tells you what is installed and what a missing extra needs.
:::
:::tab[CLI installer]
```bash
curl -LsSf https://opensmartroute.ai/install.sh | sh
```
On Windows PowerShell:
```powershell
irm https://opensmartroute.ai/install.ps1 | iex
```
Installs the `osr` command line together with the package.
:::
::::
## 2. Describe your targets
A target is anything a request can be sent to. Declare what each one is good at, what it costs and
how fast it is; the router does the rest.
```python
from opensmartroute import Router, TargetRegistry, RouteTarget, TargetKind, Capabilities, Outcome, TargetConstraints
registry = TargetRegistry(
[
RouteTarget(
"llm-small",
TargetKind.LLM,
capabilities=Capabilities(max_complexity=0.45),
cost={"usd_per_1k_tokens": 0.0002},
latency_ms=300,
quality_prior=0.55,
examples=["Hi, how are you?", "What is the capital of France?"],
),
RouteTarget(
"llm-frontier",
TargetKind.LLM,
capabilities=Capabilities(min_complexity=0.5, domains=["math", "coding"]),
cost={"usd_per_1k_tokens": 0.015},
latency_ms=2500,
quality_prior=0.93,
examples=["Prove the theorem step by step."],
),
RouteTarget(
"llm-onprem",
TargetKind.LLM,
capabilities=Capabilities(domains=["general"]),
constraints=TargetConstraints(regions=["eu"], data_boundary="private", pii_allowed=True),
cost={"usd_per_1k_tokens": 0.001},
latency_ms=1200,
quality_prior=0.7,
),
RouteTarget(
"human",
TargetKind.HUMAN,
capabilities=Capabilities(actions=["escalate"], tags=["safety"]),
cost={"usd_per_1k_tokens": 0.5},
latency_ms=300_000,
),
]
)
```
Catalogues also load from YAML or JSON files, MCP `tools/list` payloads, A2A agent cards, SKILL.md
and persona directories - see the [user guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#1-targets). With a `targets.yaml` (and optionally a
`rules.yaml`) on disk - the repository ships a pair under `examples/` - the whole setup is one line (YAML files
need `pip install 'opensmartroute[yaml]'`, JSON needs nothing):
```python
from opensmartroute import Router
router = Router.from_files("targets.yaml", "rules.yaml", state_dir=".osr-state") # state_dir keeps what it learns
```
## 3. Route
```python
router = Router(registry)
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, f"{d.confidence:.2f}") # llm-frontier 0.89
print(d.trace.explain()) # per-strategy scores and rationales
```
Every decision carries a trace: which signals were extracted, which targets policy rejected and why,
how each strategy scored the rest. Nothing is a black box.
:::note[The trace is the audit trail]
`d.trace.explain()` prints the whole decision - signals, policy rejections and per-strategy scores.
Keep it in your logs and every route is reproducible and defensible after the fact.
:::
Every `route()` call runs the same five stages - the trace records each one:
```mermaid
flowchart TB
R["router.route(request)"] --> SIG["Signals: complexity, domains, intent, PII"]
SIG --> POL{"Policy: hard constraints + rules"}
POL -->|rejected| REJ["trace.policy_rejections"]
POL -->|admissible| STR["Strategies score each candidate"]
STR --> ENS["Ensemble to utility, weighted by the objective"]
ENS --> DEC["Decision: target, confidence, alternatives, trace"]
```
## 4. Constraints and objectives
Hard constraints are filtered before any scoring - they are never traded off against quality. The
objective sets the trade-off between quality, cost and latency per request. A note with private
patient data only clears the on-prem target; the public models are excluded before scoring begins.
:::warning[Constraints are hard, objectives are soft]
A `RequestConstraints` violation removes a target from consideration entirely - it is never bought
back by a strong quality score. `Objective` weights only rank the targets that already comply.
:::
```python
from opensmartroute import RouteRequest, RequestConstraints, Objective
req = RouteRequest(
"Summarize this patient intake note.",
constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005),
objective=Objective(quality=1.0, cost=0.5, latency=0.1),
)
d = router.route(req)
print(d.target.id) # llm-onprem
print(d.trace.policy_rejections) # {'llm-small': 'data boundary public < required private', ...}
```
## 5. Close the loop
Report how the routed answer went and the router's learners (bandits, IRT, Bradley-Terry) shift
future decisions toward what actually works for your traffic:
```python
router.learn(
Outcome(
request_id=d.request_id,
target_id=d.target.id,
success=True,
quality=0.9,
cost_usd=0.002,
latency_ms=1800,
domains=d.trace.signals.domains,
)
)
```
Attach handlers to targets and `router.run(req)` executes the chosen target or plan and records the
outcomes itself - wiring real providers is the
[user guide's fourth chapter](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#4-real-providers).
## 6. The same thing from the command line
Without `--targets` the `route`, `estimate`, `serve`, `targets` and `mcp` commands use a built-in starter
catalogue (`llm-small`, `llm-frontier`, `llm-onprem`, with two rules), so the first command works straight after
the install; `osr stack init --out stack.json` writes that catalogue to disk to edit, and `-t stack.json -r stack.json`
routes through your copy. The repository's `examples/` folder holds the richer YAML pair used below.
```bash
osr route "Prove that sqrt(2) is irrational" # built-in starter catalogue, no files needed
osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan
osr -t examples/targets.yaml -r examples/rules.yaml estimate "Summarise this contract" # tokens, cost, quality per candidate
osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier
export OSR_LLM_BASE_URL=https://api.openai.com/v1 OSR_LLM_API_KEY=sk-... # the provider `serve` executes on
osr -t examples/targets.yaml -r examples/rules.yaml serve # http://127.0.0.1:8000/docs
osr -t examples/targets.yaml mcp # MCP server for your IDE, marketplace tools included
```
Routing needs no provider. Execution does: `OSR_LLM_BASE_URL` names any OpenAI-compatible endpoint
(OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter, ...) and `OSR_LLM_API_KEY` its key; without them the
served `/v1/chat/completions` answers 503 because no target has a handler, and routing still works.
`osr serve` exposes the same `/route`, `/feedback` and `/mcp` endpoints as the hosted platform and the same
`/v1` surface: OpenAI `chat/completions` (streaming, tool calling), `responses`, `embeddings`, `moderations`,
`audio/speech`, `audio/transcriptions`, `images/generations` and the Anthropic `messages` API - point the
official OpenAI or Anthropic SDK at it with `model="osr/auto"` (or `"auto"`), an MCP client at `/mcp`, or the
TypeScript client (`npm install @opensmartroute/sdk`, `apiPrefix: ""`). Every model call is compacted first
(repeated lines, stray whitespace and stale history go; the saving is reported as `tokens_saved`),
`--marketplace-refresh 86400` keeps the served catalogue in step with the marketplace, and
`--guard --redact-pii --output-guard redact --health --audit audit.jsonl` turns on the hardening the
[deploy guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md) describes.
## FAQ
::::accordions
:::accordion[Do I need API keys or a network connection?]
No. The core package is offline and deterministic - it decides *where* a request should go. You
only add provider credentials when you attach real handlers and call `router.run()`.
:::
:::accordion[How does the router pick a target?]
It extracts signals (complexity, domains, intent, PII), filters targets that fail the hard
constraints, scores the rest with an ensemble of strategies, and combines the scores into a utility
weighted by your `Objective`. `d.trace.explain()` shows every step.
:::
:::accordion[Can it route to more than models?]
Yes. A `RouteTarget` can be an LLM, an agent, a skill, a tool or a human - the `TargetKind`. The
same scoring picks the best target of any kind, and `RequestConstraints(allowed_kinds=...)` limits
it when you need to.
:::
:::accordion[Does it learn from outcomes?]
Yes. `router.learn(Outcome(...))` feeds the bandit, IRT and Bradley-Terry learners so future
decisions shift toward what actually works on your traffic. It is optional and off until you call it.
:::
::::
## Where next
- [User guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md) - targets, rules, providers, plans, learning, the CLI, latency numbers.
- [SDK guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md) - decorators, settings, error handling, threading, extension points.
- [Enterprise builder](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ENTERPRISE.md) - middleware, telemetry, stores, shadow and A/B rollout.
- [Deploy with Docker and Helm](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md) - run `osr serve` on your own infrastructure.
- [TypeScript SDK](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/sdk-typescript.md) - call `osr serve` or the platform from Node, Deno, Bun, edge and browsers.
- [Platform quickstart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_PLATFORM.md) - the hosted version of everything above.
---
# Install the platform
The **hosted platform** at [opensmartroute.ai](https://opensmartroute.ai) needs no installation - sign up, create
a key, follow the [platform quickstart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_PLATFORM.md). This page is for running the **whole platform
yourself**: the routing API with its OpenAI-compatible endpoint, the web app (public site, dashboard, operator
console), the routing SLM service and the state behind them. The same images and the same `OSR_PLATFORM_*`
settings run everywhere; only the way they are started differs.
| Where | How | For |
|---|---|---|
| Azure, from the portal | [Azure Marketplace](#azure-marketplace) - the solution template with a wizard, one resource group, no tooling (the same template deploys from the published package until the listing is live) | Teams that want the platform in their own subscription |
| Your own servers or any cloud VM | [Docker Compose](#your-own-servers-docker-compose) - PostgreSQL, Redis, Kafka and the platform as containers, from the published Compose bundle | On-premises, air-gapped, other clouds |
| Kubernetes | [`osr serve` Helm chart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md) for the router alone; the platform on Kubernetes runs the same containers as Compose | Sidecar / control-plane routing without the hosted-platform surfaces |
Every file named on this page - the Compose bundle, the Helm chart, the Marketplace package and its two files, the
checksums of the PyPI release - is on the **[downloads page](https://opensmartroute.ai/downloads)** with its version
and SHA-256; the stable links `https://opensmartroute.ai/download/`
always resolve to the current release and `https://opensmartroute.ai/download/latest.json` lists them for scripts. The
source repository is private; nothing here needs it.
Just the Python library or the `osr` command line: [README - Install](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/readme.md#install). Calling the
platform from TypeScript or JavaScript: `npm install @opensmartroute/sdk` ([TypeScript SDK](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/sdk-typescript.md)).
## What you get
Every install path deploys the same product:
- `osr-platform-api` - the FastAPI platform: `POST /api/v1/route`, the OpenAI-compatible `/v1` proxy, MCP server,
accounts, keys, workspaces, tenants, governance, marketplace, billing, notifications, support tickets and the
operator API. In cluster mode it also runs once per domain (docs, rankings, marketplace, providers, onboarding,
accounts, billing, admin, mcp, openai, routing) behind the API gateway, plus the routing SLM service.
- `osr-platform-web` - the Next.js web app: the public site and documentation, `/platform` (sign-in, dashboard,
playground, publish flow) and `/platform/admin` (the operator console).
- State: PostgreSQL (always), Redis and Kafka (cluster mode), a data volume for the audit chain and the routing SLM.
- Model providers: open-weight models through Ollama by default - on Azure the template runs the curated set as its
own container apps (CPU) or one serverless-GPU server and maps every tier and specialist target to it, with Ollama
Cloud for the frontier open weights when you give it a key; Azure OpenAI deployments only when you switch them on
(`azureOpenAiEnabled`). Anything OpenAI-compatible everywhere (connected in the console under *Providers*, or by
workspaces with their own keys under *Model providers*).
Release images: `crosrua2c6onyidguw.azurecr.io/osr-platform-api:2.1.0` and
`crosrua2c6onyidguw.azurecr.io/osr-platform-web:2.1.0` (anonymous pull; `:latest` follows the newest release,
`:` the newest patch). The web image is built without a baked hostname: it reads `OSR_PUBLIC_WEB_URL` at run
time and the browser uses the origin it is talking to, so one image serves any domain.
## Azure Marketplace
The offer **OpenSmartRoute platform** is an Azure Application (solution template): it deploys into a resource group
of your subscription, you own every resource, nothing is shared with the publisher and there is no software charge
on the Azure bill - you pay the Azure resources it creates.
**Listing status.** The offer package is the Partner Center submission; until the listing is live the offer does
not appear in a Marketplace search. The same template deploys today from the published package -
`mainTemplate.json` and `createUiDefinition.json` from the [downloads page](https://opensmartroute.ai/downloads) -
with the same wizard when you use the portal's *Deploy a custom template*, or from the command line:
```bash
curl -fsSLO https://opensmartroute.ai/download/marketplace-template # opensmartroute-mainTemplate-.json
az group create -n rg-osr -l centralindia
az deployment group create -g rg-osr -f opensmartroute-mainTemplate-*.json \
-p adminUsername=admin adminPassword='' platformEdition=enterprise clusterMode=false
```
Every wizard field below is a parameter of that template (`az deployment group what-if` lists them with their
descriptions); the outputs, first
sign-in, custom-domain, upgrade and backup steps are identical.
### Deploy
1. In the Azure portal open *Marketplace*, search for **OpenSmartRoute**, choose the plan and *Create* (or
*Deploy a custom template* -> *Build your own template in the editor* -> *Load file* with the downloaded
`mainTemplate.json`, then *Edit UI definition* with `createUiDefinition.json`).
2. **Basics** - subscription, a new or existing resource group, the region (Container Apps must be offered there -
and Azure OpenAI with the default `gpt-4.1` family when you switch those deployments on), a deployment name, the first **operator username and
password** (the console sign-in), the edition (Enterprise: tenants, audit, statistics, persisted learners;
Community: the lightest footprint).
3. **Size and state** - *Cluster mode* (Redis + Kafka, the API split into domain services scaled on demand; off =
one API replica with PostgreSQL only, right for a trial) and *Managed state* (Azure Database for PostgreSQL with
point-in-time restore, Azure Cache for Redis, Event Hubs - recommended for production; off = state containers on
Azure Files, cheapest).
4. **Hostnames** - leave empty to use the Container Apps hostnames, or enter your web and API hostnames (and let the
template create the Azure DNS zone). TLS certificates are issued in a second run once DNS resolves (below).
5. **E-mail** - Microsoft 365 (an Entra app registration with the application permission `Mail.Send`, sending as
the From mailbox), Azure Communication Services or SMTP for verification, reset, invitation, alert, usage-report
and support-ticket mail; the first transport set wins in that order. Without a transport every message waits in
the console's *Mail* page; nothing is lost.
6. **Models and learning** - the self-hosted model server (Ollama, CPU or serverless GPU; on by default, it answers
every tier), an Ollama Cloud key for the frontier open weights, Azure OpenAI deployments (off by default), the
autopilot (retrain the routing model from your own feedback) and the marketplace catalogue refresh.
7. **Sign-in and access** - password sign-in, self-serve signups per IP (0 = operators create every workspace), SSO
providers, a static admin token for scripts, public `/metrics`.
8. **Images and billing** - the release to deploy and the registry to pull from (yours, after mirroring), Stripe for
self-serve paid plans.
9. *Review + create*. A cluster-mode deployment takes about ten minutes; the first Ollama start pulls about 16 GB of
weights in the background.
The deployment's **Outputs** tab lists `webUrl`, `platformUrl`, `adminLoginUrl`, `apiUrl`, the hostnames and static
IP for your DNS records, the `asuid` verification id and, when enabled, the Azure OpenAI endpoint.
### First sign-in
1. Open `adminLoginUrl` (`/platform/admin/login`) and sign in with the operator username and password from
Basics. The console shows the deployment's health, users, workspaces, plans, providers and the support desk.
2. *Providers*: the Ollama model server the template started is connected and every tier (`llm-small`, `llm-mid`,
`llm-frontier`, `llm-onprem`) and specialist target already executes on it - Qwen3.5 4B on a CPU profile, the
dedicated-node models (GLM-4.7-Flash, gpt-oss-20b) on a dedicated profile, gpt-oss-120b / DeepSeek-V4-Flash
through Ollama Cloud when a key was given. Add OpenAI, Azure OpenAI, Anthropic, Bedrock, Vertex, vLLM or any
OpenAI-compatible endpoint here, or let workspaces bring their own keys.
3. Open `webUrl`, *Sign up* (or create the workspace from the console when self-serve signup is off), create an API
key and route a request - [platform quickstart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_PLATFORM.md) with `OSR_API_URL=`.
4. `osr login --url ` signs the command line in to your deployment.
### Custom domain (two runs)
1. First run with your hostnames entered and *Issue TLS certificates* off: the apps bind the hostnames without TLS.
2. Create the DNS records from the outputs: `CNAME -> webHostname`, `CNAME ->
apiHostname` (an apex domain gets `A -> staticIp`), and `TXT asuid. -> domainVerificationId` for each
hostname. If the template created the zone, delegate the domain to `dnsNameServers` instead - the records are
already there.
3. Run the deployment again (*Redeploy* on the deployment, same values) with *Issue TLS certificates* on. Azure
issues free managed certificates and the hostnames answer over HTTPS.
### Upgrade, backup, remove
- **Upgrade**: redeploy with a newer *OpenSmartRoute release* (image tag). The database schema migrates itself at
start-up - the same numbered ledger, pre-upgrade backup on the data share (`backups/pre-upgrade-*.jsonl.gz`) and
downgrade rules as under *Operate* below; state lives in PostgreSQL and on the data share, never in the
containers. When an upgrade replaces the
database container itself, the new replica waits for the previous one to shut down before it opens the
database (Container Apps runs both for a moment); expect the API to pause for up to a minute.
- **Backup**: a nightly job writes a logical database backup to the deployment's backup share (14 days kept).
Managed state adds point-in-time restore on the server. Restore: load the newest archive into a fresh database
with `psql`, then restart the API.
- **Remove**: delete the resource group. Nothing is left outside it.
### Sizing and cost (Azure list prices, rough)
| Configuration | Monthly Azure cost | Suits |
|---|---|---|
| Community, cluster mode off, Azure Files state | ~40-70 USD (Container Apps consumption, storage, Log Analytics) + model usage | Trial, a team |
| Enterprise, cluster mode, Azure Files state | ~150-250 USD | Production without SLA requirements |
| Enterprise, cluster mode, managed state (B1ms PostgreSQL, C0 Redis, 1 TU Event Hubs) | ~250-400 USD | Production with backups, SLAs, HA options |
| + Ollama on serverless GPU (T4) | pay per second of GPU while replicas run | Private / specialist models without cloud keys |
Model usage runs on the self-hosted Ollama apps (part of the Container Apps figure; a serverless GPU is billed per
second while replicas run) and, when you connect them, on Ollama Cloud or a closed-model provider at their own
prices; the dashboard's *Savings* page shows what routing saved against always calling the frontier model.
## Your own servers (Docker Compose)
Runs on any host with Docker Engine 24+ (Linux, or Docker Desktop for a workstation): on-premises, air-gapped, or a
VM in any cloud. Minimum 4 vCPU / 8 GiB / 40 GB disk for the platform; add what your models need if you run them on
the same host.
### 1. Get the stack
Download the Compose bundle of the current release - `docker-compose.yml`, `.env.example` and the Ollama overlays in
the layout this page uses - and unpack it:
```bash
curl -fsSLO https://opensmartroute.ai/download/compose # opensmartroute-compose-.zip
unzip opensmartroute-compose-*.zip && cd opensmartroute-*/
cp platform/.env.example platform/.env
```
The bundle's `.env.example` already names the images of its own release; the stack runs from the published images
and needs no source.
### 2. Configure
Edit `platform/.env`: the passwords and the session secret (all of them - the defaults are placeholders), the
public URLs of the web app and the API as your users will reach them, the release images, and the e-mail
transport. Everything is documented in the file; the same variables are explained in
platform/README.md (`platform/README.md`).
### 3. Start
```bash
docker compose -f platform/docker-compose.yml --env-file platform/.env pull
docker compose -f platform/docker-compose.yml --env-file platform/.env run --rm api python -m osr_platform.app --check
docker compose -f platform/docker-compose.yml --env-file platform/.env up -d
curl -fsS http://localhost:8080/readyz # every check ok once PostgreSQL is up and the schema is created
```
The `--check` line is the configuration preflight: it reads `.env` the way the API will and lists what is still a
placeholder or inconsistent - the shipped passwords, a session secret missing on a multi-replica stack, public URLs
still on `localhost`, e-mail confirmation required without a mail transport, a providers file that is not in the
container - each with the variable that fixes it, and exits non-zero on an error. The same findings are logged at
start-up and shown on the console's *Health* page. Web `http://localhost:3000`, operator console
`http://localhost:3000/platform/admin/login` (the operator username and password from `.env`), API
`http://localhost:8080/api/v1/info`. `OSR_API_IMAGE` / `OSR_WEB_IMAGE` in `.env` name the release images; a
checkout of the source can leave them empty to build locally (`up -d --build`).
**Just the router with a dashboard?** `platform/docker-compose.minimal.yml` starts four containers - PostgreSQL,
Redis, one API process serving every domain, the web app - with `OSR_PLATFORM_PROFILE=minimal`: accounts, keys,
routing, the OpenAI-compatible `/v1` proxy, MCP, billing and the console, and none of the jobs a hosted deployment
runs for itself (blog writer, newsletter, growth engine, marketplace harvest). No Kafka, no separate SLM trainer.
The same `.env` applies; switch to `docker-compose.yml` later without migrating anything - both stacks use the
same database schema.
### 4. Connect models
The stack has no cloud account. Either connect providers in the console under *Providers* (any OpenAI-compatible
endpoint: OpenAI, Azure OpenAI, Anthropic through a gateway, vLLM, Ollama, ...) or start the on-prem model server
next to it - the `llm` profile of the same Compose file - and point the router at its providers file:
```bash
# in platform/.env:
# COMPOSE_PROFILES=llm
# OSR_PLATFORM_PROVIDERS=/config/providers.ollama.yaml
docker compose -f platform/docker-compose.yml --env-file platform/.env up -d
# on an NVIDIA host add the GPU overlay:
docker compose -f platform/docker-compose.yml -f deploy/compose/ollama.gpu.yaml --env-file platform/.env up -d
```
The `ollama` service pulls the curated open weights of
deploy/compose/providers.ollama.yaml (`deploy/compose/providers.ollama.yaml`) (Qwen 3.5, Qwen2.5-Coder, GLM-OCR,
TranslateGemma, DeepSeek-R1, a Qwen3 embedding model; `OSR_OLLAMA_MODELS` trims the list on a small machine) and
that file maps the private / specialist targets to it - `deploy/compose` is mounted at `/config` in every platform
container, which is why the path above works. `OSR_PLATFORM_PROVIDERS` points the API at a providers file for
anything else.
### 5. Put it behind TLS
Terminate TLS in your reverse proxy (nginx, Caddy, Traefik, an ingress) in front of ports 3000 (web) and 8080 (API),
set `OSR_PLATFORM_WEB_URL` / `OSR_PLATFORM_BASE_URL` to the public `https://` origins, and remove the host port
mappings you do not want exposed. Several API replicas share PostgreSQL / Redis / Kafka: `docker compose up -d
--scale api=3` behind the proxy, exactly what the Azure template does.
### Operate
- **Backups**: `docker compose -f platform/docker-compose.yml exec postgres pg_dump -U osr osr | gzip >
osr-$(date +%F).sql.gz` (and `osr_telemetry`); restore with `psql` into a fresh volume. Without `pg_dump` at
hand, the API container does it from its own connection: `docker compose ... run --rm api python -m
osr_platform.maintenance backup /data/backups/osr-$(date +%F).jsonl.gz` writes every row of every table as one
gzip file, `... restore --yes` (API stopped) empties the tables and loads it back in one transaction.
- **Upgrade**: set the new tag in `.env`, `pull`, `run --rm api python -m osr_platform.app --check`, `up -d`. The
schema is a numbered ledger (`schema_migrations`): the new release applies the migrations it knows and the
database has not run yet, in order, and records each with the release that wrote it - `python -m
osr_platform.maintenance schema` shows the ledger, applied and pending. Before a pending migration that is not
additive runs on a database that already holds data, the API writes a backup to `/backups/pre-upgrade--.jsonl.gz` on its own (`OSR_PLATFORM_PRE_UPGRADE_BACKUP=false` turns that off, `OSR_PLATFORM_BACKUPS_KEEP`
is how many it keeps); an additive change needs none - the way back from it is the older release itself - and a
fresh install never writes one. The backup never holds the schema lock, so replicas already serving keep
answering while it is written. Skipping releases is fine: the migrations between them
run in sequence at the next start.
- **Downgrade**: set the previous tag and `up -d`. Every migration is marked *additive* (it only added columns,
indexes or tables) or not. An older release starts against a database a newer one migrated when every newer
change is additive - it logs a warning (also shown on the console's *Health* page under *Configuration*) and
ignores the columns it does not know. A non-additive newer change stops start-up with the release that wrote it
named: run that release, or put the database back with `python -m osr_platform.maintenance restore` of the
`pre-upgrade-*` backup the upgrade took (a restore refuses a backup written by a newer schema than the running
release knows; an older one loads and the columns added since take their defaults).
- **Edition switch**: `OSR_PLATFORM_EDITION` is read at start-up and every table exists in both editions, so moving
between `community` and `enterprise` is a restart with the new value and back - nothing to migrate. The database
remembers which edition opened it: the first start after a change logs what the switch means (community writes
no audit log and does not read the persisted learner state, so an enterprise stack that ran as community for a
while has that gap in its audit chain) and the console's *Health* page shows the same sentence under
*Configuration* until the next restart.
- **Air-gapped**: on a connected machine `docker pull` both images (and `postgres:16-alpine`, `redis:7-alpine`,
`apache/kafka:3.9.0`), `docker save | gzip`, move, `docker load`; or mirror them into your registry
(`az acr import --source crosrua2c6onyidguw.azurecr.io/osr-platform-api:2.1.0 ...`) and set `OSR_API_IMAGE` /
`OSR_WEB_IMAGE` (or the Marketplace wizard's *Image registry*).
- **Health**: `GET /readyz` on the API (database, Redis, Kafka, mail transport, routing probe), `/platform/admin/health`
in the console (readiness, traffic, breakers, alerts, the configuration preflight), `GET /metrics` for Prometheus.
## Monitoring
Every install monitors itself: the platform's own telemetry store keeps every span and event in PostgreSQL, `GET
/metrics` serves Prometheus text, `GET /readyz` and `GET /api/v1/status` answer probes and status pages, the console's
*Health* page shows readiness, traffic, latency, circuit breakers, alerts and the configuration preflight. Nothing
else is required.
**On Azure** the template and the Marketplace offer also create a workspace-based **Application Insights** resource
(`appi-osr-`, next to the Log Analytics workspace that already receives the container console logs; the
*Application Insights* checkbox of the wizard, `OSR_APPLICATION_INSIGHTS=false` in the azd environment turns it off)
and hand every platform container its connection string. Every process then exports requests, routing spans
(`dependencies`), routing events (`customEvents`), warnings and exceptions under the request's operation id, so the
end-to-end transaction view, the application map, KQL, workbooks and Azure Monitor alerts work out of the box. The
export is `OSR_PLATFORM_APPINSIGHTS=auto`: it only runs where a connection string is present *and* the host is Azure.
**On your own servers** nothing is exported anywhere. To report a self-hosted stack into an Azure subscription
anyway, create an Application Insights resource there, put its connection string in
`OSR_PLATFORM_APPINSIGHTS_CONNECTION_STRING` and set `OSR_PLATFORM_APPINSIGHTS=on` in `.env`;
`OSR_PLATFORM_APPINSIGHTS_SAMPLE_RATE` bounds the ingestion volume. The settings are described in
[docs/PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md#monitoring-what-is-always-on-and-application-insights-on-azure).
## Where the downloads come from
Every release publishes the Compose bundle, the Helm chart, the Marketplace package and the checksums of the PyPI
build to the [downloads page](https://opensmartroute.ai/downloads), together with a `latest.json` manifest (version,
build time, per-file name, size and SHA-256) for scripts. Versioned files are immutable; `latest.json` always points
at the current release. The Azure Marketplace offer is the same solution template, published through Partner Center;
a plan's image tag always names release images that already exist, so a deployment never pulls a moving target.
---
# Real-world use cases by industry
Nine industry scenarios, each a complete routing catalogue you can copy into a file and run as it stands - no
code to write: a hospital, a bank, a law firm, an online shop, an engineering team, an HR department, a
marketing team, a hotel group and an analytics team. Every scenario shows the requests real users send, the
target each one is routed to and the reason, and every one of those decisions is checked by the test suite on
every commit.
:::tip[Every decision on this page is verified, not illustrated]
The test suite reads this very page: it writes each YAML block to a file, sends every request of every
*What happens* table through the router (in process with the SDK's default objective and the command line's
default, through `osr route`, and over HTTP through the server `osr serve` starts) and asserts the *Routed to*
column; it also runs every `osr` command in every *Try it* block and checks its `# -> target` annotation.
The same tables are repeated through the hosted platform's `POST /api/v1/route` (guard middleware, workspace
policy and metering included) and, on a schedule, against the running production
platform. If any surface ever decided differently, the change could not ship. Nothing here
needs an API key or a network connection.
:::
## How to run a scenario
1. Install the command line: `pip install 'opensmartroute[yaml]'`.
2. Save a scenario's YAML block under the file name in its first line (for example `healthcare.yaml`). The one
file carries both the `targets:` and the `rules:`.
3. Route a request with the file:
```bash
osr -t healthcare.yaml -r healthcare.yaml route "What time does the clinic open on Saturdays?" # -> front-desk-fast
```
The command prints the chosen target, its confidence and the whole trace - the signals it read, the targets policy
refused and why, and every strategy's score. Add `--json` for the raw decision, `--plan` for a persona / skill /
model plan, `--event ` to route a system event without any text and `--constraint key=value` for a hard
constraint.
The same file serves the same decisions over HTTP:
```bash
osr -t healthcare.yaml -r healthcare.yaml serve # http://127.0.0.1:8000
curl -s http://127.0.0.1:8000/route -H 'Content-Type: application/json' \
-d '{"text": "What time does the clinic open on Saturdays?"}'
curl -s http://127.0.0.1:8000/route -H 'Content-Type: application/json' \
-d '{"text": "", "context": {"event": "payment.failed"}}'
```
and the hosted platform takes the same JSON at `POST /api/v1/route` with your workspace key
([platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md#2-route-a-request)):
```bash
curl -s https://api.opensmartroute.ai/api/v1/route -H "Authorization: Bearer $KEY" \
-H 'Content-Type: application/json' -d '{"text": "I want a refund for order #4821, it arrived damaged."}'
```
Each scenario follows the same shape:
| Part | What it tells you |
|---|---|
| **Situation** | Who the users are, what they ask and what must never happen |
| **Catalogue** | The targets (models, agents, skills, tools, workflows, people) and the rules, as one YAML file |
| **What happens** | Real requests, the target each one reaches, and why - the verified table |
| **Try it** | The commands to run, each annotated with the target it reaches - also verified |
Three mechanisms recur in every industry:
- **Data that must stay inside stays inside.** A target declares `pii_allowed: false` and the policy stage removes
it the moment a request carries an e-mail address, phone number, IBAN, card or national id. This is a hard stop,
never a score - a rule then *prefers* the on-premises target among what is left.
- **People are one target among the others.** A `human` target with `actions: [escalate]` and a pinned rule means
"I want to speak to a manager" or a `payment.failed` event reaches a person, not a model.
- **Events route without any text.** `when: { event: order.* }` matches the event name, so webhooks and queue
messages are routed the same way chat is (`--event order.shipped` on the command line, `context.event` over
HTTP).
## 1. Healthcare - a hospital group
**Situation.** Front-desk staff and patients ask practical questions all day (opening hours, preparation for
tests). Clinicians paste discharge notes and intake forms that name patients. Complex medication questions
deserve the best model available. Anything with patient identifiers must never leave the hospital network,
and an emergency has to reach the on-call clinician, not a chatbot.
**Catalogue.**
```yaml
# healthcare.yaml
targets:
- id: clinic-notes-onprem
kind: llm
name: Clinic model (inside the hospital network)
capabilities:
domains: [medical, general]
actions: [summarize, extract, qa, classify, generation]
max_complexity: 0.7
constraints: { data_boundary: on_prem, pii_allowed: true, regions: [eu] }
cost: { usd_per_1k_tokens: 0.001 }
latency_ms: 1200
quality_prior: 0.72
examples:
- "Summarize this discharge note."
- "Extract the medications and dosages from this intake form."
- id: front-desk-fast
kind: llm
name: Front-desk assistant (public cloud, cheap)
capabilities:
domains: [general, general_chat, customer_support]
actions: [qa, classify, summarize, generation]
max_complexity: 0.45
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.0002 }
latency_ms: 300
quality_prior: 0.55
examples:
- "What time does the clinic open on Saturdays?"
- "Do I need to fast before a blood test?"
- id: clinical-reasoner
kind: llm
name: Clinical reasoning model (public cloud, frontier)
capabilities:
domains: [medical, general]
actions: [reasoning, qa, summarize, generation]
min_complexity: 0.45
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.015 }
latency_ms: 2500
quality_prior: 0.93
examples:
- "Explain the interactions between these three medications and how to adjust the dosage."
- id: on-call-clinician
kind: human
name: On-call clinician
capabilities: { actions: [escalate], tags: [safety] }
cost: { usd_per_1k_tokens: 0.5 }
latency_ms: 300000
rules:
- name: patient-data-stays-inside
when: { contains_pii: true }
prefer: [clinic-notes-onprem]
weight: 1.0
- name: clinical-emergency-to-human
when: { actions: [escalate] }
prefer: [on-call-clinician]
pin: true
weight: 1.0
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Summarize this discharge note for patient John Doe, DOB 03/04/1961, phone +44 7700 900123: admitted with pneumonia, treated with IV antibiotics, discharged on oral amoxicillin.` | `clinic-notes-onprem` | The phone number is PII: both public-cloud models are rejected by policy before any scoring; the rule prefers the on-premises model |
| `Explain step by step how metformin dosage should be adjusted for a patient with reduced kidney function and why.` | `clinical-reasoner` | Medical domain, reasoning intent, complexity above the frontier model's floor; no identifiers, so the public model is allowed |
| `What time does the clinic open on Saturdays?` | `front-desk-fast` | A simple question: the cheapest capable model wins on utility |
| `The patient reports acute chest pain right now - escalate to the on-call doctor.` | `on-call-clinician` | The escalation intent triggers the pinned rule; every model is excluded |
**Try it.** A department can insist on the boundary even when a note happens to contain no identifier; the trace
names every target that was refused and why (`data boundary public < required on_prem`):
```bash
osr -t healthcare.yaml -r healthcare.yaml route "Summarize this discharge note for patient John Doe, phone +44 7700 900123." # -> clinic-notes-onprem
osr -t healthcare.yaml -r healthcare.yaml route "Summarize this patient intake note." --constraint data_boundary=on_prem --constraint region=eu # -> clinic-notes-onprem
osr -t healthcare.yaml -r healthcare.yaml route "The patient reports acute chest pain right now - escalate to the on-call doctor." # -> on-call-clinician
```
## 2. Banking and financial services
**Situation.** Customers ask about products in the chat widget. Back-office teams extract fields from invoices and
loan files that contain account numbers. Analysts want deep comparisons. A failed payment or a suspected fraud is
an event from the core banking system, not a sentence, and it must land on the fraud desk.
**Catalogue.**
```yaml
# banking.yaml
targets:
- id: ledger-onprem
kind: llm
name: Back-office model (bank data centre)
capabilities:
domains: [finance, customer_support, general]
actions: [extract, summarize, classify, qa]
max_complexity: 0.7
constraints: { data_boundary: on_prem, pii_allowed: true, regions: [eu] }
cost: { usd_per_1k_tokens: 0.001 }
latency_ms: 1200
quality_prior: 0.72
examples:
- "Extract the IBAN, amount and due date from this invoice."
- id: banker-fast
kind: llm
name: Customer chat model (cheap)
capabilities:
domains: [general, general_chat, customer_support, finance]
actions: [qa, classify, generation, summarize]
max_complexity: 0.45
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.0002 }
latency_ms: 300
quality_prior: 0.55
examples:
- "What is the interest rate on the savings account?"
- id: analyst-frontier
kind: llm
name: Analyst model (frontier)
capabilities:
domains: [finance, math, data_analysis, general]
actions: [reasoning, planning, generation, qa]
min_complexity: 0.45
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.015 }
latency_ms: 2500
quality_prior: 0.93
examples:
- "Compare the risk-adjusted returns of two portfolios and explain the trade-offs."
- id: fraud-desk
kind: human
name: Fraud desk
capabilities: { actions: [escalate], tags: [safety] }
cost: { usd_per_1k_tokens: 0.5 }
latency_ms: 600000
rules:
- name: account-data-stays-inside
when: { contains_pii: true }
prefer: [ledger-onprem]
weight: 1.0
- name: failed-payments-to-fraud-desk
when: { event: ["payment.failed", "fraud.*"] }
prefer: [fraud-desk]
pin: true
weight: 1.0
- name: escalation
when: { actions: [escalate] }
prefer: [fraud-desk]
pin: true
weight: 1.0
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Extract the IBAN and amount from this invoice: IBAN DE89370400440532013000, EUR 4,200 due in 30 days.` | `ledger-onprem` | An IBAN is PII: the public models are rejected, the rule prefers the data-centre model |
| `Compare the risk-adjusted return of a 60/40 portfolio against an all-equity portfolio over 30 years and explain the trade-offs step by step.` | `analyst-frontier` | Finance and mathematics, reasoning intent, high complexity: the frontier model is worth its price |
| `What is the interest rate on the savings account?` | `banker-fast` | Simple product question: cheapest capable model |
| event `payment.failed` | `fraud-desk` | No text at all: the event name carries the intent and the pinned rule sends it to people |
**Try it.** A webhook handler passes the event name through without composing a prompt; a region-bound team adds
the boundary as a constraint; `estimate` quotes a request before it is sent (tokens, cost, latency and quality for
every candidate, the recommended pick and the cheapest, fastest and best-quality alternatives):
```bash
osr -t banking.yaml -r banking.yaml route --event payment.failed # -> fraud-desk
osr -t banking.yaml -r banking.yaml route "Summarize this loan file." --constraint region=eu --constraint data_boundary=on_prem # -> ledger-onprem
osr -t banking.yaml -r banking.yaml estimate "Compare the risk-adjusted return of a 60/40 portfolio against an all-equity portfolio over 30 years."
```
## 3. Legal - a law firm
**Situation.** Associates summarise NDAs a dozen times a day - a cheap model does that well. Partners want a
careful analysis of liability clauses from the strongest model, framed by the firm's own counsel persona. Any
document naming a client stays on the firm's servers.
**Catalogue.**
```yaml
# legal.yaml
targets:
- id: counsel-frontier
kind: llm
name: Counsel model (frontier)
capabilities:
domains: [legal, general]
actions: [reasoning, summarize, qa, generation]
min_complexity: 0.4
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.015 }
latency_ms: 2500
quality_prior: 0.92
examples:
- "Review this indemnification clause and explain our liability exposure."
- id: paralegal-fast
kind: llm
name: Paralegal model (cheap)
capabilities:
domains: [legal, general]
actions: [summarize, qa, classify, extract]
max_complexity: 0.5
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.0004 }
latency_ms: 400
quality_prior: 0.6
examples:
- "Summarize the key points of this NDA in three bullets."
- id: matter-files-onprem
kind: llm
name: Matter-files model (firm servers)
capabilities:
domains: [legal, general]
actions: [summarize, extract, qa, generation]
max_complexity: 0.7
constraints: { data_boundary: on_prem, pii_allowed: true }
cost: { usd_per_1k_tokens: 0.001 }
latency_ms: 1200
quality_prior: 0.7
- id: persona-inhouse-counsel
kind: persona
primary: false
capabilities: { domains: [legal], actions: [reasoning, summarize, qa] }
instructions: "You are cautious in-house counsel. Cite the clause, flag jurisdiction risk, never give a final legal opinion."
rules:
- name: client-data-stays-inside
when: { contains_pii: true }
prefer: [matter-files-onprem]
weight: 1.0
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Review this indemnification clause and explain our liability exposure under GDPR if a sub-processor breaches - compare the two drafting options step by step.` | `counsel-frontier` | Legal reasoning at high complexity: the frontier model, above the cheap model's ceiling |
| `Summarize the key points of this NDA in three bullets.` | `paralegal-fast` | A routine summary below the frontier model's floor: the cheap model wins |
| `Draft a contract amendment for client Maria Rossi (maria.rossi@example.com) extending the term by 12 months.` | `matter-files-onprem` | The e-mail address is PII: only the firm's own model is admissible |
**Try it.** Ask for a plan and the persona is layered on whichever model wins (the output shows
`plan: persona:persona-inhouse-counsel -> llm:counsel-frontier`); its `instructions` become the system prompt when
the plan is executed:
```bash
osr -t legal.yaml -r legal.yaml route "Review this indemnification clause and explain our liability exposure under GDPR if a sub-processor breaches - compare the two drafting options step by step." --plan # -> counsel-frontier
osr -t legal.yaml -r legal.yaml route "Draft a contract amendment for client Maria Rossi (maria.rossi@example.com)." # -> matter-files-onprem
```
## 4. Retail and e-commerce customer support
**Situation.** Shoppers ask about shipping and sizes, report damaged orders and demand refunds. Order events from
the fulfilment system trigger customer notifications. A customer who asks for a manager gets one. The support
agent has tools (order lookup, refund) and runs inside the company, so it may see customer details.
**Catalogue.**
```yaml
# retail.yaml
targets:
- id: support-agent
kind: agent
name: Support agent (order lookup, refunds)
capabilities:
domains: [customer_support]
actions: [qa, action, classify, summarize]
supports_tools: true
tags: [autonomous]
constraints: { data_boundary: private, pii_allowed: true }
cost: { usd_per_1k_tokens: 0.004 }
latency_ms: 3000
quality_prior: 0.8
examples:
- "I want a refund for order #4821, it arrived damaged."
- "Where is my order? It was due yesterday."
- id: shop-chat-fast
kind: llm
name: Shop assistant (cheap)
capabilities:
domains: [general, general_chat]
actions: [qa, generation, classify]
max_complexity: 0.45
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.0002 }
latency_ms: 300
quality_prior: 0.55
examples:
- "Do you ship to Canada?"
- "What sizes does the jacket come in?"
- id: workflow-notify-customer
kind: workflow
name: Customer notification workflow
capabilities: { domains: [customer_support], actions: [action] }
cost: { usd_per_1k_tokens: 0.0001 }
latency_ms: 200
quality_prior: 0.7
- id: support-team
kind: human
name: Support team
capabilities: { actions: [escalate], tags: [safety] }
cost: { usd_per_1k_tokens: 0.5 }
latency_ms: 900000
rules:
- name: support-questions-to-agent
when: { domains: [customer_support] }
prefer: [support-agent]
weight: 0.8
- name: order-events-to-notification-workflow
when: { event: ["order.*", "shipment.*"] }
prefer: [workflow-notify-customer]
pin: true
weight: 1.0
- name: angry-customers-to-people
when: { actions: [escalate] }
prefer: [support-team]
pin: true
weight: 1.0
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `I want a refund for order #4821, it arrived damaged.` | `support-agent` | Customer-support domain and an action to perform: the rule prefers the agent with tools |
| `Do you ship to Canada?` | `shop-chat-fast` | A general question: the cheap model answers |
| event `order.shipped` | `workflow-notify-customer` | The fulfilment event is pinned to the notification workflow - no prompt involved |
| `This is the third time I am asking. I want to speak to a manager now.` | `support-team` | Escalation intent: pinned to people |
**Try it.** The fulfilment system sends its events as they happen; `--context` carries any extra facts along:
```bash
osr -t retail.yaml -r retail.yaml route --event order.shipped --context order_id=4821 # -> workflow-notify-customer
osr -t retail.yaml -r retail.yaml route "I want a refund for order #4821, it arrived damaged." # -> support-agent
osr -t retail.yaml -r retail.yaml route "This is the third time I am asking. I want to speak to a manager now." # -> support-team
```
Close the loop without code as well: when the agent has resolved the refund, report the outcome to the served
router (`POST /feedback`) or the platform (`POST /api/v1/feedback`) with the decision's `request_id` and
`target_id`, and the learners shift future support traffic toward what actually works:
```bash
curl -s http://127.0.0.1:8000/feedback -H 'Content-Type: application/json' \
-d '{"request_id": "", "target_id": "support-agent", "success": true, "quality": 0.9}'
```
## 5. Software engineering
**Situation.** Developers ask quick language questions, request SQL, and hand over multi-step repository work
(refactor, run tests, open a pull request). CI and deployment failures arrive as events and should go straight to
the coding agent.
**Catalogue.**
```yaml
# engineering.yaml
targets:
- id: coding-agent
kind: agent
name: Coding agent (repository access, tests, pull requests)
capabilities:
domains: [coding]
actions: [code_generation, code_review, action, reasoning]
min_complexity: 0.4
supports_tools: true
tags: [autonomous]
cost: { usd_per_1k_tokens: 0.008 }
latency_ms: 6000
quality_prior: 0.85
examples:
- "Refactor this module to remove the circular import and run the test suite."
- "Investigate why the deploy failed and open a pull request with the fix."
- id: code-fast
kind: llm
name: Code model (cheap)
capabilities:
domains: [coding, general]
actions: [qa, code_generation, summarize, reasoning]
max_complexity: 0.5
cost: { usd_per_1k_tokens: 0.0004 }
latency_ms: 400
quality_prior: 0.6
examples:
- "What does the Python zip function do?"
- "Write a one-line list comprehension that squares a list."
- id: skill-sql
kind: skill
name: SQL skill
capabilities: { domains: [data_analysis, coding], actions: [code_generation, extract] }
cost: { usd_per_1k_tokens: 0.001 }
latency_ms: 200
quality_prior: 0.8
instructions: "Always qualify table names with the schema; never SELECT *."
examples:
- "Write a SQL query that returns monthly active users per region."
rules:
- name: sql-to-skill
when: { domains: [data_analysis], actions: [code_generation] }
prefer: [skill-sql]
pin: true
weight: 1.0
- name: broken-builds-to-agent
when: { event: ["deploy.failed", "build.failed", "test.failed"] }
prefer: [coding-agent]
pin: true
weight: 1.0
- name: repo-work-to-agent
when: { domains: [coding], actions: [action, code_review] }
prefer: [coding-agent]
weight: 0.9
- name: hard-coding-to-agent
when: { domains: [coding], min_complexity: 0.4 }
prefer: [coding-agent]
weight: 0.7
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Refactor the payment module to remove the circular import, run the test suite, fix the failures and open a pull request.` | `coding-agent` | Repository work with actions to perform: the agent with tools |
| `Review this pull request and find the bug in the retry loop.` | `coding-agent` | Code review is agent work in this catalogue |
| `What does the Python zip function do?` | `code-fast` | A quick language question: the cheap model |
| `Write a SQL query that returns monthly active users per region from the events table.` | `skill-sql` | Data-analysis code generation is pinned to the SQL skill |
| event `deploy.failed` | `coding-agent` | The failure event is pinned to the agent |
**Try it.**
```bash
osr -t engineering.yaml -r engineering.yaml route --event deploy.failed --context service=checkout # -> coding-agent
osr -t engineering.yaml -r engineering.yaml route "What does the Python zip function do?" # -> code-fast
osr -t engineering.yaml -r engineering.yaml route "Write a SQL query that returns monthly active users per region from the events table." # -> skill-sql
```
## 6. Human resources
**Situation.** Employees ask about policies. Recruiters screen CVs full of personal data. Managers draft
performance reviews. Sensitive cases go to a human business partner.
**Catalogue.**
```yaml
# hr.yaml
targets:
- id: people-onprem
kind: llm
name: People-data model (inside the company)
capabilities:
domains: [hr, general]
actions: [extract, classify, summarize, qa, generation]
max_complexity: 0.7
constraints: { data_boundary: on_prem, pii_allowed: true }
cost: { usd_per_1k_tokens: 0.001 }
latency_ms: 1200
quality_prior: 0.72
examples:
- "Screen this resume against the job description."
- id: hr-helpdesk-fast
kind: llm
name: HR helpdesk model (cheap)
capabilities:
domains: [hr, general, general_chat]
actions: [qa, classify, summarize]
max_complexity: 0.45
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.0002 }
latency_ms: 300
quality_prior: 0.55
examples:
- "How many days of parental leave does the leave policy give?"
- id: hr-writer
kind: llm
name: HR writing model
capabilities:
domains: [hr, general]
actions: [generation, summarize, reasoning]
min_complexity: 0.3
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.003 }
latency_ms: 900
quality_prior: 0.78
examples:
- "Draft a constructive performance review for an engineer."
- id: hr-business-partner
kind: human
name: HR business partner
capabilities: { actions: [escalate], tags: [safety] }
cost: { usd_per_1k_tokens: 0.5 }
latency_ms: 3600000
rules:
- name: candidate-and-employee-data-stays-inside
when: { contains_pii: true }
prefer: [people-onprem]
weight: 1.0
- name: sensitive-cases-to-people
when: { actions: [escalate] }
prefer: [hr-business-partner]
pin: true
weight: 1.0
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Screen this resume for the data engineer role: Priya Natarajan, priya.n@example.com, +91 98765 43210, 6 years of Spark and Airflow.` | `people-onprem` | E-mail and phone number: only the in-house model may see the CV |
| `How many days of parental leave does the leave policy give?` | `hr-helpdesk-fast` | Policy question: the cheap model |
| `Draft a constructive performance review for an engineer who ships fast but skips code reviews, and explain how to phrase the feedback.` | `hr-writer` | Drafting with reasoning at moderate complexity: the writing model |
| `An employee reported harassment on the team - escalate this to the HR business partner.` | `hr-business-partner` | Escalation intent: pinned to a person |
**Try it.**
```bash
osr -t hr.yaml -r hr.yaml route "Screen this resume for the data engineer role: Priya Natarajan, priya.n@example.com, 6 years of Spark." # -> people-onprem
osr -t hr.yaml -r hr.yaml route "How many days of parental leave does the leave policy give?" # -> hr-helpdesk-fast
osr -t hr.yaml -r hr.yaml route "An employee reported harassment on the team - escalate this to the HR business partner." # -> hr-business-partner
```
## 7. Marketing
**Situation.** Copy is written all day and should cost next to nothing - a small self-hosted model does it.
Campaign planning needs a stronger model. Translations go to a dedicated translation skill so terminology
stays consistent.
**Catalogue.**
```yaml
# marketing.yaml
targets:
- id: copywriter-small
kind: llm
name: Copywriter (small, self-hosted)
capabilities:
domains: [marketing, creative, general]
actions: [generation, summarize, translate, qa]
max_complexity: 0.5
languages: ["*"]
constraints: { data_boundary: on_prem, pii_allowed: true }
cost: { usd_per_1k_tokens: 0.00005 }
latency_ms: 700
quality_prior: 0.6
examples:
- "Write three subject lines for our spring newsletter."
- id: strategist-mid
kind: llm
name: Campaign strategist model
capabilities:
domains: [marketing, data_analysis, general]
actions: [planning, reasoning, generation, summarize]
min_complexity: 0.3
cost: { usd_per_1k_tokens: 0.003 }
latency_ms: 900
quality_prior: 0.78
examples:
- "Plan a multi-channel campaign with budget allocation and KPIs per channel."
- id: skill-translate
kind: skill
name: Translation skill
capabilities: { actions: [translate], languages: ["*"] }
cost: { usd_per_1k_tokens: 0.0005 }
latency_ms: 300
quality_prior: 0.85
examples:
- "Translate this headline into Spanish."
rules:
- name: translations-to-skill
when: { actions: [translate] }
prefer: [skill-translate]
pin: true
weight: 1.0
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Write three subject lines for our spring newsletter.` | `copywriter-small` | Simple copy: the near-free self-hosted model |
| `Plan a 12-week multi-channel campaign for a B2B SaaS launch with budget allocation and KPIs per channel, and explain the trade-offs.` | `strategist-mid` | Planning and reasoning above the copywriter's ceiling |
| `Translate this landing page headline into Spanish: 'Route every request to the right model.'` | `skill-translate` | Translation intent: pinned to the skill |
**Try it.**
```bash
osr -t marketing.yaml -r marketing.yaml route "Write three subject lines for our spring newsletter." # -> copywriter-small
osr -t marketing.yaml -r marketing.yaml route "Translate this landing page headline into Spanish: Route every request to the right model." # -> skill-translate
```
## 8. Travel and hospitality
**Situation.** Guests ask quick questions at any hour. Trip planning with bookings is agent work with tools.
Anything naming a guest stays on the group's own systems.
**Catalogue.**
```yaml
# travel.yaml
targets:
- id: concierge-fast
kind: llm
name: Concierge model (cheap)
capabilities:
domains: [travel, general, general_chat, customer_support]
actions: [qa, generation, summarize]
max_complexity: 0.45
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.0002 }
latency_ms: 300
quality_prior: 0.55
examples:
- "What time is checkout?"
- id: booking-agent
kind: agent
name: Booking agent (search and reservations)
capabilities:
domains: [travel]
actions: [planning, action, reasoning, generation]
supports_tools: true
tags: [autonomous]
min_complexity: 0.3
constraints: { pii_allowed: false }
cost: { usd_per_1k_tokens: 0.006 }
latency_ms: 5000
quality_prior: 0.82
examples:
- "Plan a five-day itinerary with train bookings."
- id: guest-records-onprem
kind: llm
name: Guest-records model (own systems)
capabilities:
domains: [travel, customer_support, general]
actions: [extract, summarize, qa, classify]
max_complexity: 0.7
constraints: { data_boundary: on_prem, pii_allowed: true }
cost: { usd_per_1k_tokens: 0.001 }
latency_ms: 1200
quality_prior: 0.7
rules:
- name: guest-data-stays-inside
when: { contains_pii: true }
prefer: [guest-records-onprem]
weight: 1.0
- name: trip-planning-to-agent
when: { domains: [travel], actions: [planning, action] }
prefer: [booking-agent]
weight: 0.8
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Plan a five-day itinerary in Kyoto for a family with two kids and book the train from Tokyo.` | `booking-agent` | Travel planning with a booking to make: the agent with tools |
| `What time is checkout?` | `concierge-fast` | A trivial question: the cheap model |
| `Change the booking for guest anna.k@example.com to a late arrival on Friday.` | `guest-records-onprem` | The guest's e-mail address is PII: only the group's own model is admissible |
**Try it.**
```bash
osr -t travel.yaml -r travel.yaml route "Plan a five-day itinerary in Kyoto for a family with two kids and book the train from Tokyo." # -> booking-agent
osr -t travel.yaml -r travel.yaml route "Change the booking for guest anna.k@example.com to a late arrival on Friday." # -> guest-records-onprem
```
## 9. Data and analytics
**Situation.** Analysts ask for SQL, quick arithmetic and narrative analysis of datasets. SQL goes to a
deterministic skill, arithmetic to a calculator tool that costs nothing and never hallucinates, analysis to
a mid-tier model.
**Catalogue.**
```yaml
# analytics.yaml
targets:
- id: skill-sql
kind: skill
name: SQL skill
capabilities: { domains: [data_analysis], actions: [code_generation, extract] }
cost: { usd_per_1k_tokens: 0.001 }
latency_ms: 200
quality_prior: 0.8
instructions: "Always qualify table names with the schema; never SELECT *."
examples:
- "Write a SQL query that returns revenue per month."
- id: tool-calculator
kind: tool
name: Calculator
capabilities: { domains: [math], actions: [qa], max_complexity: 0.3 }
cost: { usd_per_1k_tokens: 0.0 }
latency_ms: 5
quality_prior: 0.99
examples:
- "What is 17 percent of 2340?"
- id: analyst-mid
kind: llm
name: Analyst model
capabilities:
domains: [data_analysis, math, general]
actions: [reasoning, summarize, qa, generation, planning]
min_complexity: 0.25
cost: { usd_per_1k_tokens: 0.003 }
latency_ms: 900
quality_prior: 0.78
examples:
- "Analyze this CSV of monthly sales and describe the trend."
rules:
- name: sql-to-skill
when: { domains: [data_analysis], actions: [code_generation] }
prefer: [skill-sql]
pin: true
weight: 1.0
- name: arithmetic-to-calculator
when: { domains: [math], max_complexity: 0.4 }
prefer: [tool-calculator]
pin: true
weight: 1.0
```
**What happens.**
| Request | Routed to | Why |
|---|---|---|
| `Write a SQL query that returns revenue per month and region from the orders table.` | `skill-sql` | Data-analysis code generation: pinned to the SQL skill |
| `What is 17% of 2340?` | `tool-calculator` | Simple arithmetic: pinned to the free, exact tool |
| `Analyze this CSV of monthly sales and explain the trend and the likely drivers.` | `analyst-mid` | Reasoning over data: the analyst model |
**Try it.**
```bash
osr -t analytics.yaml -r analytics.yaml route "What is 17% of 2340?" # -> tool-calculator
osr -t analytics.yaml -r analytics.yaml route "Analyze this CSV of monthly sales and explain the trend and the likely drivers." # -> analyst-mid
```
## The patterns behind the scenarios
| You need | Declare | What the router does |
|---|---|---|
| Personal or regulated data never leaves your boundary | `constraints: { pii_allowed: false }` on every external target; a rule `when: { contains_pii: true }` preferring the internal one | Detects e-mail, phone, IBAN, card, SSN and Aadhaar numbers in the request; policy removes every `pii_allowed: false` target before scoring |
| A whole department must stay on-premises regardless of content | `--constraint data_boundary=on_prem` (HTTP: `"constraints": {"data_boundary": "on_prem"}`) | Rejects every target with a weaker boundary and records the reason in the trace |
| Only targets in a region | `constraints: { regions: [eu] }` on the target and `--constraint region=eu` on the request | Same hard filter |
| A person handles emergencies, complaints and sensitive cases | A `human` target with `actions: [escalate]` and a pinned rule `when: { actions: [escalate] }` | Every model is excluded when the escalation intent is detected |
| System events are routed like chat | `when: { event: ["order.*", "payment.failed"] }`; send `--event order.shipped` (HTTP: `"context": {"event": "order.shipped"}`) | The event name supplies domain and intent; pinned rules pick the workflow, agent or person |
| Cheap things stay cheap, hard things get the best model | `max_complexity` on cheap targets, `min_complexity` on expensive ones | The utility trades quality against cost and latency inside the admissible set |
| Deterministic work never goes to a model | A `skill` or `tool` target and a pinned rule on its action | SQL, translation, arithmetic reach the skill or tool directly |
| A house style on top of any model | A `persona` with `primary: false`; ask with `--plan` | Fills the `persona` slot of every plan whose domain matches; its instructions become the system prompt |
## From the page to production
- **Same file, served over HTTP.** `osr -t retail.yaml -r retail.yaml serve` exposes `/route`, `/feedback`, the
OpenAI-compatible `/v1` endpoints and `/mcp` for the catalogue above; the [deploy guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md)
covers the container image, Helm chart and the hardening flags.
- **Real providers, still without code.** Point the served router at any OpenAI-compatible endpoint with
`OSR_LLM_BASE_URL` and `OSR_LLM_API_KEY`; every model target whose `metadata.model` names an upstream model then
answers `/v1/chat/completions` for real ([deploy guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md)). Attaching your own handlers in
Python is the [user guide's fourth chapter](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#4-real-providers).
- **Hosted platform.** The same decisions, traces and feedback loop behind an API key, with plans, tenants and a
dashboard: [platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md). Mount a catalogue with `OSR_TARGETS` / `OSR_RULES` when you run the
platform yourself ([install guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/INSTALL.md)); `POST /api/v1/route` takes the same `text`, `context`
(including `context.event`) and `constraints` as the examples above.
- **Hardening for multi-tenant traffic.** Health circuit breakers, budgets, input and output guards, audit and
persistent learner state come from the [enterprise builder](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ENTERPRISE.md) or, on `osr serve`, the
`--guard --redact-pii --output-guard redact --health --audit audit.jsonl` flags.
- **Measuring your own traffic.** Put your requests and the target you expect into a JSONL file
(`{"text": "...", "expected": "support-agent"}` per line) and run `osr -t retail.yaml -r retail.yaml eval traffic.jsonl`
to gate routing accuracy in CI ([evaluation](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#9-command-line)).
## How this page is verified
The test suite parses this Markdown file. For every scenario it writes the YAML block to the file named
in its first line, builds the router the way `osr` does (`load_targets` + `load_rules` + the default strategies),
routes every row of the *What happens* table (text rows as plain requests, `event` rows as an event without text)
twice - with the SDK's default objective and with the command line's default `--cost-weight 0.3` - and asserts the
*Routed to* target. The same rows then go through `osr -t -r route --json` and through `POST /route`
on the FastAPI app `osr serve` runs. Every `osr` command in every *Try it* block is executed as written, and its
`# -> target` annotation is asserted. The tables are then repeated against
the hosted platform application with the catalogue mounted as the deployment's `OSR_TARGETS` / `OSR_RULES`
(`POST /api/v1/route` with a workspace key). Change a request, a target or a rule here and the tests tell you
whether every surface still agrees.
Against a **running deployment** - production included - the same
requests go to the default catalogue the platform mounts with a real
workspace key, checking the guarantees that hold there (personal data never reaches a public-cloud model,
escalations reach the human queue, `payment.failed` and `order.shipped` route without text, translations reach
the translation skill, arithmetic reaches the calculator), recording feedback for every decision.
---
# Platform guide
The hosted platform runs OpenSmartRoute as a service. A workspace has an API key and a plan; each
request to the API is routed to a target (model, agent, skill, persona or tool) and answered with the
decision, its trace and, on request, the target's output. An OpenAI-compatible endpoint accepts
existing chat-completion clients.
This page covers calling the platform: authentication, routing, execution, the OpenAI-compatible
endpoint, feedback, plans, organizations, tenants, governance (workspace policy and budgets),
observability (probes, metrics, per-request traces and the event stream), caching, the dashboard and
the MCP server. The
[marketplace](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MARKETPLACE.md) - shared agents, skills, personas, prompts and stack templates - has
its own page. To run the router on your own infrastructure see
[Deploy with Docker and Helm](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md); to use it as a Python library see the
[user guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md) and the [SDK guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md). Every endpoint below is listed in the REST API
reference (`/docs/api`) and in `/openapi.json`.
## 1. Authentication
1. Sign up at `/platform/signup` with Microsoft, Google, GitHub, GitLab, LinkedIn (the providers the deployment
enables - `GET /api/v1/info` lists them under `sso`; the LinkedIn app the growth engine posts with doubles
as the sign-in provider unless `OSR_PLATFORM_LINKEDIN_SSO=false`) or an email address - with a password if
you want to sign in by email later (`OSR_PLATFORM_PASSWORD_LOGIN`, on by default). Signup creates a personal
workspace on the **free** plan and issues an API key. With Google configured the site also shows Google's One
Tap prompt ("Sign in to ... with google.com") to visitors who are not signed in; picking an account posts the
ID token to `POST /api/v1/auth/{provider}/credential` `{credential, next}` (Google only for now), which has
Google validate it and
answers with the same browser session as the redirect callback - no page change, a first sign-up still shows
the key once. The Google OAuth client must list the site's origin (`OSR_PLATFORM_WEB_URL`) under *Authorized
JavaScript origins* or Google refuses to show the prompt.
2. The key is shown once. Store it in a secret manager. Keys are created, renamed, rotated and revoked in
the dashboard (`/platform/dashboard/keys`) or with `POST /api/v1/keys`, `PATCH /api/v1/keys/{key_id}`
`{name}`, `POST /api/v1/keys/{key_id}/rotate` (a new secret for the same key, shown once; the old one
stops working immediately) and `DELETE /api/v1/keys/{key_id}` - all admin role, or the lead of the team a
key was minted for (`team` on `POST /api/v1/keys`, [Teams](#teams)). A key does everything its workspace
can unless you limit it: `scopes` (`read` - every GET; `route` - decisions without execution: `/route`,
`/estimate`, MCP ranking tools; `execute` - anything that can spend money: `execute: true`, `/v1/*`, the
MCP `ask` tool; `feedback`; `manage` - every other write) and `expires_in_days` (1-3650) on `POST /api/v1/keys`
or `PATCH /api/v1/keys/{key_id}` `{scopes}`; `GET /api/v1/keys/scopes` lists them. A scoped key that
steps outside its scopes is refused with `403` and `X-Key-Scope: ` before anything is
metered; an expired key answers `401 API key expired` until it is revoked (rotation keeps the expiry).
`limits` narrows what a key may spend and reach: `daily_budget_usd` / `monthly_budget_usd` cap the key's own
spend (checked like the workspace and tenant budgets - `429` with the `X-Budget-*` headers once reached;
routing without execution still works), `requests_per_minute` caps its rate below the plan's (`429` with
`Retry-After` and `X-Key-Limit: requests_per_minute`), `tokens_per_minute` caps the tokens it may send and
receive in a sliding minute (charged with the request's bound - the prompt plus the output allowance - before
any provider is called, so a burst is refused rather than billed: `429` with `Retry-After` and
`X-Key-Limit: tokens_per_minute`), `allowed_ips` pins the key to client addresses or CIDR blocks (at most 32;
the address is the one past the trusted proxies, never a header the client wrote; a call from elsewhere answers
`403` with `X-Key-Limit: allowed_ips`) and `allow_targets` lists the target ids the key may route to (the
router never leaves the list, a pinned `model` outside it answers `403` with `X-Key-Limit: allow_targets`, and
a caller's own `allow_targets` can only narrow it further). `required_headers` names headers every request with
the key must carry (a project tag, a ticket id, a cost centre - at most ten; a request without them answers
`428` with `X-Key-Limit: required_headers`), and `mcp_tools` lists the connected MCP servers' tools the key may
reach (`server:tool` or `server:*`; section 12). `PATCH /api/v1/keys/{key_id}` `{limits}` replaces
them (`{}` clears); `GET /api/v1/keys/scopes` lists the limit keys; every change is audited (`key.limited`).
Give keys that leave your hands - CI, a partner, a demo - the smallest scopes, an expiry and a budget.
3. Send the key with every metered request in either header, and - when you want the row attributed to a cost
centre without a tenant - the project tag:
```http
Authorization: Bearer osr_...
X-API-Key: osr_...
X-OSR-Project: checkout-service
```
`X-OSR-Project` (letters, digits, `.`, `_`, `-`; up to 64) is recorded on the usage row as `project` and broken
out in `GET /api/v1/usage` as `by_project` - cheaper than a tenant when all that is wanted is a line in the
report. The Google GenAI SDKs' `x-goog-api-key` header is accepted as the key too (section 4).
Public endpoints (`/api/v1/info`, `/api/v1/models`, `/api/v1/llms`, `/api/v1/rankings`,
`/api/v1/catalogue`, `/api/v1/estimate`, `/api/v1/stats/public`, `/api/v1/registry`) need no key.
### Email and password
`POST /api/v1/auth/password/login` `{email, password}` returns the same browser session as a
single sign-on callback (`session_token`, `user`, `account`, `plan`); the dashboard stores it and
sends it as a bearer token. Passwords are at least 8 characters (no composition rules) and are stored
as salted scrypt hashes. A signed-in user sets or changes theirs with `POST /api/v1/auth/password`
`{password, current_password}` - the current one is required once set and every other browser is
signed out - and removes it with `DELETE /api/v1/auth/password` as long as a provider identity remains.
Sign-in attempts are limited per address and per email; `has_password` on the user object says which
users have one. `/platform/dashboard/account` has the form.
**Two-factor authentication.** A person with a password can add an authenticator app (TOTP, RFC 6238: six
digits every 30 s, one step of drift accepted) under *Account › Two-factor authentication*:
`POST /api/v1/auth/mfa/setup` `{current_password}` returns the secret and the `otpauth://` URI,
`POST /api/v1/auth/mfa/enable` `{code}` confirms it and returns eight single-use recovery codes (shown once;
other browsers are signed out), `POST /api/v1/auth/mfa/recovery-codes` `{code}` replaces them and
`DELETE /api/v1/auth/mfa` `{current_password, code}` switches the factor off - each change is e-mailed as a
security notice (`two_factor_changed`). From then on `POST /api/v1/auth/password/login` answers
`mfa_required` with a five-minute signed `challenge` instead of a session, and
`POST /api/v1/auth/mfa/verify` `{challenge, code}` with the authenticator code (or an unused recovery code)
issues it; the sign-in page shows the code step. A password-reset link proves the inbox, not the second
factor, so `POST /api/v1/auth/password/reset` saves the password and answers the same challenge
(`password_changed: true`). Identity providers bring their own second factor and are not asked again; API
keys are the workspace, not a person, and never are. `mfa_enabled`, `mfa_enabled_at` and
`recovery_codes_left` are on the user object; an operator resets a locked-out person with
`PATCH /api/v1/admin/users/{id}` `{"mfa_reset": true}` (sessions end, the person is told). History:
`user.mfa_enabled`, `user.mfa_disabled`, `user.mfa_recovery_used`, `user.mfa_recovery_codes_replaced`,
`user.login_mfa_failed`, `user.mfa_reset`.
### Account lifecycle: confirmation, recovery, deletion
Anyone can create an account with any email address. What happens around it:
| Step | How | Endpoints |
|---|---|---|
| Email confirmation | Signup sends a link to `/platform/verify-email?token=...` (valid three days). Signing in through a provider that vouches for the address, accepting an invitation sent to it or completing a password reset also confirms it. A provider vouches when it sends `email_verified` (Google, any OIDC issuer, Entra's optional `xms_edov`) or when it only reports addresses it verified or owns - GitHub, Microsoft, GitLab and LinkedIn, as the shared button or as an organization's own connection of that kind; a generic OIDC issuer that sends no such claim leaves the address unconfirmed and the confirmation link goes out. `email_verified` on the user object. **Confirmation is required** (`OSR_PLATFORM_REQUIRE_EMAIL_VERIFICATION`, on by default): until it is done every call but `/api/v1/me`, `/api/v1/onboarding`, `/api/v1/auth/*` and `/api/v1/legal` answers `403` with `X-Email-Verification: required` - for the person's session and for the workspace's API keys alike (a key stands on an owner's confirmed address), on the OpenAI-compatible `/v1/*` proxy and the MCP server as on the REST API, and a public endpoint that personalises its answer for a key (`/api/v1/estimate`, the site assistant) treats an unconfirmed key as no key at all - anonymous limits, never the workspace's; `GET /api/v1/me` says so in `email_verification_required` and the dashboard shows the confirmation step in place of its pages, with *Send the link again*. Signups from domains reserved for testing (`example.com`, `example.org`, `*.test`, `*.local`, `*.invalid`, `*.localhost`) are refused unless `OSR_PLATFORM_SIGNUP_TEST_DOMAINS` is on (throw-away CI stacks only), and a user who never confirms is removed after `OSR_PLATFORM_UNVERIFIED_ACCOUNT_TTL_DAYS` (14) with the personal workspace only they belonged to. | `POST /api/v1/auth/verify` `{token}`, `POST /api/v1/auth/verify/send` (signed in, once a minute) |
| Forgot password | `/platform/forgot-password` emails a single-use link to `/platform/reset-password?token=...` (one hour). The answer is always `202` so it never reveals whether an address has an account; limited per address and per email. Resetting revokes every session, confirms the address, signs the person in and sends a notice. | `POST /api/v1/auth/password/forgot` `{email}`, `POST /api/v1/auth/password/reset` `{token, password}` |
| Invitations | `POST /api/v1/workspace/invites` emails the invitee a link to `/platform/invite/` (fourteen days) and returns the same link once to the inviter. | see [Organizations](#7-organizations) |
| Account history | The person's own sign-ins, password and email changes and memberships, newest first. | `GET /api/v1/me/audit` |
| Download my data | One JSON document with everything held about the person and the current workspace: profile, identities, sessions, memberships and history; the workspace with plan, members, teams and their rosters, key prefixes, tenants, policy, SSO, usage, the request log, reported outcomes and the kept evaluation reports (`evaluations`) of up to a year. Prompt text is stored only when the workspace policy turns request logging on (`log_payloads`); whatever was kept is included as `request_logs`. *Download my data* on `/platform/dashboard/account`. | `GET /api/v1/me/export?days=` |
| Delete account | *Delete my account* on `/platform/dashboard/account`: type the email (and the password when one is set). Sessions, linked identities and memberships go, workspaces only this user belonged to are disabled and their keys revoked; an organization with other members needs another owner first. A notice is emailed. | `POST /api/v1/me/delete` `{confirm, password}` |
| Delete organization | *Delete organization* on `/platform/dashboard/workspace`, owners only: type the organization's name or slug (and the password when one is set). Its keys, members, invitations, tenants, policy, SSO connection, usage and outcomes go at once and a live subscription is cancelled immediately - nothing more is billed, the paid remainder is not refunded automatically. Every member keeps their personal workspace and their session moves to it; the deletion is on the audit trail (`account.deleted`). Personal workspaces go with the user. | `POST /api/v1/workspace/delete` `{confirm, password}` |
| Getting started | The checklist of a workspace: e-mail confirmed, API key created, first request routed, first outcome reported (the playground's one-click *Good* / *Poor*), a model provider connected and - for organizations - a teammate invited; each step carries the dashboard link and the API call that completes it, `next` is the first open one and `complete` the share done. The dashboard overview shows it until every step is done. Until the first request is routed the card also asks *What is in your way?* - one click from a short list (no time yet, no provider to connect, not clear how, a different integration, something failed, still comparing, a feature missing) and an optional line; the answer reaches the operator next to the workspace and the thanks link to the page that removes the obstacle. | `GET /api/v1/onboarding`, `GET /api/v1/journey/blocker`, `POST /api/v1/journey/answer` |
Signup, confirmation, password recovery, invitation links and this checklist form the `onboarding`
domain, which a deployment can run as its own service (see [Services](#services)).
Every one of these events - plus sign-ins, failed sign-ins, role changes, removals and operator actions
- is written to the account audit log. Workspace admins read their workspace's entries with
`GET /api/v1/workspace/audit`; operators see everything at `/platform/admin/audit` (`GET /api/v1/admin/audit-log`)
and per user on `/platform/admin/users/`, where they can also issue a reset link, re-send or force the email
confirmation, disable, sign out everywhere or delete the user.
### Sign-in locations and alerts
Every browser session and every account event records the address it came from (`ip`). An address alone tells
a person little - *203.0.113.9* does not say whether that was their own phone on holiday or somebody else - so
the platform can turn it into a place (`location`: country, region, city and a label such as *Lisbon, Portugal*)
from one of two sources the operator chooses; without either, nothing is resolved and the address alone is shown:
| Source | Setting | What it gives |
|---|---|---|
| A header the ingress sets | `OSR_PLATFORM_GEOIP_HEADER` - the header's name, `CF-IPCountry` behind Cloudflare, `X-Country-Code` behind a gateway that resolves it | The country. Free, nothing leaves the platform. |
| A lookup service | `OSR_PLATFORM_GEOIP_SERVICE` - a preset (`ipinfo`, `ipapi`, `ip-api`, `ipwho`) or a URL template with `{ip}` and `{token}`; `OSR_PLATFORM_GEOIP_TOKEN` when the service wants a key | City level. One HTTPS call per new address, two seconds at most (a slow answer never holds a sign-in), cached in the database for thirty days and shared by every replica. Private and loopback addresses are never looked up (*local network*). |
Both settings are also under *Runtime configuration* in the console, so a deployment can switch them on without
a redeploy. Choose a service whose terms fit your use: sending a visitor's address to a third party is a
processing of personal data, and your privacy notice should name it.
**Sign-in alerts.** A sign-in from a device the account has not been seen on in ninety days, or - once places
are resolved - from a country none of its recent sessions came from, is announced to the person by e-mail
(`new_sign_in`, a security message that cannot be switched off) and in their notifications, with the device, the
method, the place and the address; the account history records `user.new_device` or `user.new_location`. A
person who wants to hear about every sign-in chooses *Every sign-in* under *Sign-in alerts* on
`/platform/dashboard/account` (`PUT /api/v1/me/login-alerts` `{mode}`, `new` or `always`; `login_alerts` on the
user object). The very first session of an account is not announced - the welcome covers it.
**What operators see.** The user page shows each session's place next to its address and a *Sign-in places*
card: where the account was signed in from in the last ninety days, with the addresses, the devices, how many
sign-ins and the first and last one - a traveller shows one device in several places, a takeover a new place on
a new device. The deployment audit log carries the place of every event, filters by country (`country=DE`,
`--` for events without a place), searches it with `q` and exports it in the CSV.
**Forgetting.** Addresses and places are kept with the row for `OSR_PLATFORM_NETWORK_DATA_RETENTION_DAYS`
(365; `0` = as long as the event) and then removed by the retention sweep - the event stays, the address and
the place go. *Forget addresses* on the user page (`DELETE /api/v1/admin/users/{id}/network-data`) does the same
for one person at once, for a privacy request; it is audited as `user.network_data_cleared` with the operator's
own address and place, like every operator action.
**Operators too.** A console sign-in records its address and place on the operator's session (shown under
*Operators*), `operator.login`, `operator.login_failed`, `operator.login_mfa_failed` and `operator.logout` are in
the audit log with the place, and every operator action carries the address and place of the console session it
came from. A console sign-in from a browser or a country the operator has not used in ninety days is written as
`operator.new_device` / `operator.new_location` and filed in the console's bell for every operator to see.
### Email: transports, templates, welcome, newsletter and reports
Every message the platform sends is composed once as a
branded HTML card (privacy and terms links in the
footer) with a plain-text alternative: the welcome message (greeting, a four-step checklist, dashboard
and quick-start links - sent by signup, or by the first sign-in of a person who arrived another way: a shared or
an organization's identity provider, an operator-created user, a password-reset link; never twice), address
confirmation (sent by signup; the first sign-in sends it too when the address is still unconfirmed and no link ever
went out), password reset, invitation, account deleted, password changed, the
operator test message, usage reports, alerts, support-ticket mail, product news and newsletter issues. Each is
written to the outbox first and then delivered by whichever transport is configured (the first configured one
in this order wins).
Every message follows the same design so a reader knows what they are holding before they read a word: the
wordmark, a coloured band and a small label naming the kind of message (*Security* in red, *Alert* in amber,
*Account* in ink, *Getting started* and *Report* in blue, *Release notes* and *Product news* in green,
*Newsletter* in violet, *Support* and *Feedback* in teal), one heading, short sentences that read at an
eighth-grade level (`tests/test_email_readability.py` fails a template that does not), one clear action, a
plain-text alternative, and a footer that says why the person is receiving it and where to change that. A
message that could worry someone - a new sign-in, failed attempts, a change to the account - always says what
happened, whether anything is lost, what to do if it was not them, and that a reply reaches a person.
| Transport | Setting | Notes |
|---|---|---|
| Microsoft 365 (Microsoft Graph) | `OSR_PLATFORM_M365_TENANT_ID`, `OSR_PLATFORM_M365_CLIENT_ID`, `OSR_PLATFORM_M365_CLIENT_SECRET` | Mail leaves from the Exchange Online mailbox named by `OSR_PLATFORM_MAIL_FROM` (display name `OSR_PLATFORM_MAIL_FROM_NAME`, default the brand) with `POST /users/{sender}/sendMail`, so DKIM, SPF and the tenant's retention apply as for any other mailbox; nothing is saved to Sent Items. Set-up: in Entra ID register an application, add the **application** permission `Mail.Send` (Microsoft Graph) and grant admin consent, create a client secret, and give the platform the tenant id (GUID or verified domain), client id and secret; the tenant id and client id are validated at start-up and a malformed value disables the transport with the reason in `GET /api/v1/admin/mail`. Restrict the app to the one sender with an Exchange *application access policy* (`New-ApplicationAccessPolicy -PolicyScopeGroupId -AccessRight RestrictAccess`). Tokens (client credentials) are cached until a minute before expiry and refreshed once on a `401`. |
| Azure Communication Services Email | `OSR_PLATFORM_ACS_CONNECTION_STRING` | `endpoint=https://..communication.azure.com/;accesskey=...`; the platform signs requests itself (HMAC-SHA256, no SDK). `OSR_PLATFORM_MAIL_FROM` must be a sender of a linked domain, e.g. `DoNotReply@.azurecomm.net` for an Azure-managed domain or `no-reply@yourdomain` once a custom domain is verified. |
| SMTP | `OSR_PLATFORM_SMTP_URL` | `smtp://user:pass@host:587` (STARTTLS) or `smtps://user:pass@host:465`. |
| Outbox only | none | Nothing is lost: operators read the links at `/platform/admin/mail` (`GET /api/v1/admin/mail`) and hand them over; `mail` in `GET /api/v1/info` tells the web app which case applies. |
**Delivery guarantees.** Sending is at-least-once with bounded retries. Every attempt is recorded on the outbox
row (`attempts`, `error`, `next_attempt_at`); a failure is retried on a back-off ladder (30 s, 2 min, 10 min,
30 min, 2 h - six attempts in all) by a sweeper thread in the API, one leader per Redis lease in a cluster.
A row whose sending process disappeared mid-send is retried after five minutes. Once the ladder is exhausted
the row stays in the outbox as failed for an operator to resend by hand (`POST /api/v1/admin/mail/{id}/resend`
always tries once more). ACS accepts a message with `202` and delivers asynchronously; the platform polls the
operation status briefly and records a terminal `Failed` (unverified sender, suppressed recipient) as an error
instead of a false "sent", keeping the operation id as `provider_id`. `GET /api/v1/admin/mail` returns a
`delivery` block (transport `ok`, last error, rows `retrying`, sweeper state) next to `summary.retrying` and
`summary.abandoned`; `GET /readyz` reports the same under `mail` (informational, never fails readiness).
**The gallery: every message before a customer sees it.** Operators manage what the platform says from
`/platform/admin/mail`: `GET /api/v1/admin/mail/templates` lists every kind of message with its category,
whether people can switch it off and how many went out; `GET /api/v1/admin/mail/templates/{kind}` renders it with
sample data (subject, the HTML card, the plain-text alternative) and the page shows it as a customer would see it;
`POST /api/v1/admin/mail/templates/{kind}/send` `{to}` sends that sample to the operator's own inbox (stored in the
outbox under the kind with a `[Sample]` subject; audit `mail.sample`) to check the words and the design in a real
mail client. Together with the outbox (read, resend, delete), the transport test, the suppression list, each
user's switches (`/platform/admin/users/`), the report schedule (`/platform/admin/reports`) and the campaign
switches (`PATCH /api/v1/admin/campaigns/{campaign}` `{enabled}`, on `/platform/admin/growth`), every message the
platform sends can be seen, tested, held back or re-sent from the console.
**Bounces and the suppression list.** An address the transport rejects for good (SMTP `550`-`554`, a `5.1.x`
status, "user unknown", "mailbox unavailable", an ACS `Suppressed`/`Bounced` status) is not retried: the row is
abandoned at once with the reason and the address goes on the **suppression list** (`email_suppressions`, reason
`bounce`), so no later message - of any category, newsletter included - is composed for it; such a skip is
counted as `suppressed` in `delivery`. Delivery reports keep the list honest after acceptance: with
`OSR_PLATFORM_MAIL_EVENTS_TOKEN` set, `POST /api/v1/email/events/acs?token=…` receives Azure Communication
Services' Event Grid events (subscribe the endpoint to `EmailDeliveryReportReceived` and
`EmailEngagementTrackingReportReceived`; the handshake is answered automatically, a wrong token is `401`, and the
endpoint answers `404` without the setting) and stamps each outbox row with `delivery_status` / `delivery_at`
(`delivered`, `bounced`, `spam`, `quarantined`, `expired`, `failed`); a bounce, quarantine or spam report also
suppresses the address, a soft failure (`expired`) for thirty days only. Operators see and edit the list at
`/platform/admin/mail` - `GET /api/v1/admin/mail/suppressions` (`q`, `limit`), `POST /api/v1/admin/mail/suppressions`
(`email`, `reason`, `detail` - for complaints and requests made out of band) and
`DELETE /api/v1/admin/mail/suppressions/{email}` to allow an address again; `GET /api/v1/admin/mail` carries the
list size by reason under `suppressions`, whether delivery reports are on under `delivery_reports`, and the last
thirty days of reported outcomes under `delivery.delivery_30d`. `GET /metrics` exposes the same as
`osr_mail_delivered_total`, `osr_mail_failed_total`, `osr_mail_suppressed_total`, `osr_mail_reports_total`,
`osr_mail_retrying`, `osr_mail_suppression_list` and `osr_mail_delivery_30d{status=…}`.
**The sender's address.** `OSR_PLATFORM_MAIL_POSTAL_ADDRESS` (one line, e.g. `Acme Ltd, 1 Main St, Springfield`)
is printed in the footer of every message, HTML and text, as CAN-SPAM and most national rules expect of
commercial mail; the console's mail page shows what is configured (`postal_address` in `GET /api/v1/admin/mail`).
**What people receive, and how they say no.** Mail falls into categories. *Security* (address confirmation,
password changes, sign-in and deletion notices, an API key created or revoked - to every owner, naming who did it
and the key's prefix) and *account* (welcome, invitations, payment problems, plan
changes - the owners hear when the plan moves up or down, with the new limits) are always sent. *Usage reports*, *alerts* (quota, budget and error-rate alerts that reach the owners
when a workspace has no alert channel), *support* (ticket confirmations and answers), *product news* and *getting
started* (one reminder with the next step when a workspace has not routed a request a day after signing up - sent
once per workspace, to signups of the last seven days only, by the `accounts` service on the digest cadence) are
optional: every such message carries a **Manage e-mail preferences** link and an **Unsubscribe** link in the
footer plus the `List-Unsubscribe` / `List-Unsubscribe-Post` headers (RFC 8058), so Gmail, Outlook and Apple
Mail show their own unsubscribe button (Graph accepts only `x-` custom headers, so on the Microsoft 365
transport the footer links do the job). The links carry a signed token (HMAC of the user id, valid for six
months, signed with `OSR_PLATFORM_SESSION_SECRET`), so they work without signing in:
`/platform/email/preferences?token=…` shows the switches (`GET`/`PUT /api/v1/email/preferences`), and a mail
client's one-click POST goes to `POST /api/v1/email/unsubscribe?token=…&category=…`. Signed in, the same
switches - categories, the newsletter subscription and, for owners and admins, the workspace's report schedule -
sit under **Account › E-mail** (`GET`/`PUT /api/v1/me/email-preferences`). A message to someone who switched
its category off is skipped before it is composed and counted as `skipped` in `GET /api/v1/admin/mail`; every
change is written to the audit log (`user.email_preferences`, `user.email_unsubscribed`).
**Everyone's own clock.** Scheduled mail is timed by the recipient's time zone, not the server's. The sign-up
form and the newsletter form send the browser's IANA zone (`timezone` on `POST /api/v1/signup` and
`POST /api/v1/newsletter/subscribe`); a signed-in person who never named one gets the browser's zone the first
time they open **Account › E-mail** (and is told so), and the same card lets anyone pick another zone or clear it
(`timezone` on `PUT /api/v1/me/email-preferences` and on the token variant; `schedule` in the answer names the
zone, its current offset and the hours below; operators can set it with `PATCH /api/v1/admin/users/{id}`). Until
a zone is known, UTC applies. What follows the clock: usage digests go out at `OSR_PLATFORM_REPORTS_HOUR`
(default 8) o'clock local over the recipient's own completed day, week or month; the newsletter issue at
`OSR_PLATFORM_NEWSLETTER_HOUR` (default 9) local and the evening issue at `OSR_PLATFORM_NEWSLETTER_EVENING_HOUR`
(default 18) local; the getting-started reminder, the upgrade offer, release notes
and a trial-ending notice that is not urgent only between 08:00 and 20:00 local (a deferred recipient is picked up
by the next sweep, unclaimed; the console's *Run now* / *Send reminders now* buttons do not wait). Alerts,
confirmations, invitations, receipts and support answers are never held back.
**Newsletter.** `POST /api/v1/newsletter/subscribe` (the form on `/newsletter` - the landing page that says what
the daily issue contains and how the address is treated, with the latest posts - and in the site footer of every
marketing page; off with `OSR_PLATFORM_NEWSLETTER=false`, which hides both forms) stores the address, sends a
branded confirmation and always answers `202`;
`POST /api/v1/newsletter/confirm` completes the double opt-in and sends the welcome issue with its unsubscribe
link. Every newsletter message carries a one-click unsubscribe (`POST /api/v1/newsletter/unsubscribe`, token as
JSON from the web page or as `?token=` from a mail client's `List-Unsubscribe` POST). A signed-in user with a
verified address joins straight from the preferences card (no confirmation round trip); an unverified one gets
the usual confirmation. **Registered people read it by default** (`OSR_PLATFORM_NEWSLETTER_SIGNUP_DEFAULT`, on):
the signup form says so next to the terms, `POST /api/v1/signup` files the address as a *pending* reader (source
`signup`, the browser's time zone) without a second confirmation mail, and the account's own proof of the address -
the confirmation link (`POST /api/v1/auth/verify`), a password reset, an accepted invitation, an identity provider
that vouches for it or an operator marking it verified - confirms the reader with a consent record naming that
statement. The welcome mail says the newsletter comes with the account and where to turn it off; the switch on
**Account › E-mail** shows *after e-mail confirmation* until then. An address that unsubscribed is never
re-enrolled, a reader who confirmed through the form keeps their row, and `GET /api/v1/info` reports
`newsletter_signup_default` so the site can word the form accordingly. **The blog asks readers in.** A visitor who
is not signed in and reads on (a few seconds on `/blog` or a post, or a third of the page) sees a sheet at the foot
of the page - *Keep reading with a free account* - with the deployment's identity providers (`sso` from
`GET /api/v1/info`, one click), an e-mail field that lands pre-filled on `/platform/signup`, and the sign-in link,
each carrying the page to come back to (`next`, same-origin paths only, checked again by the API's `safe_next`);
the text says the newsletter comes with the account when `newsletter_signup_default` is on. The post is never
held back: *Not now* keeps the sheet away for a week, a subscription from any newsletter form in that browser
or a signed-in browser hides it for good, and it waits for the cookie banner's answer. After signing in the
welcome page's button reads *Back to the post* / *Continue where you left off* instead of *Go to the dashboard*.
Operators manage subscribers and campaigns at `/platform/admin/newsletter`: add an
address by hand, export CSV, write a campaign in plain text (blank lines separate paragraphs, `## ` starts a
heading), preview the rendered email, send a test to one address, then send once to every active subscriber.
The daily issue follows each reader's clock: the issue of a calendar day is composed from the unmailed posts
when the first reader's zone reaches `OSR_PLATFORM_NEWSLETTER_HOUR` and every later zone receives that issue at
its own hour - one issue per reader per day, each post mailed once (`/platform/admin/content` shows the hour;
its *Send now* delivers to everyone who has not had today's issue whatever their local time, `force` composes a
fresh issue and repeats the day). **Morning and evening.** Daily readers get a second issue once their clock
passes `OSR_PLATFORM_NEWSLETTER_EVENING_HOUR` (default 18, local): it carries the posts written since the morning
issue ("N new posts since this morning", subject *evening: ...*), is composed the same way - once, when the first
zone reaches the hour - and is skipped on a day with nothing new. `-1`, or an hour not after the morning one,
sends the morning issue only; the hour is editable at `/platform/admin/settings`. Weekly readers are not
affected. **How much gets written.** However many feeds deliver, the writer turns at most
`OSR_PLATFORM_CONTENT_POSTS_PER_DAY` (default 6, 0 = no limit) entries into posts in any 24 hours, newest first
and spread evenly over the day's polls, so the blog grows steadily and both issues have news; the entries left
over wait for a later poll.
**Daily or weekly, by topic, and the reader's own page.** Every newsletter message links to `/newsletter/preferences?token=…`
(the same signed token as the unsubscribe link; `GET`/`PUT /api/v1/newsletter/preferences`) where a reader picks
`frequency` `daily` (default) or `weekly`, sets or clears the `timezone`, chooses the `topics` to receive (the blog
categories `models`, `llm`, `agents`, `research`, `ai`; an empty list or all of them means everything), sees what
is stored about them, and can erase the record outright (`DELETE /api/v1/newsletter/preferences` - the row is
gone, not marked). A reader who follows some topics receives the day's issue cut to those posts, with a subject
and opener written for what they get; a day with nothing in their topics sends them no e-mail (the issue is still
noted as theirs, so it is never repeated). `POST /api/v1/newsletter/subscribe` accepts `topics` too - the form on a
blog category page subscribes to that category - and the blog's RSS takes the same narrowing:
`/blog/feed.xml?category=agents` or `?tag=…` is a channel of its own. Weekly
readers get one digest on `OSR_PLATFORM_NEWSLETTER_WEEKLY_DAY` (0 = Monday … 6 = Sunday, default 0) at the
newsletter hour of their own zone, composed from the week's posts ("This week: …"); a day without posts sends
nothing. Consent is recorded with the confirmation: the moment, the IP address, the browser and the exact
statement the reader agreed to (`consent` on the subscriber in the operator listing), so a double opt-in can be
shown later. `GET /api/v1/admin/content/issues` reports for each issue how many distinct readers opened it and how
many clicked (`engagement.opens`, `engagement.clickers`; the single-issue endpoint adds the clicks per link).
The counting uses a one-pixel image (`GET /api/v1/newsletter/o/{issue}/{reader}/{sig}.gif`) and a click redirect
(`GET /api/v1/newsletter/c/{issue}/{reader}/{sig}?u=…`, `302` to the post, `404` on a bad signature) keyed by a
hash of the subscriber id, kept for 180 days and never joined to anything else; `OSR_PLATFORM_NEWSLETTER_TRACKING=false`
turns both off and the issues go out with direct links. Every copy also carries *View this issue in your browser*
above the card: `/newsletter/issues/{issue}` (`GET /api/v1/newsletter/issues/{issue}`) shows the same posts the
mail carried, as the blog lists them, for a client that mangles the HTML - a public page keyed by the random issue
id, never by the reader, so it is not tracked and holds nothing about anyone.
**What an issue - and every other message - reads like.** Everything the platform writes to a person is held
to plain language: sentences under about twenty words, everyday words, one idea per sentence, one action per
message, and a line that says why the person is receiving it. Readability is measured (Flesch-Kincaid
grade level, longest sentence, a short jargon list) and every release is gated so no customer-facing message
reads above grade 9 - roughly an eighth grader on the first pass. The
newsletter issue opens with how many posts there are and how long they take to read; each card carries the
title, one line on what happened, up to three **key points** (`highlights` on the post: three or four concrete
facts from the source - a number, a model name, a date, a limit - so a reader who stops there knows the news),
a **Why it matters** line the writer produces (`why` on the post, also on
`GET /api/v1/blog/{slug}`), the reading time, the source and the link; the issue ends by asking the reader what
they want more of (replies reach the operators). The blog page shows the key points and *Why it matters* above
the article; operators edit both (`PATCH /api/v1/admin/content/posts/{post_id}`, `why` and `highlights`). The
writer's house skill
asks for the same plain language, the key points, a `## Why it matters` section and - when the source allows it -
a `## What to do` section in every post, and records the draft's
readability grade next to the routing decision. **Every post is a long-form article of the blog's own**, not a
rewritten excerpt: the writer reads the source page (up to `OSR_PLATFORM_CONTENT_ARTICLE_SOURCE_CHARS` characters,
default 12,000), asks the model for the brief first - title, summary, key points, *why it matters*, an outline of six
to nine sections, tags and keywords, one small JSON object - then writes the article from that outline in plain
Markdown, aiming at `OSR_PLATFORM_CONTENT_ARTICLE_WORDS` words (default 2,000) and never under
`OSR_PLATFORM_CONTENT_ARTICLE_MIN_WORDS` (default 1,500): a draft that stops short is continued once from where it
ended, covering only the sections still missing. **Every post ends its argument with a `## How OpenSmartRoute
helps` section**, in front of *What to do*: what a team that routes its requests through the router gains from this
particular story and what it would set up - written from a fixed list of product facts and the facts of the story,
nothing else; on the site a card under that section takes the reader to the playground, the catalogue and the
set-up guide. The brief plans it, the article pass writes it, and a post whose article skipped it gets the section
in one more metered pass (`passes` shows `product`); a post that fell back to the excerpt or the brief's own text
carries a short hand-written version angled by the story's tags (`routing.product_section` says `model` or
`template`); posts written before the section existed receive it on the next poll, a few at a time, written by the
model for each story - `POST /api/v1/admin/content/posts/product-sections` (*Add product sections* on the console's
Posts tab) does up to a hundred at once, `model: false` uses the hand-written text only. Both passes are metered requests of the editorial workspace and
both are graded for the learners; the post's `routing` carries them (`passes`) and the measured length (`length`:
words, target, minimum, whether it came in short). When the article pass cannot produce at least 400 words the
post keeps the brief's own text or the attributed excerpt and says so (`reason`). The three lengths are
operator-editable at `/platform/admin/settings` and take effect on the next post. **No picture, no post**: with `OSR_PLATFORM_CONTENT_REQUIRE_IMAGE`
(default `true`) a story becomes a post only when it has a real picture that answers as an image - the feed's
own, or what the article page advertises (`og:image`, `twitter:image`, ``; logos and share banners never count) - otherwise the entry is
marked `no-image` and left alone, and the issue only ever carries posts with a picture, so nobody opens a
digest with a blank or broken image. `POST /api/v1/admin/content/posts/images` keeps probing older posts for
a picture (*Fetch missing images* on the console's Posts tab). The platform keeps its own copy of every post's
picture: when a post is written its image is downloaded (16 MB cap; sources that answer
`application/octet-stream` are sniffed for JPEG, PNG, GIF and WebP), brought to web size - no wider than 1,600
pixels, encoded as WebP, transparency kept, animated GIFs left as they are - and stored; each poll copies up to 25
more of the older posts, newest first, and `POST /api/v1/admin/content/posts/images/store` copies up to 200 on
demand (*Copy pictures* on the Posts tab). The site
serves the copy at `/blog/{slug}/image` (`GET /api/v1/blog/{slug}/image`: a week of cache, `ETag`; the web app
keeps each picture for a day so the API answers once per replica, not once per reader), so pages,
the RSS enclosure and the newsletter never embed a third-party host and a source that later disappears does not
leave a broken picture; until the copy exists the address redirects to the source, and a source that cannot be
copied is tried again after a week. The post's `image.source` keeps the original address. Registering a feed
(`POST /api/v1/admin/content/feeds`) looks at the URL first: a feed is named after its own title unless a name is
given; a blog's home page yields the feed it advertises in its `` (`` (`base_url`, `api_key` or `api_key_env`, optional `api_key_header`, `extra_headers`,
`timeout_s`, `max_retries`, `concurrency` and `queue_timeout_s` - see *Capacity* below) and `models.`
(`provider`, `model`, optional `system_prompt` and
request defaults). `{"kind": "mock"}` is a provider that needs no endpoint and no key: it answers
deterministically in-process (the prompt echoed through `reply`, a template over `{model}`, `{prompt}` and the
call count; JSON when `response_format` asks; a tool call when the prompt names a tool), streams word by word,
embeds with a stable vector per text and fails on cue (`fail_when` - a substring of the prompt -, `fail_every`,
`latency_ms`) - the provider for tests, demos and CI pipelines, metered like any other. At start-up the file is imported into the console additively: providers the store does not
know yet and targets that are not mapped yet are added, rows an operator edited are left alone. A provider
the file created and later stopped naming is disabled (its offers and mappings stay, and it is enabled again
the day the file names it) - a deployment that drops a cloud provider takes its credential with it, and left
enabled every call to it would fail with *authentication failed*. Providers an operator created are never touched.
* the **operator console**, `/platform/admin/providers`: add an endpoint from a preset, store the key inline or name
the environment variable that holds it, *Check* it (`GET /models` with its credentials records
reachability, latency and the upstream model ids, which the mapping form then offers), map targets to
upstream models, disable or delete. Every write re-binds the handlers in the API at once, and every other
API replica re-reads the store within `OSR_PLATFORM_PROVIDERS_RELOAD_S` (30 s) - no restart, no redeploy.
The providers owner also runs that check on its own every `OSR_PLATFORM_PROVIDERS_CHECK_S` (300 s; 0 = only
on click) for every enabled provider - one replica per interval - so the health the public catalogue, the
workspace Providers page and the alerts show is never older than that.
The platform's providers are what **every workspace executes on by default**: a fresh workspace needs nothing
connected - its Providers page lists the included models under *Included with the platform* (per provider, with
health and the target -> upstream model pairs), `GET /api/v1/workspace/providers` returns them as
`platform_providers`, and the getting-started step *Models ready* is ticked. A mapping's request defaults travel
with every call: the deployment's Ollama mappings carry `reasoning_effort: none` on every chat target except the
reasoning-by-design ones (`research-agent`, `gpt-oss`), because a
thinking model asked for a short answer otherwise spends the caller's whole `max_tokens` thinking and returns an
empty message - a request that sends its own `reasoning_effort` overrides the default.
`GET /api/v1/providers` is the public, OpenRouter-style view of what a deployment executes on, rendered
by the site at `/providers`: one row per
executable target with `provider`, `model`, the recognised reference model, `input_usd_per_1m` /
`output_usd_per_1m` (from the reference catalogue when the upstream model is known, else the target's
declared cost), context window, tool and reasoning support, circuit-breaker `health` and seven-day traffic;
plus each provider's host, kind, health and model count. Credentials never appear: the console shows only
whether a key is set and its last four characters (`api_key_set`, `api_key_hint`). Keys stored through the
console live in the platform database next to the organization SSO secrets; prefer `api_key_env` when the
host injects secrets. Endpoints: `GET|POST /api/v1/admin/providers`, `GET|PATCH|DELETE
/api/v1/admin/providers/{id}`, `POST /api/v1/admin/providers/{id}/check`, `PUT|DELETE
/api/v1/admin/providers/models/{target_id}`. Targets without a mapping still route (decisions, plans,
traces, quotes) and report `executable: false`; `GET /api/v1/providers` also counts the `builtin` tools that
run without a provider.
#### Bring your own provider (workspace keys)
Every workspace (feature `byok`, on all plans of the default catalogue) connects its **own** endpoints and keys at
`/platform/dashboard/providers` - Azure OpenAI / Azure AI Foundry, Amazon Bedrock (`bedrock-runtime..
amazonaws.com/openai/v1` with a Bedrock API key), Google Gemini (`generativelanguage.googleapis.com/v1beta/openai`)
or Vertex AI (`-aiplatform.googleapis.com/v1/projects//locations//endpoints/openapi`),
OpenAI, Anthropic (spoken natively over its Messages API - no gateway needed), Mistral, Groq, xAI, Cohere,
OpenRouter, an in-house Ollama, vLLM or LiteLLM gateway, or any
other OpenAI-compatible URL - and maps catalogue targets to models on them (Azure: the deployment name; Bedrock:
the model id). From then on every request of that workspace that executes - `POST /api/v1/route` with
`execute: true`, `/v1/chat/completions`, the MCP `ask` tool - runs the chosen target on the workspace's endpoint
with the workspace's credentials. The router, the trace, quotas, budgets and the learners are unchanged; only
where the call goes differs, so the usage lands on the workspace's own cloud bill and prompts stay inside the
data boundary it chose. Rules: a workspace mapping wins over the platform's for the same target; a target only
the workspace mapped is executable for that workspace alone (`GET /api/v1/targets` and `GET /v1/models` are
per workspace); unmapped targets keep the platform's providers, so a workspace can bring one Azure deployment
and leave the rest to the platform. Disabling a provider withdraws its mappings; deleting it removes them.
Responses name the answering endpoint: `opensmartroute.provider` on chat completions and `result.provider` on
`/api/v1/route` (`/`).
Credentials are stored AES-256-GCM encrypted under `OSR_PLATFORM_SECRET_KEY` (falling back to the session
secret; set it explicitly on multi-replica deployments), bound to the workspace, never returned by the API
(`api_key_set`, `api_key_hint` only) and never resolved from the API container's environment - a workspace
cannot borrow the deployment's keys by leaving the field empty. A key may also be given **by reference**, so the
secret itself never enters the platform database: `keyvault:///[/]` (or the secret's
`https://.vault.azure.net/secrets/...` URL) is read from Azure Key Vault with the API container's managed
identity when the client is built (the identity needs the *Key Vault Secrets User* role; the value is cached for
five minutes, so a rotation in the vault reaches the router without a redeploy), `env://NAME` reads a
variable of the API container (the operator's store of last resort), and `identity://` needs no secret at
all: the container's managed identity mints an access token for the resource (`https://cognitiveservices.azure.com`
for Azure OpenAI and Azure AI Foundry, `https://ai.azure.com`) and sends it as the bearer credential - keyless,
rotation-free; give the identity the *Cognitive Services OpenAI User* role on the resource (the console's *Azure
OpenAI / Foundry (managed identity)* preset fills the reference in, and *Hugging Face Inference Providers* is a
preset too). The platform's own file providers may use every form in `api_key`. A reference is not secret: the console shows it whole (`api_key_hint`,
`api_key_reference: true`), and a vault that refuses the read is what *Check* reports. The endpoint must be a **public** host: base
URLs on loopback, private, link-local or reserved addresses, bare hostnames and `.internal` / `.local` names
are refused (`400 base_url must be a public endpoint`), at save time and again before *Check*, so a workspace
cannot aim the platform's network position at its database, Redis, the model servers or the cloud metadata
service; credentials embedded in the URL are refused too. `OSR_PLATFORM_BYOK_ALLOW_PRIVATE=true` lifts the
host rule on a self-hosted deployment whose workspaces are trusted and whose providers live on the internal
network (the development compose stack sets it). Members read the page; adding, changing,
rotating (`api_key`) or removing (`clear_api_key`) a key, mapping and deleting need the `admin` role (API keys
act as the workspace). Every change is in the workspace audit log (`provider.created`, `provider.updated` with
`key_rotated`, `provider.mapped`, `provider.unmapped`, `provider.deleted` - never the key). *Check* calls the
endpoint's `/models` with the stored key and offers the returned ids on the provider card and in the mapping dialog;
the dashboard runs it right after connecting or editing a provider unless told not to. Endpoints:
`GET|POST /api/v1/workspace/providers`, `GET|PATCH|DELETE /api/v1/workspace/providers/{id}`,
`POST /api/v1/workspace/providers/{id}/check`, `PUT|DELETE /api/v1/workspace/providers/models/{target_id}`
(`provider_id`, `model`, optional `system_prompt` and request `defaults`). Up to 20 providers per workspace; a
change is in force on the next request on every API replica and service (the workspace's revision is compared
per request, so a removed provider is never used again).
#### Self-hosted models
The deployment runs its own model servers next to the cloud providers: the [model server](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MODELS.md) (the
OpenSmartRoute model host - Ollama and OpenAI APIs over llama.cpp, CPU or CUDA; the Ollama image remains a
supported alternative, `ollamaImage` / `OSR_OLLAMA_IMAGE` picks) with a curated set of
open weights on a shared file share (`ollamaEnabled`, on by default on Azure; the compose stacks run the same set,
pulled by `OSR_OLLAMA_MODELS`). Every model is stored and mapped under its ollama.com tag; where that tag's blob is
Ollama's own conversion that a standard llama.cpp cannot read (Qwen3.5, gpt-oss, GLM-4.7-Flash, GLM-OCR,
TranslateGemma), `ollamaModelSources` names the stock GGUF on Hugging Face the model server pulls instead
(`qwen3.5:4b=hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M`), so the whole set is hosted by the OpenSmartRoute model server
without the Ollama image; `ollamaModelImages` stays for a per-model image exception. On CPU profiles
**every model runs as its own always-warm model server** (one model pinned in memory per
server), so a request never waits for another target's weights to be re-read from the share - on a shared
CPU server that eviction churn, not the inference, is the latency. The set is the strongest open model per job -
by the published benchmarks, not by name - that answers a short request in a few seconds on its hardware, so the
on-prem and specialist targets of the catalogue execute without data leaving the deployment:
| Target | Model | Job |
|---|---|---|
| `llm-onprem` | `qwen3.5:4b` | Private chat, reasoning, extraction and drafting: Qwen 3.5, 201 languages, image input, tool calls, thinking, 256K context. |
| `llm-small`, `llm-mid`, `llm-frontier` | `qwen3.5:4b` on CPU; `gpt-oss:20b` / `glm-4.7-flash` for the two upper tiers on a dedicated or GPU profile; `gpt-oss:120b` / `deepseek-v4-flash` through Ollama Cloud with a key | The cloud tiers of the catalogue. Every request stays on open weights; the dedicated-node and Ollama Cloud sets lift the upper tiers where they apply, and an operator who wants closed models connects them in the console or switches `azureOpenAiEnabled` on. |
| `writer-small` | `qwen3.5:4b` | Content - posts, emails, product copy - and translation. |
| `support-agent` | `qwen3.5:4b` | Fast support replies and classification. |
| `coder-small`, `coding-agent` | `qwen2.5-coder:7b` (`glm-4.7-flash` on a dedicated or GPU profile) | Code generation, explanation and repair; GLM-4.7-Flash is the 30B-A3B mixture-of-experts that leads the 30B class on SWE-bench Verified and τ²-Bench. |
| `vision-ocr` | `glm-ocr` | OCR and document understanding - text, tables, formulas, figures, seals - #1 on OmniDocBench 1.5 with 0.9B parameters. |
| `skill-translate` | `translategemma:4b` | Translation between 55 languages (Gemma 3 based). |
| `research-agent` | `deepseek-r1:8b` (dedicated or GPU profiles) | DeepSeek-R1-0528 distilled into Qwen3-8B: visible chain-of-thought for research and analysis - thinking answers take their time by design. |
| `embed-small` | `qwen3-embedding:4b` | Multilingual text embeddings (2560 dimensions, 100+ languages) for `POST /v1/embeddings`; one forward pass answers fast even on CPU. |
| `gpt-oss` | `gpt-oss:20b` (dedicated or GPU profiles) | OpenAI's open-weight reasoning model - a mixture-of-experts whose ~3B active parameters decode at CPU speed once the weights sit in RAM. |
The frontier open weights no single node hosts - Kimi K2.7 Code, DeepSeek-V4-Flash, gpt-oss-120b - answer through
**Ollama Cloud** (`https://ollama.com/v1`, the same OpenAI-compatible surface) when `ollamaCloudApiKey`
(`OSR_OLLAMA_CLOUD_API_KEY`, an API key from ollama.com/settings/keys) is set: `ollamaCloudTargets` maps
`coding-agent`, `research-agent`, `gpt-oss`, `llm-mid` and `llm-frontier` to them and replaces the local mapping of
those targets; without the key they stay on the local models. Any other ollama.com model is one console mapping
away (provider `ollama-cloud`).
`ollamaWorkloadProfile` picks the hardware. `Consumption` (the default) serves the small set: one always-warm
4 vCPU / 8 GiB app per model, each answering one generation at a time and scaling out to `ollamaMaxReplicas`
(default 4, `OSR_OLLAMA_MAX_REPLICAS`) on bursts. A dedicated size (`E4`, `E8`, ... - E-series memory fits the
19 GB GLM-4.7-Flash) adds the `ollamaLargeTargets` models, **one node per model**: only models that earn a whole
node are in that set - the mixture-of-experts coder and `gpt-oss:20b`, whose ~3B active parameters decode at CPU
speed once the weights sit in RAM, and the visible-reasoning `deepseek-r1:8b`, whose answers are slow by design.
A node costs the same whether it holds one model or four, and Ollama runs each loaded model in its own runner, so
`ollamaNodeGroups` packs models that fit together onto one always-warm server per node with every model resident
and preloaded (`OLLAMA_MAX_LOADED_MODELS` = the group size): a short OCR or embedding call runs beside a long
reasoning generation instead of waiting for it, and each grouped model keeps its own provider name
(`ollama-`) so console mappings and health rows do not move. What limits a group is memory - the resident
working sets must fit the node (about 22 GiB of an E4's 28; the per-model `WorkingSetBytes` metric is the figure
to add up). The default keeps the two 22 GiB models (`glm-4.7-flash`, `gpt-oss:20b`) on nodes of their own and
packs `deepseek-r1:8b` with `translategemma:4b`, `glm-ocr` and `qwen3-embedding:4b`; `qwen3.5:4b`, the busiest
model, stays a Consumption replica that scales out on bursts - three nodes and one replica instead of four nodes
and four replicas, every model as warm as before. `OSR_OLLAMA_NODE_GROUPS` (a JSON array of
`{"name": ..., "models": [...]}`) overrides the grouping; a model named by no group keeps the per-model layout.
A model server that drops out of the template when a model moves into a group is retired by the
post-provision hook (`OSR_OLLAMA_PRUNE=false` keeps it).
Dense 9-12B chat upgrades are deliberately not served on CPU (a dense model that size cannot answer a short
request in seconds there); they return with a serverless GPU (`Consumption-GPU-NC8as-T4`, `Consumption-GPU-NC24-A100`; needs GPU
quota in a region that offers it), where one shared server keeps several models resident and decodes 8 / 32
requests at once per replica (the T4 at 8K context with the KV cache in `q8_0`, the A100 the whole set at 16K),
scaling out to `ollamaMaxReplicas`; the replicas share the model store and pull weights under a
lock. `ollamaTargets` / `ollamaLargeTargets` change the mapping; the model tags are pulled at start-up and kept
on the share, and `GET /api/v1/providers` shows which targets the servers answer for. **Azure OpenAI is opt-in**:
`azureOpenAiEnabled` (`OSR_AZURE_OPENAI_ENABLED`, default `false`) creates the account and the `openAiModels`
deployments and hands the `azure` provider every target the Ollama sets leave unmapped - which also gives image
generation (`tool-image-gen` → `gpt-image-2`) and, as the fallback for `transcribe-small`, `gpt-transcribe` a
home; off, those two wait for a console mapping (the self-hosted speech server covers transcription when
`voiceEnabled`). Ollama serves text, chat, code, vision (image input) and embeddings.
**Sizing a model server.** The numbers behind those profiles are arithmetic, not folklore, and the SDK does it
for you (`opensmartroute.inference`, `osr capacity`). A model in memory is its weights (parameters × bits per
weight: a `Q4_K_M` file is about 4.85 bits, `Q8_0` 8.5, `F16` 16, `gpt-oss`'s `MXFP4` 4.25) plus a KV cache the
server reserves for `OLLAMA_CONTEXT_LENGTH × OLLAMA_NUM_PARALLEL` tokens (2 × layers × KV heads × head size ×
2 bytes per token in `f16`; `q8_0` halves it), plus 5-20 % of runtime buffers. On a CPU, generation is
memory-bandwidth bound - every token re-reads the active weights, so a dense 8B `Q4_K_M` model (4.8 GB) decodes at
about 4 tokens/s on a 20 GB/s node while a 30B-A3B mixture reads only its ~3B active parameters and runs like a
3B model once its 18 GB sits in RAM - and prompt evaluation is compute bound (about 2 FLOPs per active parameter
per token). A cold load streams the weights from the model store, which on a network share is the 40-150 s
that used to dominate answers. `osr capacity show qwen3.5:4b glm-4.7-flash --memory 28Gi --cores 4` prints the
footprint, whether it fits after the OS headroom, tokens/s and the warm and cold latency of a typical request;
`osr capacity plan --memory 28Gi [--hosts N] [--shrink]` packs a set onto hosts largest first,
shrinks a model that does not fit (coarser KV cache, shorter context, fewer slots) when asked, names what fits
nowhere and prints the `OLLAMA_MAX_LOADED_MODELS` / `OLLAMA_NUM_PARALLEL` / `OLLAMA_CONTEXT_LENGTH` /
`OLLAMA_KV_CACHE_TYPE` / `OLLAMA_KEEP_ALIVE` / `OLLAMA_MAX_QUEUE` lines that pin exactly that set (`--strict`
exits 2 when a model is unplaced, for CI). With `--url http://:11434` the shapes come from the server's
`/api/show` (exact layer and head counts, file size, trained context) instead of the tag heuristic, and
`osr capacity ps --url ...` lists what it holds in memory right now, on CPU or GPU, resident or expiring.
Defaults - 8K context, one slot, `f16`, 12 % headroom, 20 GB/s, 40 GFLOP/s per core, 0.3 GB/s cold reads - are
`OSR_INFERENCE_*` settings. The test suite runs the curated set above through the planner against the 8 GiB
Consumption app and the 14 / 28 GiB dedicated nodes, so a model that stops fitting its app fails CI before it
fails a deployment.
**Warm models, not cold loads.** The measured latency of a self-hosted answer was never the inference - it was
the cold load, so the platform now removes it twice over. Every model server loads its set into memory as soon
as the weights are pulled (`OSR_OLLAMA_PRELOAD=1` on the per-model CPU apps and the A100, where the whole set
stays resident) and keeps it there (`OLLAMA_KEEP_ALIVE=-1`). Independently, the providers owner asks every
self-hosted Ollama provider what it holds (`/api/ps`) on each health pass and, with
`OSR_PLATFORM_PROVIDERS_KEEP_WARM_MAX` > 0 (Azure sets it to 1 on CPU profiles and to the resident count on a
GPU), loads any mapped model that is not resident - but only when the provider maps at most that many distinct
models, so a server with more models than memory is never made to evict them in turn. `GET /api/v1/providers`
shows the result: `providers[].loaded` lists what each server holds and `models[].warm` says whether a target's
upstream model answers from memory (`null` when the endpoint cannot say); the operator's *Check* records the same.
`osr capacity ps --url` is the command-line view of the same fact. Voice is a second self-hosted server: `voiceEnabled`
(`OSR_VOICE_ENABLED`, on by default) adds `osr-voice`, a [Speaches](https://speaches.ai) container
(faster-whisper speech to text, Kokoro text to speech, OpenAI-compatible `/v1/audio/*`, CPU, weights preloaded
into its own share) and maps the `transcribe-small` / `speak-small` targets to it; `voiceSttModel`
(`OSR_VOICE_STT_MODEL`, default Whisper large-v3-turbo - the accuracy of large-v3 with a 4-layer decoder, on
4 vCPU / 8 GiB; `Systran/faster-whisper-small` runs on 2 vCPU / 4 GiB) and `voiceTtsModel` pick the models -
the compose stacks do the same with `COMPOSE_PROFILES=llm,voice`. With voice off, map `speak-small` to OpenAI or
Azure OpenAI in the console (`gpt-4o-mini-tts`); the OpenAI Realtime API (a WebSocket session, not a request the
router can score) and video generation always need a cloud provider called directly. The router decides
which target answers either way.
**The router ranks on what the deployment measured and bills, not on the catalogue's word.** A CPU-hosted 4B model
writes about five tokens a second: a one-line classification in ten seconds, an article in minutes. The reference
catalogue declared it at 1.2 s and at five times the price of the small cloud model, so the router, the quotes and
the latency SLO all read a fiction - production routed almost nothing to the six self-hosted servers while the
platform's own writers paid the cloud model. Every routing process now calibrates its executable models every
`OSR_PLATFORM_CALIBRATION_INTERVAL_S` seconds (300; 0 leaves the declared figures) from the last
`OSR_PLATFORM_CALIBRATION_DAYS` (7) of answered requests: the median latency becomes the target's `latency_ms`, and
when enough answers carry a token count and the time grows with it, a least-squares line splits it into a fixed
part and a per-token part (`latency_per_token_ms`), so the router expects the length of *each* answer - the slow
box keeps short requests and loses long drafts, and `max_latency_ms` rejects on the same expectation. The price
becomes the reference card of the model the mapping names (the card the estimate and the usage pricing already
bill) when the catalogue has one; a local model no card prices keeps its declared price, which is what the
deployment bills. The declared figures stay under the target's `metadata.declared`, the pass records itself under
`metadata.calibrated` (samples, fit, where the price came from), `GET /api/v1/targets` shows both, the rank trace
carries the per-request estimate and `GET /api/v1/learning` -> `calibration` lists what the last pass set. Five
answered requests are the least a target needs before its measurement replaces the catalogue.
#### Capacity: admission control and overflow
A model server has a fixed number of generations it can run at once - one on a CPU replica, 8 or 32 on a GPU -
while a burst can bring thousands of chat completions in the same second. The platform therefore never queues
requests on the model server. Every provider carries `concurrency` (the console field or the file key; `0` =
unbounded, the default for cloud endpoints) and `queue_timeout_s` (file key, default 20 s): that many calls run
on the endpoint at once, the next ones wait up to the timeout for a slot, and a call that gets none is
**re-routed** by the fallback loop to the next executable target - a chat pinned to `llm-onprem` with the
self-hosted server saturated answers from a cloud tier and says so in `opensmartroute.fallback_from`; `osr.fallbacks:
false` or a candidate list that is entirely saturated returns `503` with `Retry-After` instead. The re-route is a
routing decision like the first one, so every policy still applies: a request with `constraints.data_boundary:
on_prem` (or a tenant / workspace policy that sets it) is **never** spilled to a cloud target, even when the caller
listed cloud models as fallbacks - it waits for a slot and then answers `503`. The Azure template sets the
Ollama provider's `concurrency` to replicas x parallel slots automatically; the compose stacks set it
next to `OSR_OLLAMA_NUM_PARALLEL`. The gates are per process and visible in
`GET /api/v1/admin/health` (`admission[]`: `concurrency`, `inflight`, `waiting`, `admitted`, `rejected`, `peak`,
`avg_hold_ms`) and `GET /api/v1/admin/providers`.
The API side holds `OSR_PLATFORM_MAX_INFLIGHT` requests per process (default 10 000): route handlers run on the
worker-thread pool, which is widened to that number at start-up (the framework default is 40 threads, i.e. 40
concurrent completions per replica), and the HTTP server answers `503` beyond it rather than queueing sockets. In
cluster mode the API and the `openai` service scale on in-flight requests (`apiMaxReplicas`), so ten thousand
concurrent requests are spread over replicas, admitted to the model servers up to their capacity and spilled
over to the cloud tiers for the rest - the routing decision, trace, budget and quota logic is identical on both
paths.
**Stopping without dropping a request.** A replica that receives its stop signal (a deploy, a scale-in, a node
move) does not close its listener at once: `GET /readyz` answers `503` with `status: draining` so the load
balancer moves new requests to the other replicas, while this one keeps accepting for `OSR_PLATFORM_DRAIN_S`
seconds (default 10 - the readiness probe's period times its failure threshold); then the listener closes and the
requests still in flight - a long streamed completion included - get `OSR_PLATFORM_GRACEFUL_SHUTDOWN_S` seconds
(default 120) to finish before the process stops. `/healthz` stays `200` throughout: the process is healthy, it is
leaving. The Azure template keeps at least `apiMinReplicas` API replicas (default 2) in cluster mode so one can
leave while the other answers, sets the container's termination grace period to cover both windows and probes
readiness every 5 seconds. Set `OSR_PLATFORM_DRAIN_S=0` on a laptop to stop at once.
### Response headers
| Header | On | Meaning |
|---|---|---|
| `X-Request-Id` | every response | The id to quote in feedback and support requests; echoed back when you send one. The same id (and your W3C `traceparent`, when you send one) is forwarded to the provider that answers an executed request, so one id follows a request from your logs through the platform into the provider's. |
| `X-OSR-Data-Protection` | `/api/v1/route`, `/v1/chat/completions` | The data-protection mode that acted on the request (`detect`, `mask`, `block`), only when personal or health data was found (section 9). |
| `Server-Timing` | every response | `app;dur=` - time spent inside the platform. |
| `X-OSR-Target` | `/v1/chat/completions`, `/v1/embeddings`, `/v1/audio/*`, `/v1/images/generations` | The target id that answered. |
| `X-OSR-Confidence` | `/v1/chat/completions` | The router's confidence in that choice. |
| `X-OSR-Charge-USD` | `/api/v1/route` (executed), `/v1/chat/completions` | What the workspace is charged for the request under the price version in force (see *Usage pricing*). |
| `X-Wallet-Balance`, `X-Wallet-Available` | `402` | The credits balance of a wallet-billed workspace in cents, and what is left after the charges not yet settled and the holds of requests in flight. |
| `X-OSR-Provider`, `X-OSR-Fallback-From` | `/v1/audio/speech` | The `provider/model` that produced the audio and the targets that failed before it (the body is audio, so the decision travels in headers). |
| `Retry-After` | `429`, `503` | Seconds until the per-minute window, the monthly quota or the budget period resets; on `503`, when every candidate model server was at capacity, the estimated wait for a slot. |
| `X-Quota-Limit`, `X-Quota-Used`, `X-Quota-Reset` | `429` (quota) | Your plan's requests per month, how many the workspace used, and the unix time the month rolls over. |
| `X-Budget-Limit`, `X-Budget-Used`, `X-Budget-Period` | `429` (budget) | The exhausted workspace, tenant or key budget in USD (used includes the holds of requests in flight) and its period (`daily` / `monthly`). |
| `X-Upgrade: true` | `403`, `429` (quota) | The feature exists but is not in your plan, or a bigger plan would keep routing. |
| `X-Edition: enterprise` | `403` | Your plan includes the feature (tenants, statistics, audit) but this deployment runs the community image, which cannot serve it. |
| `X-Key-Scope` | `403` | The API key was created with scopes and lacks the named one (`read`, `route`, `execute`, `feedback`, `manage`). |
| `ETag`, `Cache-Control` | public catalogue reads | Conditional requests: send `If-None-Match` and receive `304` when nothing changed. |
## 4. OpenAI-compatible endpoint
Point an OpenAI SDK at `$OSR_URL/v1` and set `model` to `auto`. The router chooses the target per
request, runs it and returns a standard `chat.completion` whose `model` field is the target that
answered. Decision metadata is added in an `opensmartroute` object that other clients ignore.
```python
from openai import OpenAI
client = OpenAI(base_url=f"{OSR_URL}/v1", api_key=OSR_API_KEY)
reply = client.chat.completions.create(
model="auto",
messages=[{"role": "user", "content": "Write a haiku about routing."}],
)
print(reply.model) # e.g. "llm-small" - the target the router chose
print(reply.choices[0].message.content)
```
```json
"opensmartroute": {
"request_id": "7f3c...", "target": "llm-small", "kind": "llm", "confidence": 0.88,
"plan": [], "alternatives": ["llm-mid"], "cost_usd": 0.00012, "latency_ms": 412.5,
"fallback_from": [], "tokens_saved": 0
}
```
`plan` lists the persona, skill and tool targets the router composed with the model (tool slots are offered to the
model as functions and run by the platform when the model calls them, unless the request brought its own `tools`);
`tokens_saved` is the estimated prompt saving of the compaction step (whitespace, repeated lines and stale history
removed before the model was called - only present when it saved anything).
Options outside the OpenAI schema go in the request body and are ignored by other providers:
| Field | Meaning |
|---|---|
| `model` | `auto` / `osr/auto` (default) lets the router choose; `osr/auto:cheap` / `:fast` / `:quality` / `:private` / `:agentic` / `:llm` preset the objective, a data-boundary constraint or the allowed kinds (explicit `osr.*` options win); a target id pins that target. |
| `models` | Candidate list: the router chooses among them and falls back down the list when a target fails. |
| `osr.objective` / `osr.constraints` | Same as on `/api/v1/route`. |
| `osr.tenant` | Tenant slug (or the `X-OSR-Tenant` header); applies the tenant's constraints (**enterprise**). |
| `osr.plan` | `true`/`false` to force or suppress plan composition; omitted = whatever the plan tier allows. |
| `osr.plan_mode` | `sequential` / `parallel`: how the plan's skills and tool calls run (see `plan_mode` on `POST /api/v1/route`). |
| `osr.fallbacks` | `false` disables the automatic retry on the next best target (default `true`). |
| `stream` | `true` streams server-sent events in the OpenAI format: each provider token is relayed as it arrives; the last chunk carries `usage` and the `opensmartroute` metadata. Quota, routing and provider errors before the first token are ordinary HTTP errors; a fallback to another target happens only before the first token. |
| `tools`, `tool_choice`, `response_format`, `stop`, `n`, `seed`, `logprobs`, `top_logprobs`, `parallel_tool_calls` | Forwarded to the provider unchanged. A tool call comes back as `message.tool_calls` with `finish_reason: "tool_calls"`; send the `tool` turn back in `messages` (with `tool_call_id`) and the model finishes the answer. Unknown fields are passed through too. |
Message `content` may be a list of parts: text parts form the request, `image_url` / `input_audio` / `file`
parts reach the provider unchanged and make the router prefer multimodal targets (`ModalitySignal`).
`GET /v1/models` lists `auto`, `osr/auto`, its `:cheap` / `:fast` / `:quality` / `:private` / `:agentic` / `:llm`
variants, every executable target and every catalogue model on offer by the token (OpenRouter's shape with the
price this workspace pays; `model: "openai/gpt-4.1"` routes among its endpoints, `"openai/gpt-4.1@azure"` pins one -
see *Models by the token* under section 6).
`POST /v1/embeddings` serves the OpenAI embeddings API on the same key: `model: "auto"` routes among the
targets tagged `embeddings` (`capabilities.tags: [embeddings]`, or `metadata.embeddings: true`) that a
provider maps; a target id pins one. Requests are metered like chat completions and the `opensmartroute`
metadata block names the target and its cost. Without an embeddings-capable target the endpoint answers `501`.
### Prompt registry
The instruction a request uses can live in the workspace instead of in the calling code. `/api/v1/prompts` keeps
prompts by `name` with immutable numbered versions (`POST /api/v1/prompts` adds version 1 of a new name or the
next version of an existing one), **labels** that deploy a version (`production`, `staging`; a label names one
version, so `PUT /api/v1/prompts/{name}/{version}/labels` moves it - that is the release), `{{variable}}`
placeholders with optional defaults (`{{tone|calm}}`; the variables a version uses are recorded with it) and a
preview (`POST /api/v1/prompts/{name}/render` with `version` or `label` and `variables`) that renders without
routing. Two kinds: a **text** prompt becomes the system instruction, in front of the request's own; a **chat**
prompt (`[{role, content}]`) puts its system turns there and prepends its user / assistant turns to the history
as examples. A request names one in `osr.prompt` (chat completions) or `prompt` (`/api/v1/route`):
`{"name": "support-reply", "label": "production", "variables": {"company": "Acme"}}`; `version` pins a number,
neither label nor version means the newest. A missing variable without a default is a `400` naming it; an unknown
prompt or label a `404`, before anything is routed or metered. The resolved `name@version` is on the response
(`X-OSR-Prompt`) and in the request's context, so the trace and the request log show which wording acted; the
guard and the data-protection pass see the rendered instruction as the model will. `GET /api/v1/prompts` lists
names with newest version, version count and labels; `GET /api/v1/prompts/{name}` the versions with content;
`DELETE` removes a version or a name; the workspace export carries every version. Up to 200 prompts of 100
versions each; writes take the `manage` scope and are audited (`prompt.version_created`, `prompt.labels_changed`,
`prompt.version_deleted`, `prompt.deleted`). The usage row of a request that named a prompt keeps the
tag under `facts.prompt`, and `GET /api/v1/prompts/{name}/stats` (`?days=`, 1-90, default 30) compares the versions
on it: per version the requests, the answered share, tokens, cost, median latency and the outcomes reported on them
(success rate, mean quality) - the figures a prompt change is judged by, next to each version on the dashboard.
Dashboard: *Prompts*.
### Conversations and memory
The desktop app, the browser extension's side panel and the playground share one chat path, so the platform
gives all of them the same three things at once. Two `osr` options on `POST /v1/chat/completions` turn them on:
| Option | Effect |
|---|---|
| `osr.conversation: true` | Start a conversation the platform keeps. The history the client sends with this first message is imported once; the `conversation` object of the answer's `opensmartroute` block names the conversation (`id`). |
| `osr.conversation_id` | Continue that conversation: send **only the new message**. The platform prepends the last 12 turns verbatim and, as a system note, a rolling summary of everything older (extractive - one line per turn that left the window, never invented), so a thread never outgrows the model's context and the same thread continues from any client of the workspace. An id the caller does not own is `404`. |
| `osr.memory` | Place the long-term memory in front of the request and learn from this message (`true` / `false`; unset = the subject's switch, on by default). Works with or without a conversation. |
Every answer then carries a `conversation` object in its `opensmartroute` block: `id`, `new`, `turns`, `topic_shift` (the question shared
no vocabulary or domain with the recent turns and was not a follow-up - a client offers *Move to a new chat*),
`resumed` and `idle_s` (the conversation had been idle for hours), `remembered` (facts learned from this message,
with their kind), `forgot` (memories removed on request), `memories_used` and `memory` (the switch). The
conversation id is also the `X-OSR-Conversation-Id` response header.
**Long-term memory is rules, not a model.** Facts a person states about themselves are extracted from their
own messages - *my name is …*, *I work at …*, *I'm a data engineer*, *I'm based in …*, *we use Python and dbt*,
*I prefer short answers*, *always answer in French*, *I'm working on …*, *remember that …* - and stored per
subject as `name`, `role`, `employer`, `location`, `language`, `preference`, `uses`, `project` or `note`
(one value for name, location and language; at most 200 memories, the least used dropped first). Before a
request the identity facts, the memories whose words the request mentions and the freshest notes (12 at most)
are written into the system message as *What you know about the person you are helping*; *forget that I work
at Acme* removes the matching memory. Nothing is inferred, summarised by a model or shared across subjects.
**Whose memory.** A personal workspace is one person: its memory and conversations belong to the workspace,
whatever key or device is used. Inside an organization each signed-in user has their own; an API key has its
own (`key`), because the platform cannot tell who holds a shared key. `GET /api/v1/memory` says which
(`subject`: `workspace`, `user` or `key`).
| Endpoint | Purpose |
|---|---|
| `GET /api/v1/conversations` | The caller's conversations, newest first (`?client=desktop`, `?q=` searches titles and summaries). |
| `GET /api/v1/conversations/{conversation_id}` | One conversation: title, summary, the stored turns (`turns_list` with target, request id and cost per answer, so feedback keeps working after a thread is reopened). |
| `PATCH /api/v1/conversations/{conversation_id}` | Rename (`{title}`); the desktop and the extension show the same title. |
| `POST /api/v1/conversations/{conversation_id}/split` | A topic change the person confirmed: the turns from `from_seq` on become their own conversation. |
| `DELETE /api/v1/conversations/{conversation_id}`, `DELETE /api/v1/conversations` | Delete one, or every conversation of the subject. |
| `GET /api/v1/memory` | The switch, the subject and every memory (kind, content, uses, disabled). |
| `PUT /api/v1/memory/settings` | `{enabled}` - off: nothing is placed and nothing is learned (stateless chat). |
| `POST /api/v1/memory` | Add a fact by hand (`{content, kind?}`; `409` when already known). |
| `PATCH /api/v1/memory/{memory_id}` | Edit (`content`) or pause (`disabled`) a memory; a paused fact is never re-learned. |
| `DELETE /api/v1/memory/{memory_id}`, `DELETE /api/v1/memory` | Forget one, or everything. |
Conversations and memories are deleted with the workspace. The dashboard's Memory page
(`/platform/dashboard/memory`) lists both; the desktop app's Settings and the extension's settings page show
the memory list with the same switch.
### Knowledge graph and context engineering
Memory remembers what was said; the **knowledge graph** keeps *facts* - typed, sourced, weighted, expiring
triples in one graph per workspace (the SDK's [`opensmartroute.knowledge`](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#knowledge-graph-and-context-engineering),
stored in the platform database). It fills itself from the traffic it serves: what a person states about
themselves (`person:` - `works_at`, `has_role`, `located_in`, `speaks`, `prefers`, `uses`,
`works_on`, `noted`, `mentions`), what their requests were about and where they were routed
(`asked_about`, `routed_to`), what a live web search cited (`cites`, expiring) and what the workspace adds by
hand. Before every routed chat the graph's context engineer chooses, ranks and trims what the model should
know into a token budget (600 tokens by default, per subject) and accounts for every line; the answer's
`opensmartroute.knowledge` block says how many tokens were placed, which sections, what was dropped and which
facts were learned from the message. The personal section is placed by the graph only when conversation
memory is off - otherwise memory already put those facts in front of the model and the graph adds what memory
cannot: the workspace's shared facts that match the request. A person's facts are never shown to another
subject of the same organization.
**Self-managing.** The process that owns the `knowledge` domain (the API, or the `knowledge` service in
cluster mode) runs a maintenance pass every `OSR_PLATFORM_KNOWLEDGE_MAINTAIN_S` (an hour by default; one
replica per interval): unconfirmed facts lose confidence and are dropped under the floor, expired citations go,
and when TensorFlow is on the image a TransE link predictor
([`KnowledgeEmbedding`](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md#knowledge-graph-and-context-engineering)) is refitted over each active
workspace's shared facts and stored - `GET /api/v1/knowledge/suggest` answers from it (*which domain is this
target probably good at that nobody declared*), falling back to frequency over peers without it.
| Endpoint | Purpose |
|---|---|
| `GET /api/v1/knowledge` | The switch and budget, the subject's entity, graph statistics (facts, entities, per predicate, per source), the trained model's report, the predicate vocabulary and the export formats. |
| `PUT /api/v1/knowledge/settings` | `{enabled?, budget_tokens?}` - off: nothing is placed and nothing is learned; the budget (0-4000) bounds the engineered context. |
| `GET /api/v1/knowledge/facts` | The facts the caller may see - their own personal facts and the workspace's shared ones (`?q=` word search, `?predicate=`, `?personal=true|false`, `?limit=`). |
| `POST /api/v1/knowledge/facts` | Add a fact by hand (`{predicate, object, subject?, confidence?}`; the subject defaults to the person, a shared fact names any non-person entity such as `target:llm-small`). |
| `DELETE /api/v1/knowledge/facts` | Delete by `subject`, `predicate` and/or `object` (query parameters; another person's facts are refused). |
| `DELETE /api/v1/knowledge` | Delete every personal fact of the subject. |
| `POST /api/v1/knowledge/forget` | `{phrase}` - *forget that I work at Acme* drops the personal facts the phrase names. |
| `POST /api/v1/knowledge/context` | `{text, budget_tokens?}` - preview the engineered context the graph would place for a message: the text, every line placed with its score, tokens and source, and what was dropped and why. |
| `GET /api/v1/knowledge/suggest` | `?subject=&predicate=&k=` - likely objects for a fact the graph does not hold, from the trained link predictor when there is one. |
| `GET /api/v1/knowledge/export` | `?format=jsonld|ntriples|graphml|cypher` (`&personal=false` leaves the personal facts out) - the graph for rdflib / Jena / GraphDB, Gephi / NetworkX or Neo4j / Memgraph. |
| `POST /api/v1/knowledge/train` | Fit the link predictor now (`409` without TensorFlow, `400` with fewer than 24 shared facts). |
The knowledge graph is deleted with the workspace. The dashboard's Memory page (`/platform/dashboard/memory`)
shows it under *Knowledge graph*: the switch and the context budget, the facts the person may see in plain words
with search and delete, *Add a fact*, *Forget that ...*, a preview of the context the graph would place for a
message, the export in the four formats and - when the image has TensorFlow - *Train now*.
### Responses API
`POST /v1/responses` serves the OpenAI **Responses API** - the surface the OpenAI Agents SDK and newer client
code speak - over the same routed chat path and the same key: `input` (a string or a list of message,
`function_call` and `function_call_output` items), `instructions`, flat `tools`, `tool_choice`,
`max_output_tokens`, `text.format` and the `osr` routing options are translated to a chat completion; the
answer is a `response` object (`output` items of type `message` / `function_call`, `usage.input_tokens` /
`output_tokens`, the `opensmartroute` block) and `stream: true` emits the typed events (`response.created`,
`response.output_text.delta`, `response.function_call_arguments.delta`, `response.completed`). Conversation
state is kept per workspace: every response is remembered for `OSR_PLATFORM_RESPONSES_STORE_TTL_S` seconds
(default one hour; `store: false` opts a response out, the answer's `store` says whether it was kept) and a
follow-up turn may name it as `previous_response_id` - its turns come first, the new `input` after. The state
lives on Redis when configured, else in the platform database (`responses` table), so every replica sees it and
it survives a restart either way; a workspace can never resolve another workspace's id, and expired rows are
dropped by the finance pass. An unknown or expired id is refused with `400` telling the caller to send the
full `input`, never silently ignored. Only function tools are proxied: a built-in tool (`web_search`,
`file_search`, `code_interpreter`, ...) is refused with `400` rather than dropped, so a completed response never
implies a tool ran that did not. Metered like a chat completion.
### Anthropic Messages API
`POST /v1/messages` serves the Anthropic **Messages API** - the surface of the Anthropic SDKs, Claude Code and the
agent frameworks built on them - over the same routed chat path and the same key: point the client's base URL at
the platform and send the API key as `Authorization: Bearer` or `x-api-key`. `system` (string or `text` blocks),
`messages` with `text` / `image` (base64 or URL) / `tool_use` / `tool_result` blocks, `tools` with `input_schema`,
`tool_choice` (`auto` / `any` / `tool` / `none`, `disable_parallel_tool_use`), `max_tokens`, `stop_sequences`,
`metadata.user_id` and the `osr` routing options are translated to a chat completion; the answer is a `message`
object (`content` blocks of type `text` / `tool_use`, `stop_reason` `end_turn` / `max_tokens` / `tool_use` /
`stop_sequence`, `usage.input_tokens` / `output_tokens`, the `opensmartroute` block) and `stream: true` emits the
Anthropic events (`message_start`, `content_block_start`, `content_block_delta` with `text_delta` /
`input_json_delta`, `content_block_stop`, `message_delta`, `message_stop`). Errors use the
`{"type": "error", "error": {"type", "message"}}` envelope. `model` is `auto` or a target id as everywhere else.
Extended thinking is accepted and ignored; PDF `document` blocks and server-side tools are refused with `400`.
Metered like a chat completion. `POST /v1/messages/count_tokens` answers `{"input_tokens": n}` from the router's
own token estimate - no provider call, nothing metered, the `route` scope suffices.
### Google Gemini API
`POST /v1beta/models/{model}:{action}` serves the Google **Gemini API** - the surface of the Google GenAI
SDKs and the Gemini CLI - over the routed chat path and the same key, `action` being `generateContent`,
`streamGenerateContent` or `countTokens`: point the client's `api_endpoint` /
`base_url` at the platform and send the key as `x-goog-api-key` (or `Authorization: Bearer`). `contents` with
`text`, `inlineData` / `fileData` (images), `functionCall` and `functionResponse` parts, `systemInstruction`,
`tools` with `functionDeclarations` (Gemini's upper-case schema types become JSON Schema), `toolConfig`
(`AUTO` / `ANY` / `NONE`, `allowedFunctionNames`), `generationConfig` (`temperature`, `maxOutputTokens`,
`stopSequences`, `topP`, `responseMimeType: application/json`) and an `osr` block of routing options translate to a
chat completion; the answer is a `GenerateContentResponse` (`candidates[0].content.parts` of `text` /
`functionCall`, `finishReason` `STOP` / `MAX_TOKENS` / `SAFETY`, `usageMetadata`, `modelVersion`, the
`opensmartroute` block). `:streamGenerateContent?alt=sse` emits the same objects as `data:` frames, the last one
carrying `finishReason` and `usageMetadata`; `:countTokens` answers `{"totalTokens": n}` from the router's estimate
(nothing metered, the `route` scope suffices); `GET /v1beta/models` lists what the key may name. Errors use
Google's `{"error": {"code", "message", "status"}}` envelope. `model` is `auto` or a target id (the SDKs'
`models/` prefix is accepted); the same routes answer under `/v1`. `cachedContent` and Google's built-in tools
(search grounding, code execution) are refused with `400`. Metered like a chat completion.
### Rerank
`POST /v1/rerank` (`query`, `documents` as strings or `{"text": ...}` objects - up to 1 000 -, optional `top_n`,
`return_documents`) answers in the Cohere / Jina shape: `results[]` ordered by `relevance_score` (0-1) with each
document's original `index` and, when asked, its text; `model` is `auto` (the embeddings-capable target the
router picks) or a target id. The query and the documents are embedded in one call and ranked by cosine
similarity, so any embeddings provider gives you a reranker; one metered `rerank` row per call.
### JSON repair
A request whose `response_format` asks for JSON (`json_object` or `json_schema`) gets its answer repaired when the
model left it almost right: a Markdown fence or a sentence around the value, a trailing comma, a string or a
bracket left open when `max_tokens` ran out. The smallest edit that makes the text parse is applied - values are
never invented, a key cut before its value is dropped - and `opensmartroute.json_repaired: true` marks the answer.
On a stream the missing suffix goes out as one more content delta before the finish chunk, so the concatenated
deltas parse; a repair that would have to change bytes the client already holds is not attempted.
### Moderation
`POST /v1/moderations` (`input`: a string or a list of strings / `text` parts) answers in the OpenAI moderation
shape - `results[].flagged`, `categories`, `category_scores` - from the router's own guards rather than a hosted
safety classifier: `prompt_injection` (the instruction-injection lexicon), `router_steering` (sentences that talk
to the router about the request), `confounder_gadget` (token soup appended to inflate complexity), `pii`
(reported under `categories.pii`; `flag_pii: true` also raises `flagged`), `oversize` and `control_characters`.
The `opensmartroute` block of each result lists the reasons and the PII types found. No provider call, nothing
metered; the `route` scope suffices. Use it to screen prompts before spending on a model.
### Voice and images
The same key and the same router serve the OpenAI audio and image APIs, so `client.audio.transcriptions.create`,
`client.audio.speech.create` and `client.images.generate` work unchanged with `model: "auto"`:
| Endpoint | Routes among | Answer |
|---|---|---|
| `POST /v1/audio/transcriptions` (multipart `file` + `model`, optional `language`, `prompt`, `temperature`, `response_format` `json` / `verbose_json` / `text` / `srt` / `vtt`) | targets tagged `transcription` (`capabilities.tags`, or `metadata.transcription: true`) - the catalogue's `transcribe-small` | The transcription object (`text`; `language`, `duration`, `segments` with `verbose_json`; plain text for the text formats) plus `opensmartroute`. Files up to 25 MB; the audio never enters the routed text, only the file name and your `prompt` do. |
| `POST /v1/audio/speech` (`input` up to 4 096 characters, optional `voice`, `response_format` `mp3` / `opus` / `aac` / `flac` / `wav` / `pcm`, `speed`, `instructions`) | targets tagged `speech` (or `metadata.speech: true`) - `speak-small` | The audio bytes with the media type of the format; the decision is in `X-OSR-Target`, `X-OSR-Provider` and `X-OSR-Fallback-From`. |
| `POST /v1/images/generations` (`prompt`, optional `n`, `size`, `quality`, `style`, `response_format` `url` / `b64_json`) | executable image tools (`actions: [image]`) - `tool-image-gen` | The OpenAI images object (`created`, `data[].url` or `b64_json`, `revised_prompt`) plus `opensmartroute`. |
A target id instead of `auto` pins one target (`400` when it cannot do that job, `404` when unknown); without any
capable executable target the endpoint answers `501` and says which tag to add. The voice and image targets
execute once they are mapped to a provider that serves the corresponding OpenAI path - OpenAI, Azure OpenAI, the
self-hosted Speaches server (`voiceEnabled` / the `voice` compose profile) for audio - and are metered under the
endpoints `transcriptions`, `speech` and `images`; a transcription target may declare `metadata.usd_per_minute`
so the cost follows the audio duration the provider reports. Uploads must be an audio container (`flac`, `m4a`,
`mp3`, `mp4`, `mpeg`, `mpga`, `oga`, `ogg`, `opus`, `wav`, `webm`; 25 MB), the file name is reduced to its base
name and the declared content type to an audio or video type; the audio bytes never enter the routed text, the
request memory, the trace or the logs. Calls to the media providers never follow redirects, cap the answer at
64 MB, forward a fixed whitelist of options, and only relay an `audio/*` body as speech. `osr.objective`,
`osr.constraints`, `osr.tenant` and
`osr.fallbacks` apply to speech and images as on chat completions; a transcription (a form upload) carries no
routing options, the workspace policy and the `X-OSR-Tenant` header still apply. Embeddings, transcription and
speech targets are never chosen for a chat completion routed as `auto`; an image tool may be, in which case the
chat answer is the picture as Markdown.
`POST /v1/route` and `POST /v1/feedback` are aliases of `POST /api/v1/route` and `POST /api/v1/feedback`
(same body, same plan gates) for gateways that forward only the `/v1` prefix.
### TypeScript client
`@opensmartroute/sdk` (published on the npm registry with every release -
`npm install @opensmartroute/sdk`) is a zero-dependency TypeScript client for the endpoints above - Node 20+,
Deno, Bun, edge runtimes and browsers, `fetch` based. The same client talks to the hosted platform and to a
self-hosted `osr serve` (`apiPrefix: ""`, because the server mounts the routes at the root). Its version
follows the platform release and its types are generated from that release's OpenAPI document;
the full guide is the [TypeScript SDK](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/sdk-typescript.md) page.
```ts
import { OpenSmartRoute, RateLimitError } from "@opensmartroute/sdk";
const osr = new OpenSmartRoute({ baseUrl: "https://opensmartroute.ai", apiKey: process.env.OSR_API_KEY! });
const decision = await osr.route({ text: "Refund order #123", objective: { cost: 0.6 }, top_k: 3 });
await osr.feedback({ request_id: decision.request_id, target_id: decision.target.id, success: true, quality: 0.9 });
const reply = await osr.chat.completions.create({ model: "auto", messages: [{ role: "user", content: "Hi" }] });
console.log(reply.model, reply.opensmartroute.confidence); // typed `OsrMetadata`
const stream = await osr.chat.completions.create({ messages: [{ role: "user", content: "Hi" }], stream: true });
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta.content ?? "");
const local = new OpenSmartRoute({ baseUrl: "http://localhost:8000", apiKey: "my-token", apiPrefix: "" });
```
| Method | Endpoint |
|---|---|
| `route(req)`, `feedback(outcome)`, `estimate(req)` | `POST /api/v1/route`, `/feedback`, `/estimate` |
| `targets()`, `me()`, `quota()`, `info()`, `health()`, `whoami()` | `GET /api/v1/targets`, `/me`, `/quota`, `/info`, `/healthz`, `/whoami` (`osr serve`) |
| `chat.completions.create(body)` | `POST /v1/chat/completions`; `stream: true` returns an `AsyncIterable` of `chat.completion.chunk` objects parsed from the SSE stream (`finalCompletion()` assembles the answer) |
| `embeddings.create(body)`, `models.list()` | `POST /v1/embeddings`, `GET /v1/models` |
| `responses.create(body)` | `POST /v1/responses`; `stream: true` returns the typed Responses events (`finalResponse()` assembles the answer, `responseText()` reads it) |
| `messages.create(body)`, `messages.countTokens(body)` | `POST /v1/messages` (Anthropic Messages API); `stream: true` returns the Anthropic events (`finalMessage()` assembles the answer, `messageText()` reads it); `POST /v1/messages/count_tokens` |
| `audio.speech.create(body)`, `audio.transcriptions.create({ file, ... })`, `images.generate(body)` | `POST /v1/audio/speech` (bytes + headers), `/v1/audio/transcriptions` (multipart), `/v1/images/generations` |
| `rest.get/post/put/patch/del(path, { path, query, body })` | every route of this API, typed from the OpenAPI document (`rest.get("/api/v1/keys")`, `rest.del("/api/v1/keys/{key_id}", { path: { key_id } })`) |
| `keys`, `tenants`, `governance`, `providers`, `notifications`, `support`, `handoffs`, `evaluations`, `billing`, `marketplace`, `insights`, `memory`, `knowledge` | convenience groups over `rest` for the workspace surface (`keys.create / rotate / revoke`, `tenants.upsert`, `governance.setPolicy`, `providers.check / map`, `notifications.inbox`, `support.open`, `evaluations.run / replay / sample / get`, `billing.invoices`, `marketplace.search`, `insights.usage / savings / audit / sla`, `memory.get / add / forget / conversations`, `knowledge.facts / add / forget / context / suggest / export / train` ...) |
Options: `tenant` (`X-OSR-Tenant`), `app` (`X-OSR-App`), `headers`, `timeoutMs`, a custom `fetch`; per call
`signal`, `requestId` (`X-Request-Id`) and `tenant`. Non-2xx answers raise `OpenSmartRouteError` with `status`,
`detail`, `requestId`, `retryAfter` and a stable `code`; `AuthenticationError` (401), `PermissionDeniedError`
(403, `upgradeRequired`), `NotFoundError` (404) and `RateLimitError` (429, `budget` from the `X-Budget-*`
headers) are subclasses. Request types (`RouteIn`, `FeedbackIn`, `ChatCompletionIn`, `EmbeddingsIn`) are
generated from this API's OpenAPI document; `RouteDecision`, `ChatCompletion`, `ChatCompletionChunk` and
`EmbeddingsResponse` type the answers. The package README documents the development and live-test workflow.
## 5. Feedback
`POST /api/v1/feedback` reports how a routed answer turned out. The learners (Bradley-Terry, IRT,
LinUCB) update on the workspace's own traffic. Included in every plan of the default catalogue
(feature `feedback`).
```bash
curl -s "$OSR_URL/api/v1/feedback" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"request_id": "7f3c...", "target_id": "llm-frontier", "success": true, "quality": 0.9, "latency_ms": 2100}'
```
Fields: `request_id`, `target_id`, `success` (required); `quality` (0..1), `cost_usd`, `latency_ms`,
`domains`, `complexity`, `preferred_over` (the id of a target this one beat in a comparison), `role` (the plan
slot the outcome grades - `persona`, `skill`, `tool` or `llm` - when it is about a companion of a composed answer
rather than the target that answered). The `request_id` must be one the workspace itself routed (`/route`,
`/v1/chat/completions`, `/estimate`, MCP): an unknown id, or another workspace's, is a 404, as is an unknown
`target_id` - a catalogue target, one of the workspace's installed listings, or a published skill / persona a plan
discovered for the request (section 11) are known.
Every report is kept against its request: `GET /api/v1/activity` summarises it per row (`outcome`),
`GET /api/v1/trace/{request_id}` lists the reports (`outcomes`) next to the spans, and in the enterprise
edition the audit chain records an `outcome` entry after the `decision` entry (section 10). The dashboard
has the same form without curl: expand a row on `/platform/dashboard/activity` (or open its trace) and
report good / poor, a quality score and, optionally, the target that would have done better - the latter
is recorded as a pairwise preference for the Bradley-Terry learner.
**The platform reports what it can see itself.** Almost no client calls `/feedback`, so an execution's own
assessment is kept the same way and needs nobody: when the answer was judged (a handler that rates its own
output, or the implicit reading of the answer - a refusal, a truncation, an empty reply), and when a
**fallback** settled a comparison (the target that answered a request another one had just failed beat it
on that request), the outcome is written against the request and published on the `feedback` topic the SLM
service trains from - for the answering target and, with their `role`, for every persona, skill and tool the
plan composed the answer with, so the companions share the answer's judgement. An answer nothing was found
wrong with is not an assessment and is not recorded.
### The human queue
Routing to a person is a real execution, not an error. When a decision lands on a `human` target
(escalation intent, high risk, low automated confidence - the defer strategy scores this), `execute: true`
and the chat proxy **queue a ticket** instead of failing: the answer names the ticket, costs zero tokens,
and the ticket keeps the request text (with PII restored - human targets declare `pii_allowed`) and the
router's reason for deferring in a few plain sentences: what the request reads as (domain, difficulty,
personal data, jailbreak risk), which targets policy ruled out and why, how the human target ranked against
the runner-up, which strategies carried it, and which targets failed before the router re-routed there.
The full decision trace stays one link away.
| Endpoint | What it does |
|---|---|
| `GET /api/v1/handoffs` | The workspace's tickets, newest first, with per-status counts (`?status=pending`). |
| `GET /api/v1/handoffs/{handoff_id}` | One ticket: text, reason, status, answer. |
| `POST /api/v1/handoffs/{handoff_id}/resolve` | A person answers (`{answer, quality?}`); the resolution feeds an outcome to the learners, so deferral itself is learned. |
| `POST /api/v1/handoffs/{handoff_id}/cancel` | Withdraw a pending ticket; recorded as a failed outcome. |
The dashboard's Human queue (`/platform/dashboard/handoffs`) shows how many tickets wait and for how long,
lists each ticket with the request and the reasons side by side and a *Full trace* link into the activity
log, and lets a teammate answer it - the answer goes back as the response and is recorded as the outcome with
a quality in words (*Solved* ... *Could not help much*) - or cancel it, without curl.
### Evaluation
Outcomes say how the router did on the traffic it got; an **evaluation** says how it would do on the prompts
you choose, before you send them. `POST /api/v1/evaluations` takes a labelled dataset - the JSON-lines file
`osr eval` reads, one object per row - and routes every row on the workspace's own pool: its policy, tenants
(`tenant` in the body), allow / deny lists, API-key limits, installed listings and providers all apply, exactly as
on `/route`. **Nothing is executed and nothing is metered**: a run calls no model, costs nothing, counts against
no quota, and leaves no trace or event in the workspace's history (the rows are not traffic). The dataset is
routed and discarded; only the report is kept - the newest 50 runs per workspace, readable at
`/platform/dashboard/evaluation`.
```bash
curl -s "$OSR_URL/api/v1/evaluations" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"name": "support prompts, week 32", "rows": [
{"text": "What is 12 * 37?", "expected": "tool-calculator"},
{"text": "Summarise this clause in plain words: ...", "acceptable": ["llm-mid", "llm-frontier"]},
{"text": "List customers with no orders this year.", "scores": {"skill-sql": 1.0, "llm-mid": 0.7}, "task_type": "sql"},
{"text": "Translate good morning into French."}]}'
```
A row is `text` plus how it is judged: `expected` (the target id that should win), `acceptable` (several ids
that all count), or per-target `scores` in `0..1` (RouterBench style; the best score is the label and the
chosen target's score is its *realised quality*). `context`, `history` and `task_type` travel with the row.
Rows without a label are fine - they still give the mix, the price and the confidence. `objective`
(`{"cost": 0.6}`) and `constraints` are those of `/route`. The report carries:
| Field | Meaning |
|---|---|
| `router.accuracy`, `accuracy_ci` | Share of labelled rows whose chosen target is a hit, with the Wilson 95 % interval for that many rows; `hits` and `labelled` are the counts behind it. |
| `router.mean_quality` | Mean score of the chosen target over the rows that score targets. |
| `router.mean_cost_per_1k`, `mean_latency_ms`, `mean_confidence`, `abstain_rate` | Declared list price and latency of the chosen targets, averaged over the routed rows; the router's confidence in its choice; how often it abstained. |
| `router.ece`, `brier`, `reliability` | Calibration on labelled rows - expected calibration error (0 = the router is right exactly as often as it says), Brier score and the ten confidence bins - reported from 20 labelled rows. |
| `savings` | The mix's mean price against always sending every row to the dearest target the policy admitted (`pool`), as a share. |
| `baselines` | The same rows and the same admissible pool routed by the policies every router must beat - random, always the cheapest, always the strongest `quality_prior`, always the dearest - each with accuracy, interval, quality, price and latency (`baselines: false` skips them). |
| `route_mix`, `by_domain`, `confusion`, `misroutes` | Where the rows went (with per-target accuracy and price), accuracy per first domain, the `expected -> got` pairs and the labelled rows the router got wrong with their confidence. |
| `frontier` | With `frontier: true` (up to 500 rows): accuracy and mean price at cost weights 0, 0.3, 0.6, 1 and 2 - the accuracy-against-spend curve of your own prompts. |
| `decisions` | Every row's decision: the prompt's first words, expected, chosen target and kind, hit / miss / unlabelled, confidence, quality, price, latency, domains and complexity (the first 1,000; the aggregates cover all). |
A run routes at most `OSR_PLATFORM_EVAL_MAX_ROWS` rows (default 2000, `413` beyond it) and for at most two
minutes (`truncated: true` with `evaluated` when it stopped early). `GET /api/v1/evaluations/sample` returns
labelled rows made from the catalogue itself (ontology seed prompts scored by capability fit), so a workspace
can see a report before it has a dataset; the page's *Template* button writes the row format. From the command
line, `osr eval rows.jsonl --remote` (after `osr login`, or with `--url` / `--token`) sends the same file to the
signed-in workspace instead of a local catalogue, prints the headline, the misroutes and the baselines, keeps the
run on the Evaluation page and honours `--min-accuracy` for a CI gate; `--baselines`, `--frontier`, `--limit`,
`--preset` and `--json` mean what they mean locally.
That gate is a few lines in any CI. For GitHub Actions the repository ships a composite action
(`.github/actions/osr-eval/action.yml` - copy it into your own repository, or paste the step below): it installs
the package, runs `osr eval` locally against a catalogue or remotely against the signed-in workspace, fails the
job below `min-accuracy` and writes the accuracy to the job summary and to an `accuracy` output.
```yaml
# .github/workflows/routing.yml - fail the build when a catalogue or rule change lowers routing accuracy
on: [pull_request]
jobs:
routing:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- name: Routing accuracy gate
run: |
pip install "opensmartroute[yaml]"
osr -t targets.yaml -r rules.yaml eval evals/routing.jsonl --baselines --min-accuracy 0.85
- name: The same rows on the hosted workspace # optional: what production would decide
env: { OSR_API_KEY: ${{ secrets.OSR_API_KEY }} }
run: osr eval evals/routing.jsonl --remote --min-accuracy 0.85
```
Azure Pipelines, GitLab, Jenkins and CircleCI run the same two commands; exit status `2` is the failed gate.
`POST /api/v1/evaluations/replay` is the other direction: it re-routes the prompts the workspace kept under its
`log_payloads` policy (section 9; `409` naming the policy when nothing is kept) through today's router - with
everything the learners took from your outcomes, every catalogue and rule change since - and compares each
decision with the target that answered then: `changed` / `kept` and the `moves` between targets, the kept token
counts priced at the new target's list price against the provider cost metered then (`cost`; decision-only
requests carry no tokens and are left out), and `agreement` with the outcomes people reported - `good_kept`
(accepted answers the router still sends the same way), `good_changed`, `bad_changed` (rejected answers it now
sends elsewhere) and `bad_kept`. `days` (1-90) and `limit` bound the window; a prompt masked by the
`redacted` mode is replayed as kept, placeholders included.
| Endpoint | What it does |
|---|---|
| `GET /api/v1/evaluations` | The workspace's kept runs, newest first, each with its headline `summary`; `max_rows`, `keep_runs` and `frontier_max_rows` say what the deployment allows. |
| `POST /api/v1/evaluations` | Route a labelled dataset (`rows`, `name`, `objective`, `constraints`, `tenant`, `baselines`, `frontier`) and keep the report; `route` scope, `201`. |
| `POST /api/v1/evaluations/replay` | Replay the kept prompts of the last `days` through today's router (`limit` newest first) and keep the comparison; `route` scope, `201`, `409` when nothing is kept. |
| `GET /api/v1/evaluations/sample?n=40` | A labelled sample dataset built from the catalogue, with the row format explained. |
| `GET /api/v1/evaluations/{evaluation_id}` | One run with its full `report`. |
| `DELETE /api/v1/evaluations/{evaluation_id}` | Remove a kept run. |
**Datasets from production.** The prompts your users actually send are the dataset that matters, and the outcomes
they reported are its labels. `POST /api/v1/evaluations/datasets` saves logged requests as a kept dataset - the
*Add to dataset* action of the Activity page: name `request_ids`, or take a window over the prompts kept under
`log_payloads` (`days`, `limit`, `outcome` `good` / `bad` / `rated` / `any`, `target_id`), or send explicit
`rows`, or mix the three. A logged request becomes `{text, expected, scores, context, task_type}`: the kept
prompt, the target that answered as `expected` when its outcome was good (with the reported `quality` as its
score), the request id under `context.source` so the same request is never added twice. `GET
/api/v1/evaluations/datasets` lists them (rows and labelled counts), `GET .../datasets/{id}` returns the rows
(`?format=jsonl` downloads the file `osr eval` reads), `POST .../datasets/{id}/rows` adds to one, `POST
.../datasets/{id}/run` evaluates it on today's pool and keeps the report (the body carries the run options -
`objective`, `constraints`, `tenant`, `baselines`, `frontier`; the report names the `dataset_id`), `DELETE`
removes it. A workspace keeps up to 50 datasets of up to 5 000 rows; all of it takes the `route` scope.
**Scores from other frameworks.** A dataset downloads in the shape Ragas, DeepEval and Langfuse read
(`?format=ragas|deepeval|langfuse`; every row carries its `row` index and, for logged requests, the `request_id` in
the metadata), and the scores those tools compute come back with `POST /api/v1/evaluations/datasets/{id}/results`:
`format` names the shape - `ragas` (the result table, `to_pandas().to_dict("records")`: numeric columns are the
metrics), `deepeval` (test results with `metrics_data`: name, score, success), `langfuse` (run items with their
`scores`) or `scores` (plain `{row | request_id | input, scores: {name: value}, passed}` rows from TruLens, Arize
or a script) - `tool` the framework for the record, `threshold` passes a row whose every metric reaches it when the
tool gave no verdict. Rows join by index, request id or prompt text; the run is kept as an `external` evaluation
(per-metric mean / min / max, the pass rate, the per-row scores; 409 when nothing joins), and a row that came from a
logged request receives the mean score as an outcome with source `eval:` (`feedback: false` leaves that out),
so the activity page, the trace and the learners see what the other tool thought.
**The answer judge.** The runs above grade *which target was chosen*; the judge grades *the answer the customer
received*. With a `judge` block in the workspace policy (section 9: `sample`, `target`, `threshold`, `endpoints`,
`targets`, `max_per_day`) a reproducible share of the executed answers is graded by the judge model after the
answer has left, the grade becomes the request's outcome (`feedback` rows with `source: judge:`, read by
the learners, the rankings and the replay comparison like a hand-reported one), and every verdict keeps its
reason. `GET /api/v1/evaluations/judge` (`days`, `limit`) returns the block in force, what was judged per
answering target over the window (judged, graded, good, mean score, cost) and the latest verdicts; `POST
/api/v1/evaluations/judge/run` `{request_id}` grades one kept request now (the Activity page's *Judge this
answer*; `execute` scope, `409` naming `log_payloads` when the prompt and answer were not kept). Judging is
metered as its own endpoint (`judge`) on the workspace's usage, never charged to the request it grades.
Every run is recorded in the workspace's audit trail as `evaluation.run`, travels with the data export
(`GET /api/v1/me/export` -> `workspace.evaluations`) and is deleted with the workspace. A tenant-pinned API key
cannot run evaluations (`403`) - they are the workspace's view of its whole pool. The TypeScript client has them
as `client.evaluations.run / replay / sample / list / get / delete`.
**Quality gates.** An evaluation says how good the router is on your prompts *today*; a gate says whether it
is *still* that good after the catalogue moved, a provider was swapped, a model was retired or somebody edited
the workspace policy - before your own customers notice. Keep the dataset you signed off on as a gate (*Keep as a
quality gate* on the Evaluation page, or `POST /api/v1/evaluations/gates` with the same `rows`, `objective`,
`constraints` and `tenant` as a run) with the thresholds it must hold: `min_accuracy` (an absolute floor, 0..1)
and / or `max_drop` (how many accuracy points a run may fall below the run that set the baseline - the first
pass; default 0.05). The gate is routed at once - that first pass is the baseline - and again every
`OSR_PLATFORM_EVAL_GATES_INTERVAL_S` (six hours; `0` = only on request and after a policy change) and whenever
the workspace policy is replaced or removed. Each pass is a kept evaluation named *Gate · …*, so the full report
is one click away. A pass under the floor is `fail`, one too far under the baseline `regressed`; the first
failing pass files a warning in the workspace inbox (`workspace.evaluation_gate_failed`, linking the report) and
the audit trail (`evaluation.gate_failed`), a scheduled pass that keeps failing stays quiet, and the pass that
clears both thresholds again files `evaluation_gate_recovered`. A run on request or after a policy change always
announces a failure. Nothing is executed or metered. A workspace keeps up to ten gates of up to 500 rows each,
at least five of them labelled; creating and deleting a gate takes the admin role, running one the `route` scope.
| Endpoint | What it does |
|---|---|
| `GET /api/v1/evaluations/gates` | The gates with their last verdict (`last_state`, `last_accuracy`, `last_detail`, `last_reason`), the baseline and the counters; `interval_s`, `max_gates`, `max_rows`, `min_labelled` say what the deployment allows. |
| `POST /api/v1/evaluations/gates` | Keep a dataset as a gate (`name`, `rows`, `objective`, `constraints`, `tenant`, `min_accuracy`, `max_drop`) and route it now; `201` with the judged gate. |
| `POST /api/v1/evaluations/gates/{gate_id}/run` | Route the gate now and judge it. |
| `DELETE /api/v1/evaluations/gates/{gate_id}` | Forget the gate (its past runs stay in the list of evaluations). |
**Shadow routing.** A dataset and a gate replay prompts; a shadow candidate rides the *live* traffic. Start one
(*Shadow routing* on the Evaluation page, or `POST /api/v1/shadow`) with the policy you are considering - objective
weights (`quality`, `cost`, `latency`, `quality_floor`) and / or the request-shaping constraint keys of the
workspace policy (`deny_targets`, `allow_targets`, `max_cost_per_1k`, `max_latency_ms`,
`preferred_max_latency_ms`, `region`, `data_boundary`, `contains_pii`) - and from then on every request the
workspace routes is routed a second time under the candidate, inside the same pool and with the same tenant,
key limits and installed listings. The second decision is never served, executed, metered or charged; what is
kept is two target ids and the cost, quality estimate and latency each would have had (no text). `GET
/api/v1/shadow` sums it up: agreement between the two, the spend delta, the quality and latency deltas, where the
candidate moves traffic from and to, the requests it would have refused, and a plain-words `verdict` once a few
dozen requests are in. When the comparison convinces you, `POST /api/v1/shadow/promote` writes the candidate into
the workspace policy - its constraints are merged, its weights become the policy's `objective`, the default for
every request that sets none (an explicit objective on a request still wins) - which is a policy change, so every
quality gate the workspace keeps runs again on it: a promotion that regresses a kept dataset shows up in the
inbox like any other bad edit. One candidate runs at a time; it ends when promoted, stopped (`DELETE
/api/v1/shadow`) or after `max_decisions` requests (20,000 by default), and earlier candidates stay readable
(`GET /api/v1/shadow/{candidate_id}`). Starting, stopping and promoting take the admin role; a tenant-pinned key
is refused.
| Endpoint | What it does |
|---|---|
| `GET /api/v1/shadow` | The running candidate, its `summary` (agreement, cost / quality / latency deltas, `moves`, `refused`, `verdict`, `progress`), the most recent comparisons and the earlier candidates. |
| `POST /api/v1/shadow` | Start a candidate (`name`, `objective`, `constraints`, `sample` = share of requests shadowed, `max_decisions`); a running one is stopped. |
| `GET /api/v1/shadow/{candidate_id}` | One candidate, running or ended, with its summary and comparisons. |
| `POST /api/v1/shadow/promote` | Make the candidate (`candidate_id`, default the running or latest one) the workspace policy. |
| `DELETE /api/v1/shadow` | Stop the running candidate. |
### Observed usage: the browser extension's ledger
The browser extension (`platform/extension`, product page `/extension`) sees calls a team makes *outside* the router - on ChatGPT, Claude,
Gemini, Copilot, Perplexity and the OpenAI, Anthropic, Google and OpenRouter consoles. With "Share observed
usage" turned on it reports the **facts** of each call and never the text:
```bash
curl -s "$OSR_URL/api/v1/usage/observed" -H "Authorization: Bearer $OSR_API_KEY" -H "X-OSR-App: browser-extension" \
-H "Content-Type: application/json" \
-d '{"items": [{"ts": 1760000000, "source": "chat", "site": "chatgpt.com", "vendor": "openai", "model": "gpt-5",
"prompt_tokens": 1284, "completion_tokens": 310, "task_type": "summarise", "complexity": 0.3}]}'
```
| Endpoint | What it does |
|---|---|
| `POST /api/v1/usage/observed` | Records up to 500 observations (`ts`, `source` = `chat` / `console` / `app`, `model`, optional `vendor`, `site`, token counts, `task_type`, `complexity`). Any other field - a prompt, messages - is refused with `422`. Each row is priced at the observed model's catalogue list price (`estimated_cost_usd`, `null` when the model is unknown) and re-quoted through the router for the same token shape (`recommended_target`, `recommended_cost_usd`). Needs the `feedback` scope. |
| `GET /api/v1/usage/observed` | The stored observations, newest first (`?days=`, `?limit=`). |
| `GET /api/v1/usage/opportunity` | The report: calls, tokens, observed spend, the routed quote for the priced calls, `gap_usd` / `gap_pct`, breakdowns by model, source, task and day, and the ten models with the widest gap with the target the router would have chosen. |
Observations are kept for 90 days and deleted with the workspace. The report compares the same request
shapes on both sides - it is a price difference, not a quality claim - and the dashboard shows it on
`/platform/dashboard/spend`.
### Insights: the figures read back as findings
The dashboard shows numbers; the insights endpoint says what they mean. It reads the workspace's own metered
rows - the window against the one before it, per target and per day - and returns findings, each with the
evidence behind the sentence and the page that acts on it: cost against the previous window with the targets
that moved it, a day whose cost sits far above the average, failures and their rate, speed against the previous
window, one target carrying almost all the traffic, decisions without execution, savings against the baseline
and spend the extension saw outside the router. No model is involved: every sentence is computed, so asking
twice gives the same answer. The dashboard overview carries it as *Ask about your usage*.
| Endpoint | Purpose |
|---|---|
| `GET /api/v1/insights` | `?days=` (1-90, default 7): a `headline`, the `totals` of the window and the one before, the `findings` (`kind`, `severity` = `warning` / `good` / `info`, `title`, `detail`, `evidence`, `action.href`), the `suggestions` people ask and the `method`. |
| `POST /api/v1/insights/ask` | `{question, days?}` - a question in plain words (*why did my cost go up*, *is anything slower*, *where could I save*); the subject is read from the question (`intent`: cost, savings, speed, failures, models, outside, volume) and the matching findings come back with one `answer` sentence. A tenant-pinned key gets its tenant's figures only. |
### Bill analysis: the invoice or usage export you already have
A team that has not routed yet still has a bill. Every provider, gateway and tracing tool exports usage as a
table - a date, a model, request and token counts, often the amount charged - and every provider issues a PDF
invoice. Post either and the platform recognises the columns - or, for an invoice, the vendor, the billing period
and every line that names a model with an amount - aggregates it per day, provider and model and prices every
line twice: what the bill says
(or, when the export carries no amounts, the model's catalogue list price) and what the router's recommended
target would have charged for requests of the same token shape. Nothing in an export is text - dates, names,
quantities and amounts are read, every other column or word (addresses, tax ids, people) is ignored and never
stored.
The layouts recognised (`source_format`) - the tables from their header row, an invoice from its text:
| Format | Where it comes from | What the parser reads |
|---|---|---|
| `openai-usage` | platform.openai.com → Usage → Export | hourly buckets: `n_requests`, `n_context_tokens_total`, `n_generated_tokens_total` per model |
| `openai-costs` | platform.openai.com → Billing → Costs → Export | `line_item` (`gpt-4o, input`) and `amount`; the tokens are backed out of the amount at list price |
| `anthropic-usage` | console.anthropic.com → Usage → Export CSV | `usage_input_tokens_no_cache`, cache reads and writes, `usage_output_tokens` per `model_version` |
| `azure-openai` | Azure Cost Management → Cost analysis → Download | `MeterName` (`gpt-4o-0806 Inp glbl Tokens`), `Quantity` in thousands, `Cost` |
| `google-billing` | Google Cloud Billing → Reports → Download, or the BigQuery billing export | `SKU description` (`Gemini 2.5 Pro Input Tokens`), `Usage amount`, `Cost`, `Service description` |
| `aws-cur` | AWS Cost Explorer CSV or a Cost and Usage Report (Bedrock) | `lineItem/UsageType` (`USE1-Claude3Haiku-input-tokens`), `lineItem/UsageAmount`, `lineItem/UnblendedCost` |
| `openrouter` | openrouter.ai → Activity → Export | one row per generation: `created_at`, `model`, `tokens_prompt`, `tokens_completion`, `cost` |
| `request-log` | LiteLLM spend logs, Langfuse, LangSmith and Helicone exports, this platform's own Activity CSV | one row per request (a timestamp with a clock time, no request count): each row counts as one request |
| `generic` | anything else: CSV, TSV, JSON, JSON lines or an `.xlsx` workbook (first sheet, Excel dates understood) | a date, a model and token or amount columns under any of some sixty header spellings; a file without a model column (this platform's Usage CSV, a finance summary) becomes one line per day |
| `invoice` | the PDF invoice from Billing → Invoices (OpenAI, Anthropic, OpenRouter and Mistral bill through Stripe; Azure, Google Cloud and AWS issue their own), or its text pasted | the issuer, the billing period (`Aug 1 – Aug 31, 2026`, `Billing period: ...`, else the invoice date), and every line that names a model with an amount - `GPT-4o input tokens 12,345,678 $2.50 / 1M $30.86`, `gpt-4o-0806 Inp glbl Tokens 12,340 1K Tokens 0.0025 30.85`, `Vertex AI Gemini 2.5 Pro Input Tokens ... $10.00`; quantities become tokens (Azure's thousands understood), a line without one has its tokens backed out of the amount at list price; totals, tax, credits and payments are not usage, and the notes say when the model lines do not add up to the invoice total |
Cloud bills are roll-ups, so their hourly rows are never counted as requests; a line without a request count is
quoted as requests of about 2 000 input tokens. An invoice dates every line at the start of its billing period
and the projection per month uses the whole period. A scanned invoice (a picture with no text layer) and a
password-protected PDF are refused with what to do instead, and an old `.xls` should be saved as `.xlsx` or CSV.
```bash
curl -s "$OSR_URL/api/v1/usage/imports/upload" -H "Authorization: Bearer $OSR_API_KEY" \
-F "file=@openai-invoice-2026-08.pdf" -F "name=August, OpenAI"
curl -s "$OSR_URL/api/v1/usage/imports/upload" -H "Authorization: Bearer $OSR_API_KEY" \
-F "file=@openai-usage-2026-08.csv" -F "name=August, OpenAI (usage export)"
curl -s "$OSR_URL/api/v1/usage/imports?dry_run=true" -H "Authorization: Bearer $OSR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "one line", "provider": "anthropic",
"text": "date,model,input_tokens,output_tokens,cost\n2026-08-01,claude-sonnet-4,120000,30000,0.81"}'
```
| Endpoint | What it does |
|---|---|
| `POST /api/v1/usage/imports` | Analyses `{name, text, filename?, provider?}` - `text` is the export as the provider gave it (CSV, TSV, JSON or JSON lines, up to 8 MB) or the text of an invoice (its line items with amounts), `provider` names the provider when the rows carry none (`auto` infers it from the model names) - and keeps the analysis unless `?dry_run=true`. The answer is the analysis: `headline` and `story` (the figures in plain sentences), `source_format` (the table above), the columns recognised, totals, `baseline_cost_usd` against `routed_cost_usd`, `savings_usd` / `savings_pct`, the projected monthly figures, `by_provider`, `by_model`, `daily`, the lines (up to 5 000 inline) and `notes` on what was assumed. A file without a date or a token / amount column is refused with `400`, a larger one with `413`. |
| `POST /api/v1/usage/imports/upload` | The same analysis for a multipart upload (`file` plus optional `name` and `provider` fields), for curl, the dashboard and integrations that hold the file rather than its text; the way in for a PDF invoice (read from its text layer with `pypdf`) and for an `.xlsx` workbook. |
| `GET /api/v1/usage/imports` | The analyses kept for the workspace, newest first, with their headline figures and the `formats` and `providers` the parser accepts. |
| `GET /api/v1/usage/imports/samples` | Sample exports in each provider's own layout with invented figures - OpenAI usage and costs, Anthropic, Azure OpenAI, Google Cloud billing, AWS Bedrock, OpenRouter, Mistral, Cohere, xAI, DeepSeek, Groq, Together, LiteLLM, Langfuse, this platform's activity export and the text of an OpenAI invoice (`id`, `provider`, `name`, `source`, `format`, `text`) - to try the analysis before touching a real bill: post one as `text` with `?dry_run=true`. The dashboard's *Try a sample export* menu does exactly that. Every sample is what the parser is tested against. |
| `GET /api/v1/usage/imports/{import_id}` | One analysis with its lines; `?format=csv` downloads the lines, `xlsx` a workbook (summary, by provider, by model, daily, lines), `pdf` the report. |
| `DELETE /api/v1/usage/imports/{import_id}` | Deletes an analysis and its lines (recorded in the audit trail). |
| `GET /api/v1/usage/report` | The workspace's own routed usage as the same report: a `headline` and `story` in sentences, then requests, tokens in / out / cached, provider cost and the workspace's charge per day, model, provider and API key over `?days=` (up to 366, `?tenant=`, `?key_id=`), and the saving against the savings baseline. `?format=csv` downloads the day-model-provider lines, `xlsx` a workbook with every breakdown, `pdf` the report. |
The routed side is the estimator's quote under the catalogue and rules of today, so every figure is labelled an
estimate; lines whose model the router cannot quote are carried over at what they cost today and the notes say
how much of the bill was quoted. Analyses are the customer's own data: kept until they delete them, and deleted
with the workspace.
### Client activity: what the extension and the desktop app did
Most of what the browser extension and the desktop app do happens on the person's own machine and is never a
routed request - the pre-send card, a redaction, a compaction, a writing fix, the command bar opening, an answer
pasted back. Each app keeps a small queue of those moments and posts it to the workspace a few minutes later, so
the dashboard's Apps page (`/platform/dashboard/apps`) shows how the apps are used, what they caught and what
they saved. Every event is **a kind, a hostname, counts and figures - never text**:
```bash
curl -s "$OSR_URL/api/v1/usage/activity" -H "Authorization: Bearer $OSR_API_KEY" -H "X-OSR-App: desktop" \
-H "Content-Type: application/json" \
-d '{"items": [{"ts": 1760000000, "app": "desktop", "kind": "answer.received", "target_id": "gpt-4.1-mini",
"request_id": "6f1c...", "latency_ms": 2300, "version": "1.5.0"},
{"ts": 1760000010, "app": "browser-extension", "kind": "pii.redacted", "site": "chatgpt.com", "n": 2}]}'
```
| Endpoint | What it does |
|---|---|
| `POST /api/v1/usage/activity` | Records up to 500 events (`ts`, `app` = `browser-extension` / `desktop`, `kind` as `area.action`, optional bare-hostname `site`, `target_id`, `request_id`, `n`, `tokens`, `saved_usd`, `latency_ms`, `tier`, `task_type`, `version`). A field of any other name is refused with `422`, a `site` with a path or query with `400`. Needs the `feedback` scope. The plan caps events per UTC day: a batch that would cross the cap answers `429` with `Retry-After` and records nothing. |
| `GET /api/v1/usage/activity` | The events, newest first (`?days=`, `?limit=`, `?before=` to page, `?app=`, `?kind=pii.*`). |
| `GET /api/v1/usage/activity/summary` | Totals per app, kind and site, the per-day series, and what the plan keeps (`retention_days`) and allows (`limit_per_day`, `today`). |
What the plan decides: **Free** keeps events 7 days and records up to 5 000 a day, **Pay as you go** and **Pro**
30 days and 50 000 a day, **Team** 90 days and 500 000 a day, **Enterprise** 365 days with no daily cap; a
workspace's `retention_days` override (custom agreements) wins over the tier. The retention sweep purges each
workspace at its own cutoff. The kinds the first-party clients send are listed in [EXTENSION.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/EXTENSION.md)
and [DESKTOP.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/DESKTOP.md); both apps have a **Log my activity to the workspace** switch (on by default, since
there is no text in the events) and administrators can pin it through the extension's managed storage.
Every request the extension routes through the platform also carries `X-OSR-App: browser-extension`. Any
client may name itself that way (`osr-cli`, `langchain`, your own service name; free text is reduced to a
short slug): the value is stored on each metered row and `GET /api/v1/usage` returns `by_app` - requests,
tokens and cost per app, `null` for calls that sent no header - which the *Usage* page shows as **By app**.
## 6. Plans and quotas
| Plan | Requests / month | Overage | Requests / min | Keys · seats · tenants | Features | Support |
|---|---|---|---|---|---|---|
| **free** | 15,000 | refused (`429`) | 30 | 2 · 5 · 0 | route + trace, OpenAI-compatible proxy, feedback, your own model providers (`byok`) | community |
| **payg** - $0 / month, prepaid credits | unlimited | n/a - paid by the token, `402` when the credits run out | 300 | 10 · 10 · 0 | + plans; every model on offer by the token (*Models by the token*) | community |
| **pro** - $49 / month or $490 / year, 90 days free without a card | 300,000 | $10 per 100k | 300 | 10 · 25 · 0 | + plans (`plan`/`execute`) | e-mail, first answer within 48 h |
| **team** - $199 / month or $1,990 / year, 90 days free without a card | 2,000,000 | $8 per 100k | 1,000 | 25 · 100 · 10 | + tenants, `/stats`, hash-chained `/audit` | priority, first answer within 8 h |
| **enterprise** - from $999 / month, through sales, 90 days free without a card | 20,000,000 | $5 per 100k | 3,000 | 100 · 1,000 · 50 | + organization SSO, region pinning, custom quotas and retention | dedicated, first answer within 4 h, 99.9 % monthly availability target with service credits |
Quotas count metered requests (a decision, a chat completion or embedding through the proxy, an estimate, a
feedback event) per **calendar month (UTC)**; the counter resets on the first. `GET /api/v1/info` (`plans`)
returns the values for the deployment you are calling (`OSR_PLATFORM_PLAN_OVERRIDES` patches them per
deployment), `GET /api/v1/me` returns your plan and `GET /api/v1/quota` where you stand:
```json
{"plan": "pro", "period": "month", "used": 241300, "limit": 300000, "remaining": 58700, "percent": 80.4,
"state": "warning", "warn_at": 0.8, "resets_at": 1767225600, "today": 9120, "requests_per_minute": 300,
"overage": {"allowed": true, "usd_per_100k": 10.0, "requests": 0, "estimated_usd": 0.0}}
```
`state` is `ok`, `warning` (80 % and above), `overage` (beyond the quota on a plan with an overage rate and a
live subscription - requests keep flowing and are invoiced) or `exhausted` (beyond the quota on any other plan -
metered calls answer `429`). The dashboard shows the same bar on the overview, the governance page and *Plan
and billing*; the `quota` alert rule fires at 80 % and escalates at 100 % (workspace channels, or the owners
by e-mail when the workspace has no e-mail channel - outbox kind `quota_notice`; the `budget` and `wallet` rules
reach the owners the same way, as `budget_notice` and `credits_notice`). Upgrades are made in the
dashboard (`/platform/dashboard/billing`).
* `429 rate limit exceeded` - per-minute window; wait `Retry-After` seconds.
* `429 monthly quota exhausted` - `X-Quota-Limit`, `X-Quota-Used`, `X-Quota-Reset` and `X-Upgrade: true`.
* `403` with `X-Upgrade: true` - the feature is not in your plan.
* `403 ... requires the enterprise edition` with `X-Edition: enterprise` - the deployment runs the community
image, which does not expose tenants, statistics or audit regardless of plan.
* `403 this API key lacks the '' scope` with `X-Key-Scope` - the key was created with scopes.
Every feature a plan sells is live on the workspace the moment the plan applies - there is no separate
activation. *Plan and billing* (`/platform/dashboard/billing`) lists them under **What the plan includes**,
each one a link to the page that uses it (providers, activity, integrations, tenants, health, audit trail,
workspace SSO); a feature the plan lacks names the cheapest plan that has it with an *Upgrade* button. The
sidebar marks pages the plan does not include with a lock, and the page itself explains the gap.
**Support tiers.** `Plan.support` is `community` (free: documentation, the assistant, the issue tracker),
`email` (pro), `priority` (team) or `dedicated` (enterprise, with an SLA). Tickets opened from a workspace on
priority or dedicated support start at `high` priority at least - a `normal` or `low` request is raised, a
higher one is kept - and the operator notice and inbox line name the plan and the tier
(`Acme (team plan, priority support)`).
**Service levels.** Each tier states what it promises and the platform measures against it, so the pricing
page, the portal and the API read one source (`plans.py`): `support_response_hours` is the first-response
target of the support tier (community none, e-mail 48 h, priority 8 h, dedicated 4 h) and `sla_uptime_pct`
the monthly availability target the plan carries (99.9 % on enterprise; 0 elsewhere). The target is a
contractual figure: the hosted platform is one stack per region, the service credits that apply when a
month misses it are defined in the enterprise agreement, and the number the platform reports is its own
measurement - not an external monitor. `GET /api/v1/sla` reports
the calendar month so far from the workspace's own records: `availability` - the share of its metered
requests the platform answered without a 5xx (a refused request - quota, policy, bad input - or a provider
error is not a platform failure) against `target_pct`; `support` - the tickets opened this month, how many
were answered, the median and slowest first response in hours, how many were answered within `target_hours`
and how many are past it (`breached`, answered late or still waiting). `met` is `true`, `false` or `null`
(nothing to measure yet); `committed` says whether the plan promises anything. *Plan and billing* shows the
same figures under **Service level** on every plan, labelled *committed* or *no commitment on this plan*.
**Talk to sales.** Plans sold through sales (`contact_sales`, the enterprise plan) show *Talk to sales* on the
pricing page and in the portal instead of a checkout; the button opens the form on `/contact`
(`?plan=` preselects the plan). `POST /api/v1/sales/inquiries` (`email`, `name`, `company`, `plan`,
`interest` hosted / self-hosted / both / other, `volume`, `message`; no key needed, five per address and day,
a filled `website` field marks a bot) records the inquiry with a reference (`OSR-S-00007`), e-mails the
requester a confirmation (outbox kind `sales_received`), the support recipients a notice (`sales_operator`)
and the console inbox a line; a signed-in caller's workspace is attached. A second request from the same
address within a day returns the open inquiry. Operators work the queue under `/platform/admin/sales`
(`GET /api/v1/admin/sales/inquiries?status=&q=`, `GET|PATCH /api/v1/admin/sales/inquiries/{key}` - status
`new` / `contacted` / `qualified` / `won` / `lost`, `assigned_to`, internal `note`; audit events
`sales.inquiry_opened`, `sales.inquiry_updated`) and close the loop on the workspace page: plan, custom
quotas, retention and region.
**Custom quotas and retention.** Enterprise agreements override a workspace's numbers without a new plan: an
operator sends `limits` with `PATCH /api/v1/admin/accounts/{id}` - any of `requests_per_month`,
`requests_per_minute`, `max_keys`, `max_tenants`, `max_members` and `retention_days` (whole numbers; the object
replaces the previous overrides, `{}` returns the workspace to the plan as sold; audit event
`account.limits_changed`). `GET /api/v1/me` then reports the effective plan with `"custom": true` and the
overridden values, `GET /api/v1/quota` enforces the custom quota, the key, seat and tenant limits apply as the
plan's own would, and *Plan and billing* shows a **custom quotas** badge. `retention_days` gives the workspace
its own request-row retention: the retention sweep purges its rows at that age and leaves them out of the
deployment-wide purge (`OSR_PLATFORM_RETENTION_DAYS`, which must be on for the sweep to run);
`GET /api/v1/governance` reports the effective `retention_days`. The operator console edits the same fields
under **Custom quotas and retention** on the workspace page (`/platform/admin/workspaces/`);
`GET /api/v1/admin/retention` lists the workspaces with their own retention under `usage.overrides`.
### Billing: subscriptions, trials, overage, invoices and cancellation
Paid plans are flat tiers sold as one Stripe subscription per workspace, monthly or yearly. Self-serve billing
is on when the deployment sets all three of `OSR_PLATFORM_STRIPE_SECRET` (the `sk_live_...` key),
`OSR_PLATFORM_STRIPE_WEBHOOK_SECRET` (the `whsec_...` of the endpoint below) and `OSR_PLATFORM_STRIPE_PRICES`
(JSON plan -> price id); `billing: true` in `GET /api/v1/info` tells the web app. Without them every billing
endpoint answers `501` and operators set plans by hand (`PATCH /api/v1/admin/accounts/{account_id}`). The
Stripe calls are plain HTTPS from the API, no SDK.
`OSR_PLATFORM_STRIPE_PRICES` keys: `"pro"` is the monthly recurring price, `"pro:year"` the annual one (sold
when present - the pricing page shows the switch), `"pro:overage"` a **metered** price on a Stripe Billing
Meter whose event name is `OSR_PLATFORM_STRIPE_METER_EVENT` (default `osr_overage_requests`, unit = requests),
and `"pro:overage:year"` the same metered price with a yearly interval - Stripe only accepts items of one interval
per subscription, so a plan with overage is sold per year only when both `"pro:year"` and `"pro:overage:year"` exist
(a yearly subscriber's overage is then invoiced with the annual renewal). Each metered line lives on its own
Stripe product - *OpenSmartRoute Pro - extra requests* with the unit label `request`, *... - metered usage* -
because Checkout and the invoice list every line by product name: a buyer must read *extra requests, $0.0001 per
request*, not *OpenSmartRoute Pro* twice. A setup run on a deployment created before this moves an overage price
left on the plan product to its own (lookup key transferred; set the printed price ids again).
`osr-platform-stripe-setup --webhook ` (ships with the platform API; run it inside
the API container) creates all of them from the plan
catalogue (products, monthly / yearly / metered prices, the meter, the default Billing Portal configuration the
*Manage subscription* button opens - payment method, billing details, invoices, cancel at period end, no plan changes
- and the webhook endpoint with the seven events below),
idempotently - run it again after changing a price and the lookup keys move to the new prices - and prints the
JSON, its base64 form for the deployment configuration and the webhook signing secret; the key comes from
`STRIPE_SECRET_KEY` in the environment (start with a test-mode key and `--dry-run`):
```json
{"pro": "price_...", "pro:year": "price_...", "pro:overage": "price_...", "pro:overage:year": "price_...",
"team": "price_...", "team:year": "price_...", "team:overage": "price_...", "team:overage:year": "price_...",
"enterprise": "price_...", "enterprise:overage": "price_..."}
```
To also invoice what a workspace spent by the token on the same invoice as its plan, run the setup with
`--usage-meter-event` (bare, or with your own event name): it creates a second Billing Meter whose unit is one
cent and a metered price per plan (the `"pro:usage"` / `"pro:usage:year"` keys), and prints
`OSR_PLATFORM_STRIPE_USAGE_METER_EVENT`. With that variable set, a checkout and a plan change carry the usage
item next to the overage one, and the same sync loop reports the month's `charge_usd` in cents
(`usage_charge_reports`, one `billing.usage_reported` audit entry per increment). Without it the charges are
still priced, stored and shown on *Usage and charges* - they simply never reach a Stripe invoice.
**Tax.** Checkouts ask Stripe Tax to compute VAT / GST / sales tax from the buyer's billing address and to collect a
tax id, which is what the pricing page and the terms promise. Stripe only does that once the account's **head
office address and tax registrations** are set (Stripe -> Settings -> Tax); until then Stripe refuses a session that
asks for tax. The platform does not let that stop a purchase: it creates the session again without the tax fields,
remembers the refusal (`Billing.tax_problem`) and raises the deployment alert `stripe_tax_unconfigured` (console
*Health* and *Finance*, `GET /api/v1/admin/health` -> `alerts`) reading Stripe's tax settings
(`Billing.tax_status`, cached an hour) - critical once a checkout had to drop tax, a warning before. Buyers then pay
the list price with no tax line; fix the Stripe account, and the next checkout computes tax again.
**Activation.** A Stripe account that has not finished activation (business details, bank account, card payments
turned on) refuses every paid Checkout Session - yet it *accepts* a free trial ($0 today), so a buyer could type a
card into a trial that would be cancelled at its end without a word. The API reads the
account from Stripe (`Billing.account_status`, Stripe's account object, cached an hour) and raises the critical deployment alert
`stripe_account_inactive` while charges are disabled or card payments are not active, naming what is missing; while
it stands, no Checkout Session is created at all - a plan with or without a trial, credits or a paid listing - and the
buyer gets `503 Payments are not open on this deployment yet - the operators have been alerted.
Contact sales to buy in the meantime.` with `Retry-After` on both `POST /billing/checkout` and
`POST /billing/credits/checkout`, never Stripe's developer sentence. Finish the activation under Stripe ->
Settings -> Business and Settings -> Payment methods; checkouts go through and the alert clears within the hour.
A checkout adds the plan's overage item next to the recurring one; a plan change swaps both. Every
`OSR_PLATFORM_OVERAGE_SYNC_S` seconds (default hourly, `0` = never) the API reports each billed workspace's
requests beyond its quota this month as meter events (one increment per sync, de-duplicated by identifier and a
compare-and-set row in `overage_reports`, so replicas never double-report; each report is a
`billing.overage_reported` audit entry). The first checkout of a workspace starts the plan's `trial_days` (90 on
every subscription plan) **without a payment method**: the plan applies at once and the workspace returns to free
when the trial ends unless it subscribes (see *Free trials, coupons and the launch offer*); `"coupon": ""` pays
now instead. A plan whose `trial_needs_card` is true carries its trial into Stripe Checkout instead (card collected,
the subscription cancels if none is on file when the trial ends). A workspace that had a subscription or a trial
before gets no second trial.
| Endpoint | Who | What |
|---|---|---|
| `POST /api/v1/billing/checkout` `{plan, interval?}` | owner, admin | `interval` is `month` (default) or `year`. A workspace without a live subscription gets a Stripe Checkout URL (`?upgraded=` on return). A workspace that already has one has the subscription's prices swapped in place with prorations - never a second subscription - the plan applies at once and the answer's `url` points back at `/platform/dashboard/billing?changed=`. `400` when the plan is not sold for that interval. |
| `GET /api/v1/billing/subscription` | any member | `plan`, `billing_state`, `period_end`, `subscription`, `has_portal`, `invoices_available`, live `cancel_at_period_end` / `stripe_status` / `trial_end` / `interval` (null when Stripe is unreachable), `can_manage`, `purchasable` (per paid plan: intervals sold here and the trial a first checkout starts) and `overage` (`billed`, `usd_per_100k`, `reported_requests` this month). |
| `POST /api/v1/billing/portal` | owner, admin | A Stripe Billing Portal session (`{url}`): payment method, billing address, receipts, resume a scheduled cancellation. `404` before the first checkout. |
| `GET /api/v1/billing/invoices` | any member | The last 24 invoices: `number`, `status`, `amount_due` (minor units), `currency`, `created`, `hosted_invoice_url`, `invoice_pdf`. `404` before the first checkout. |
| `POST /api/v1/billing/cancel` | owner, admin | Cancel at the end of the paid period; the plan stays until Stripe sends `customer.subscription.deleted`. `404` without an active subscription. |
| `POST /api/v1/billing/webhook` | Stripe | Signed events (`Stripe-Signature`, HMAC-SHA256 over `t.payload`, 5-minute tolerance); every event id is processed once. |
Point the Stripe webhook endpoint at `POST /api/v1/billing/webhook` with these events enabled:
`checkout.session.completed`, `customer.subscription.created`, `customer.subscription.updated`,
`customer.subscription.deleted`, `invoice.paid`, `invoice.payment_failed`, `charge.refunded` (the last one
credits a refund back to the wallet; `osr-platform-stripe-setup` registers all seven). The workspace's
`billing_state`
(`GET /api/v1/me` -> `account.billing_state`, also `stripe_subscription` and `billing_period_end`) follows
the Stripe subscription status:
| Stripe status | `billing_state` | Plan |
|---|---|---|
| `active`, `trialing` | `active` | the paid plan (`metadata.plan`, or the plan whose price the subscription carries) |
| `past_due` | `past_due` | kept - Stripe is retrying the charge |
| `incomplete` | `incomplete` | unchanged - the first payment has not completed |
| `unpaid`, `paused` | `unpaid` | `free` |
| `canceled`, `incomplete_expired`, subscription deleted | `canceled` | `free` |
`invoice.payment_failed` moves the workspace to `past_due`, writes a `billing.payment_failed` entry to the
workspace's audit trail (`GET /api/v1/workspace/audit`) and sends the owners a dunning message (outbox kind
`billing_payment_failed`, "Update payment method" linking to the hosted invoice); `invoice.paid` moves it
back to `active` and, when the workspace was past due, tells the same owners the invoice is settled (outbox kind
`payment_received`; inbox notice `workspace.payment_received`). The dashboard (`/platform/dashboard/billing`) shows the state badge, a warning banner while
past due, the renewal or end date, *Manage subscription* (portal), *Cancel at period end* and the invoice
table; members see the state and invoices but only owners and admins get the buttons.
### Usage pricing, credits and provider finance
Every metered request carries **two prices**: the provider cost the router optimised (`cost_usd`, what the
executed model cost upstream) and the customer charge (`charge_usd`, what the workspace pays), plus the token
split (`prompt_tokens`, `completion_tokens`, `cached_tokens`), the `provider_id` and `upstream_model` that
executed and the `price_version` the charge was computed under - so a margin report months later reproduces
it. Successful `/api/v1/route` executions and `/v1/chat/completions` answers return the charge in the body
(`result.charge_usd`, and `charge_usd` inside the `opensmartroute` object of a chat answer) and in the
`X-OSR-Charge-USD` header.
The charge comes from a **pricing engine** with two books per model. The *provider price book* holds rates
per unit (`token_in`, `token_out`, `token_cached` in USD per million; `call`, `image`, `second` per unit):
`catalogue` rows are the list prices of the model catalogue (*Seed from catalogue* in the console, and rewritten
by every catalogue refresh of the rankings loop so a vendor's price change reaches the books on its own - never
overwriting the others), `manual` and `contract` rows of a provider beat the list price. *Pricing rules* turn
cost into charge: `markup_pct` (`{"pct": 20}`), `markup_fixed` (`{"usd_per_million": 0.5}`), `per_million`
(`{"usd_per_million": 3.0, "unit": "token_out"}` sets the customer rate outright), `tier` (`{"tiers":
[{"upto_tokens": n, "markup_pct": p}, ...]}` against the month's token volume), `promo` (`{"pct_off": 10}`),
`min_margin` (`{"pct": 5}` - the reseller's floor: after every markup and promotion, no unit is charged under the
provider rate plus that percent, so a tariff typed below a contract rate or a 100 % giveaway can never sell below
cost), `fx` (`{"rate": 0.92}`) and `min_charge` (`{"cents": 1}`). Rules have a scope - `account`
(one workspace), `plan` or `deployment` - and the most specific wins, then `priority`; without any markup rule
`OSR_PLATFORM_DEFAULT_MARKUP_PCT` applies (default `0`: customers pay provider cost). Every change to a book or
rule freezes a new price version (`GET /api/v1/admin/finance/versions`) and writes an audit entry with
`before`, `after` and the `reason` the console requires.
Three semantics the engine guarantees on the metered row. A charge that prices to **zero on purpose** (a 100 %
`promo`, a zero `per_million` tariff) stays zero - only a request that neither the books nor the executing
handler could price falls back to the reported provider cost. **Tiers** read the workspace's token volume of the
month *before* the request being priced (`tier` boundaries are crossed by the first request past them). The
pipeline is **USD-only**: an `fx` rule may scale the charge but `POST /api/v1/admin/finance/rules` refuses one
that names another currency (`400`), and a foreign-currency quote that reaches the engine anyway leaves the row
*unpriced* (`charge_usd` null) rather than mislabelled. Unpriced rows - also the ones a pricing failure left
behind - are retried by the **finance pass** (every `OSR_PLATFORM_FINANCE_SYNC_S` seconds, default 5 minutes, and
on `POST /api/v1/admin/finance/wallets/sync`), each written exactly once; the `finance` block of the analytics
reports how many remain (`unpriced`), the console shows them as a warning, and revenue and margin exclude them until
they are priced.
A workspace pays for usage in one of three **billing modes** (`billing_mode` on the account, set by an operator
with `PUT /api/v1/admin/finance/accounts/{account_id}/billing-mode`): `subscription` (default - the plan's flat
fee plus overage; usage charges reach the same Stripe invoice through a dollar Billing Meter when
`OSR_PLATFORM_STRIPE_USAGE_METER_EVENT` names it, one increment per sync with compare-and-set bookkeeping in
`usage_charge_reports`), `wallet` (prepaid credits, below) or `invoice` (net terms, recorded for the statement).
Credits are bought through a Stripe Checkout in payment mode (`POST /api/v1/billing/credits/checkout`
`{amount_cents}`, whole dollars between `OSR_PLATFORM_CREDITS_MIN_CENTS` and `OSR_PLATFORM_CREDITS_MAX_CENTS`, tax
collected by Stripe Tax); the webhook's `checkout.session.completed` with `metadata[credits_cents]` writes a
`purchase` entry keyed by the event id, `charge.refunded` a `refund` entry keyed by the refund id. The ledger is
append-only - corrections are `adjustment` / `promo` entries with a reason (`POST
/api/v1/admin/finance/accounts/{account_id}/ledger`) - and the balance is its sum.
**Admission and settlement.** Before a call that can incur provider cost runs, an **upper bound** of what it can
cost is held on both sides of the ledger: every target the router may pick is priced by the finance engine at
the prompt's tokens plus the output allowance (`max_tokens` when the caller sets one, else
`OSR_PLATFORM_SPEND_BOUND_OUTPUT_TOKENS`, 4096) under the workspace's tariff, and the dearest quote times
`OSR_PLATFORM_SPEND_BOUND_ATTEMPTS` (fallback attempts that may each bill, default 1) is the bound - provider
cost for the budgets, customer charge for the wallet. A surface that cannot describe its request falls back to the
workspace's mean charge over its last hundred spending requests (at least `OSR_PLATFORM_SPEND_RESERVE_MIN_USD`),
which is an estimate, not a cap, and is labelled as such. Budgets (`daily_budget_usd` / `monthly_budget_usd` of
the workspace, a tenant or a key) are a **strict cap on provider cost**: a request is refused with `429` when
recorded cost + the provider-cost holds of every request still in flight + its own bound reach the limit,
`X-Budget-Used` includes the holds and `GET /api/v1/policy` shows them as `held_usd`. A wallet workspace is
admitted only while balance - charges not yet settled - customer-charge holds in flight - bound does not go
below zero (a bound that exactly meets the remaining credits is admitted; `402` otherwise, `X-Wallet-Balance`
and `X-Wallet-Available` in cents) - so a one-cent wallet never admits a two-dollar first call. The hold is
written, the checks are repeated against every hold now committed (two replicas admitting at the same instant
both see each other and the later one steps back), it is renewed every half TTL while the request is still
running, and it is released when the row is recorded; a hold left by a dead process stops being renewed and
expires after `OSR_PLATFORM_SPEND_RESERVE_TTL_S` seconds. Every metered row also persists the plan it was sold
under and the month's token volume before it, and a row the database refused when the request finished (an
outage after the provider was paid) is written to `usage-journal.jsonl` on the data volume and replayed by the
next finance pass with its original timestamp - a paid-for request is never invisible for longer than the
outage. Repricing (`reprice`) prices such rows under the tariff in force at their own timestamp with those
persisted inputs; only a row that tariff still cannot price (a broken rule closed after the fact) is priced under
the current tariff, and the pass reports it as `approximate` rather than changing a charge silently. Settlement is
cumulative and idempotent: each finance pass debits every wallet workspace *everything charged through today
minus everything ever debited* as one `usage` entry whose idempotency key carries the running total - a
sub-cent day accumulates, a month-end remainder is taken on the first pass of the next month, a row priced late
is owed on the next pass, and two replicas settling the same state write one entry. The `wallet` alert warns at
`OSR_PLATFORM_WALLET_LOW_PCT` of the last top-up.
| Endpoint | Who | What |
|---|---|---|
| `GET /api/v1/billing/usage` | any member | Usage with both prices, `group` = `day` (default) / `model` / `provider` / `key` / `tenant`, filters `key_id`, `tenant`, `model`, `provider_id`, `days`; `format=csv` downloads it. |
| `GET /api/v1/billing/wallet` | any member | `billing_mode`, `balance_cents`, `last_topup_cents`, `pending_cents` (charged, not yet settled), `reserved_cents` (held for requests in flight), `available_cents` (what one more request is admitted against), purchase bounds and the active wallet alerts. |
| `GET /api/v1/billing/wallet/ledger` | any member | The append-only ledger behind the balance, newest first. |
| `POST /api/v1/billing/credits/checkout` `{amount_cents, save_payment_method?, auto_topup?}` | owner, admin | A Stripe Checkout URL (payment mode); returns to `/platform/dashboard/billing?credited=`. `501` without Stripe. |
| `PUT` / `DELETE /api/v1/billing/wallet/auto-topup` | owner, admin | Auto top-up rule (*Models by the token*, below). |
| `GET /api/v1/billing/statement?month=YYYY-MM&format=json\|markdown` | any member | The monthly credits statement: opening and closing balance, every entry, usage by model. |
The operator side lives under `/api/v1/admin/finance`: `prices` (list, add, end, `prices/seed`), `rules` (list,
add, end), `versions`, `quote` (price a hypothetical request), `alerts` (the money alerts below, as the console's
*Margin watch* shows them), `wallets` and `wallets/sync` (run the finance pass
now: expire dead holds, reprice, settle), per-workspace
`ledger` and `billing-mode`, `providers` (prepaid balance, credit limit, low-balance line, contract terms),
`providers/{id}/reconcile?period_start&period_end` (expected spend from the metered rows against the invoices of
the period, with the models behind it) and `provider-invoices` (record, then move to `disputed` / `paid` -
settlement is recording `paid`). A prepaid provider whose projected balance crosses its low-balance line raises a
deployment alert (`provider_balance`). Two more alerts guard the reseller's margin on the console health page and
in the alert stream: `margin` - a model whose priced requests of the last 24 hours were charged below provider
cost (`critical`) or under `OSR_PLATFORM_MARGIN_ALERT_PCT` gross margin (`warning`, default 5 %) - and
`above_list_price` - one per provider, counting the enabled offers whose storefront rate per million tokens exceeds
the vendor's catalogue list price (the worst one named), so the platform no longer beats buying direct there. Reads are open to every operator; writes need the `finance` role (price
books, rules, ledger, billing mode, invoices, settlement) or `provider_manager` (provider accounts and pricing,
the provider endpoints themselves - create, edit, delete, check, target mappings - and the offers listed on them) -
`superadmin` passes everything. `GET /api/v1/admin/analytics` carries a `finance` block: usage revenue, provider
cost, gross margin, the wallets and the breakdowns by day, provider, model, workspace and plan.
### Models by the token: offers, prices, credits and provider preferences
Besides routing with `auto`, the platform sells named models by the token, the way OpenRouter does. You pick
a model from the storefront, call it under its catalogue id, and pay the price shown per million tokens; the
platform chooses which provider endpoint serves the call unless you pin one.
Three ids appear throughout this section:
| Id | Example | Meaning |
|---|---|---|
| Catalogue id | `openai/gpt-4.1` | A model card in the reference catalogue (`GET /api/v1/llms`): capabilities and the vendor's list price. |
| Provider | `azure` | An endpoint the deployment can call, configured by the operator under *Providers*. |
| Offer | `openai/gpt-4.1@azure` | That model, served by that provider. The `@provider` suffix pins the endpoint. |
An offer is an ordinary routing target: `auto` competes it with the rest of the catalogue, plans may use it,
the learners grade it. Nothing is added to `targets.yaml` to sell a model.
#### Find a model and its price
The storefront at [`/models?view=offers`](/models?view=offers) lists every model on offer with two prices: the
vendor's list price and the price your workspace pays, per million input and output tokens (cached input when
the model supports it). Signed in, you see your own plan's prices; signed out, the deployment's public tariff.
Each row opens the endpoints behind it with their health and last week's traffic.
The same data over the API:
| Endpoint | Returns |
|---|---|
| `GET /api/v1/offers` | One row per model: `prices.input_usd_per_1m` / `output_usd_per_1m` / `cached_usd_per_1m`, `priced_by` (`rules` when a pricing rule shaped the price, `list` when the vendor's price stands), the endpoints. Filters `q`, `vendor`, `provider`. |
| `GET /api/v1/offers/{vendor}/{model}` | One model with its endpoints and its `GET /v1/models` entry. |
| `GET /api/v1/llms/{vendor}/{model}` | The catalogue card, with the same `offers` block. |
| `GET /v1/models` | The OpenAI-compatible list, in OpenRouter's shape (below). |
`GET /v1/models` returns each model on offer with `pricing.prompt` / `completion` / `input_cache_read` in USD
**per token** as strings, `context_length`, `architecture`, `top_provider` and `supported_parameters` - the
fields OpenRouter clients read - plus an `osr` block with the `endpoints` (provider, upstream id, `executable`,
`healthy`, region, data policy) and the vendor `list_price`. Tools written against OpenRouter's listing work
unchanged.
The price you see before the call is the price recorded after it: usage is metered under the catalogue id even
when the provider reports its own name (`gpt-4.1` on Azure is `openai/gpt-4.1`), and the customer price comes
from the pricing engine in the section above - a contract row of the provider in the price book beats the list
price, then rules per deployment, plan or account apply.
#### Call a model by name
`POST /v1/chat/completions` (and the Responses and Messages dialects over it) accepts a catalogue id or an offer
id in `model`, alongside `auto` and target ids. A `models` list may mix all three.
```bash
curl -s https://api.opensmartroute.ai/v1/chat/completions \
-H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"model": "openai/gpt-4.1", "messages": [{"role": "user", "content": "Summarise this thread."}]}'
```
| `model` | What happens |
|---|---|
| `openai/gpt-4.1` | The router picks among the enabled endpoints serving that model - by price, health and learned quality - and falls back down them when one fails. |
| `openai/gpt-4.1@azure` | Only that endpoint. `503` with `Retry-After` when it cannot be used right now. |
| `auto` | The whole catalogue, offers included. |
The response says which endpoint answered: `model` and the `X-OSR-Target` header carry the offer id
(`openai/gpt-4.1@azure`), `X-OSR-Charge-USD` and `charge_usd` in the response's `opensmartroute` object carry what
the call cost you, and the usage row records `provider_id` and `upstream_model`.
Key, workspace and tenant `allow_targets` / `deny_targets` may name a catalogue id; it stands for every endpoint
of that model.
#### Steer the choice: provider preferences
Preferences travel in the request body under `osr.provider`. OpenRouter's top-level `provider` object is
accepted as an alias and is never forwarded upstream.
```json
{
"model": "openai/gpt-4.1",
"messages": [{"role": "user", "content": "..."}],
"provider": {"order": ["azure", "openai"], "data_collection": "deny", "sort": "price"}
}
```
| Key | Effect |
|---|---|
| `order: ["azure", "openai"]` | Try the first named provider's endpoint first; the rest are the fallback pool in that order. |
| `only: [...]` | Use only these providers' endpoints. |
| `ignore: [...]` | Never use these providers' endpoints. |
| `data_collection: "deny"` | Skip every endpoint whose provider states that it trains on, logs or retains prompts. |
| `zdr: true` | Zero data retention: use only endpoints whose provider has *declared* that it keeps nothing (`zero_retention: true`, or `retains_days: 0` with neither training nor prompt logging). An endpoint with no data policy at all is skipped - the provider has to say so. |
| `sort: "price"` \| `"latency"` \| `"throughput"` \| `"quality"` | Re-weight the objective: the cheapest, the fastest or the best endpoint that can answer. |
| `max_price: {"prompt": 2, "completion": 8}` | USD per million tokens; dearer endpoints are excluded. |
| `allow_fallbacks: false` | Do not fall back to another endpoint (same as `osr.fallbacks: false`). |
Preferences that exclude every endpoint of the named model are refused with `400` rather than silently routed
elsewhere. An unknown key is a `400` too.
A workspace sets defaults for all of its requests with the policy (`PUT /api/v1/policy`, key
`provider_preferences`, the same object, or the *Providers* block of the policy dialog); a request's own preferences are layered on top.
#### Pay as you go
The `payg` plan has no monthly fee and no request quota. Usage is paid from prepaid credits
(`billing_mode: wallet`); when they run out, requests stop with `402`. The abuse controls are the plan's
`requests_per_minute` and the per-key spend limits below.
- **Switch to it:** `POST /api/v1/billing/checkout` `{"plan": "payg"}` applies the plan at once - no Stripe
Checkout, no payment method - and puts the workspace on the wallet. It answers `409` while a subscription is
live; cancel that first. Moving to a subscription plan later takes the workspace off the wallet again.
- **Buy credits:** the billing page offers fixed packs and a custom amount; `POST /api/v1/billing/credits/checkout`
is the API. `GET /api/v1/billing/wallet` reports the packs, the purchase fee and the balance.
- **Watch the balance from a client:** every priced chat answer of a wallet workspace carries `X-Wallet-Balance`,
the credits (in cents) one more request would be admitted against, so an integration can top up before the
`402` arrives.
| Setting | Default | Effect |
|---|---|---|
| `OSR_PLATFORM_DEFAULT_PLAN` | `free` | The plan a self-serve signup starts on; `payg` makes new workspaces pay by the token from day one. |
| `OSR_PLATFORM_CREDITS_PACKS_CENTS` | `1000,5000,20000` | The credit packs offered on the billing page (within the purchase bounds). |
| `OSR_PLATFORM_CREDITS_FEE_PCT` | `0` | Purchase fee, percent of the credits bought. |
| `OSR_PLATFORM_CREDITS_FEE_CENTS` | `0` | Purchase fee, fixed part in cents. |
| `OSR_PLATFORM_AUTO_TOPUP_COOLDOWN_S` | `3600` | After one auto top-up attempt, charged or declined, the wallet is left alone for this long. |
The fee is a second line item in Stripe Checkout and is booked on the ledger as a zero-sum pair of `fee`
entries - collected, not spendable - so the balance carries only the credits and revenue analytics separate
the fee from the spread.
**Auto top-up.** A wallet workspace can have credits bought for it before the balance reaches zero:
1. Buy credits once with `save_payment_method: true` (the billing page: *Keep this card for auto top-up*). The
Checkout keeps the card on the workspace's Stripe customer for off-session charges; the webhook stores its
label (`visa 4242`), never the number. Passing `auto_topup: {amount_cents, threshold_cents}` on the same
purchase switches auto top-up on as soon as it completes.
2. Set or change the rule with `PUT /api/v1/billing/wallet/auto-topup` `{amount_cents, threshold_cents}` - buy
`amount_cents` whenever the available balance falls to `threshold_cents` or below (whole dollars within the
purchase bounds, the threshold below the amount; `409` without a saved card). `DELETE` turns it off;
`?forget_card=true` also detaches the card at Stripe. Owners and admins only.
3. The finance pass (the same one that settles usage, several times a day; `POST /api/v1/admin/finance/wallets/sync`
runs it now) charges the saved card off-session and credits the wallet under the PaymentIntent id, so the
`payment_intent.succeeded` webhook for the same intent is a no-op. One attempt per cooldown, whatever the
outcome: a declined card is recorded as `auto_topup.error`, raises the `wallet:auto_topup` alert on the
billing page and in the workspace alerts, and is not retried until the cooldown passes or the rule is saved
again.
`GET /api/v1/billing/wallet` reports the rule under `auto_topup` (`enabled`, `amount_cents`, `threshold_cents`,
`payment_method`, `last_attempt_at`, `error`, `available` - the deployment has Stripe). Every step is in the audit
log: `billing.payment_method_saved`, `billing.auto_topup_changed`, `ledger.auto_topup`, `billing.auto_topup_failed`.
**Referrals: give credits, get credits.** Every workspace has a referral link, `/r/`, on *Refer a
team* in the dashboard (`GET /api/v1/referrals`: the code, the link, the terms, the referrals so far and what
they earned; `POST /api/v1/referrals/invite` e-mails the link to one address with a line from the sender,
twenty a day). The site remembers the code the way it remembers the first touch and hands it to the signup
(`attribution.ref` on `POST /api/v1/signup` and on a sign-in with a provider that creates the account); the
latest referral link a browser opened is the one that counts. A referral is *signed up* when the new workspace
exists, *qualified* when its owner has confirmed the address and the workspace has made its first successful
request, and *rewarded* when the finance pass has credited both wallets - `OSR_PLATFORM_REFERRAL_REWARD_CENTS`
to the referrer and `OSR_PLATFORM_REFERRAL_WELCOME_CENTS` to the new workspace (both `500` by default), as
`promo` ledger entries that name the referral. A referrer earns at most `OSR_PLATFORM_REFERRAL_MAX_REWARDS`
rewards (`20`; `0` = no cap) - a referral past the cap is *capped* and the invited side still gets its welcome
credit. Self-referrals (the same person or address, a member of the inviting workspace) are not recorded and a
workspace is referred once. `GET /api/v1/referrals/lookup?code=` tells a visitor on a link whether it is live
and what they are offered, naming the inviter by initials only; the inviter sees the invited workspaces by
initials. `OSR_PLATFORM_REFERRAL_ENABLED=false` switches the programme off (links keep working as plain sign-up
links); all four are run-time settings. Operators read the programme on `GET /api/v1/admin/referrals` and run
the qualification pass with `POST /api/v1/admin/referrals/sweep`. Events: `referral.signed_up`,
`referral.rewarded`, `referral.capped`, `referral.invited`.
**Money that moves is announced.** Whenever credits land on the wallet - a purchase at checkout or an automatic
top-up - the owners receive *Credits added* (outbox kind `credits_added`: the amount, the balance now, the source
and any purchase fee) and the workspace inbox gets `workspace.credits_added`; a Stripe refund sends *Payment
refunded* (`credits_refunded`, the balance after it) and files `workspace.credits_refunded`. Each goes out once
per ledger entry, whichever of the finance pass or the webhook wrote it. Both are `account` mail and always go out.
#### Per-key spend limits
`limits.spend_limit_cents` (per calendar month) and `limits.spend_limit_daily_cents` (per UTC day) on
`POST /api/v1/keys` cap what one key may **charge** the workspace: charge recorded so far, plus the charge side
of its requests in flight, plus the bound of the request being admitted. A request that would pass the cap is
refused with `429`, `X-Key-Limit: spend_limit_cents` and the `X-Budget-*` headers in dollars. Budgets
(`daily_budget_usd`) remain the cap on provider cost.
#### Operators: list, price and pause offers
The console page **Offers** (`/platform/admin/offers`) does everything in this section without curl.
1. **Connect a provider** under *Providers* and press **Check**. The check calls the provider's `/models` and
reports, under `offers`, which upstream ids the reference catalogue recognises.
2. **List models.** One click creates an offer for every recognised id. Upstream ids that resolve to the same
card are one offer (`gpt-4.1` and `gpt-4.1-2025-04-14` both mean `openai/gpt-4.1`). Azure and Bedrock do not
return deployments from `/models`: name them in the `models` field.
3. **State the data policy** the provider publishes - `trains`, `retains_days`, `logs_prompts`,
`statement_url`, `stated_at`. This is what `data_collection: deny` reads; an empty policy never excludes the
endpoint.
4. **Set the price** on the *Finance* page: a contract row for the provider in the price book and the mark-up
rules. Without either, the vendor's list price stands.
5. **Pause or remove** an offer from the same page when a provider is unreliable or a contract ends. Paused
offers leave the storefront and `GET /v1/models` at once.
The same over the API:
| Endpoint | Purpose |
|---|---|
| `POST /api/v1/admin/providers/{provider_id}/check` | Probe the provider; `offers.recognised` lists the upstream ids the catalogue knows. |
| `POST /api/v1/admin/providers/{provider_id}/offers` `{models?, region?, data_policy?}` | List models: one offer per recognised id (`models` names them explicitly, else the probe's list is used). |
| `DELETE /api/v1/admin/providers/{provider_id}/offers` | Remove every offer of the provider. |
| `GET /api/v1/admin/offers` | Every offer with its provider, prices and state. |
| `GET /api/v1/admin/offers/desk` | The reseller's desk: every offer with what the deployment pays the provider per million tokens and what a workspace on the public tariff is charged (`rates`, with `priced_from` `contract` when a provider price-book row sets the cost, else `list`), the margin per unit, whether the offer declares zero data retention, and the last seven days - requests, tokens, success rate, realised provider cost, customer charge and margin. `totals` sums the week; `losing` lists the offers that lose money on paper or in practice. |
| `POST /api/v1/admin/offers` `{model, provider_id, upstream_model?, region?, data_policy?, enabled?, notes?}` | Add one offer by hand. |
| `PATCH` / `DELETE /api/v1/admin/offers/{vendor}/{model@provider}` | Pause (`enabled: false`), re-point (`upstream_model`), edit the data policy, or remove. |
`OSR_PLATFORM_OFFERS_AUTO_LIST=true` lists the recognised models on every provider check without the button.
**An Ollama server offers only the models it is mapped to run.** Several Ollama servers often share one model
store, so each one's `/api/tags` lists every model on the disk while only the mapped one is resident and
kept warm; an offer for any other model would send a chat into a cold load of tens of gigabytes and time out.
List models therefore creates offers only for the upstream ids the server's target mappings name (the rest are
returned under `unserved`), and every sync pauses an enabled offer whose server is not mapped to run the
model, with the reason in the offer's notes. A provider is recognised as an Ollama server by its kind, by
port `11434` or by a name starting with `ollama-`. Providers without mappings are not restricted.
**A provider you cannot pay is unhealthy, not expensive.** When a prepaid provider's projected balance is
exhausted (see *Usage pricing, credits and provider finance*), every target it executes - offers and
hand-written mappings alike - leaves the pool `auto` routes over, a request pinning one of them answers `503`
with `Retry-After`, the storefront and `GET /v1/models` mark the endpoint as not executable, and the
`provider_balance` alert names the provider on the console health page and in the alert stream. A workspace's
own provider (BYOK) is never affected: it pays for its own calls.
### Free trials, coupons and the launch offer
A **coupon** grants a paid plan for a number of days without payment - no Stripe, no payment method. Redeeming
one puts the workspace on the plan at once and records `account.trial_end` (and `coupon_code`) on
`GET /api/v1/me`; when the trial ends without a subscription having taken over, the workspace returns to `free`
(on its next request and by a sweep every `OSR_PLATFORM_TRIAL_SWEEP_S` seconds, default 15 minutes; audit entry
`billing.trial_ended`). The owners hear at each step, whether or not the deployment has Stripe: *Your* (plan) *trial
has started* the moment it begins (outbox kind `trial_started`: until when, that no card was taken, what happens
at the end), *Your* (plan) *trial ends on* (date) three days before (`trial_ending`, the lifecycle campaign) and
*Your* (plan) *trial has ended* when the sweep moves the workspace back (`trial_ended`); the workspace inbox carries
the same (`workspace.trial_started`, `workspace.trial_moved` when the trial moves to another plan,
`workspace.trial_ended`) and each is a `plan.changed` event (`reason` `trial`, `trial_moved`, `trial_expired`)
for the subscribed channels. While a coupon trial runs the quota is the plan's, overage is not billed (there is no
subscription), and the workspace may move to any other plan the coupon covers with the same end date. A Stripe
subscription that completes during the trial clears the deadline. Coupons work on deployments without Stripe.
The **plans' own free trials** ride the same machinery. Every subscription plan whose `trial_days` is set and
whose `trial_needs_card` is false (the shipped catalogue: Pro, Team and Enterprise, 90 days each) is carried by
the built-in promotion `FREE-TRIAL`, which the API keeps in step with the catalogue at start and whenever an
operator changes a plan (`PATCH /api/v1/admin/plans/{name}`): it covers exactly those plans, advertises the
longest of their trials and grants each plan its own number of days. The first `POST /api/v1/billing/checkout`
of a workspace therefore starts the trial at once, without Stripe and without a card - on a deployment that has
no Stripe at all, and on one whose Stripe account cannot charge yet. One trial per workspace: a workspace that
had any trial or subscription before is refused (`409`) and pays instead. `GET /api/v1/info` shows the promotion
with `builtin: true`; the running trial reports `coupon: null`; the operator console lists the row as created by
`plans` and cannot type over its terms for long. Set every plan's `trial_days` to `0` and the promotion switches
itself off.
The **launch offer** is an operator's coupon on top, created at start-up from the environment and applied
automatically (the newest live promotion wins a checkout):
| Variable | Default | Meaning |
|---|---|---|
| `OSR_PLATFORM_LAUNCH_COUPON` | unset | The code, e.g. `LAUNCH100` (3-32 letters, digits, dashes). Unset = no launch coupon. |
| `OSR_PLATFORM_LAUNCH_COUPON_DAYS` | `100` | Days of the plan the coupon grants. |
| `OSR_PLATFORM_LAUNCH_COUPON_UNTIL` | unset | `YYYY-MM-DD` (UTC) after which it can no longer be redeemed; unset = until an operator disables it. |
| `OSR_PLATFORM_TRIAL_SWEEP_S` | `900` | How often expired coupon trials are moved back to free (`0` = only on the workspace's next request). |
An **auto-apply** coupon is the promotion: `GET /api/v1/info` advertises it as `promotion` (`code`, `trial_days`,
`plans`, `valid_until`, `description`, `builtin`), the pricing page and the plan cards say "*N*-day free trial", and
`POST /api/v1/billing/checkout` applies it to every eligible workspace **instead of** starting a Stripe Checkout -
nobody has to type the code. Eligible = the coupon covers the plan, is enabled, within its deadline and cap,
the workspace has no live subscription and has not redeemed this code before (for the built-in trial: has never
had a trial). The answer then carries `coupon`, `builtin`,
`trial_days`, `trial_end` and a `url` back to `/platform/dashboard/billing?redeemed=`. Send `"coupon": ""`
to skip the trial and the promotion and pay (the *Subscribe now* button on the trial card), or `"coupon": ""`
to use a specific one.
| Endpoint | Who | What |
|---|---|---|
| `GET /api/v1/billing/coupons` | any member | `promotion` on offer, the workspace's running `trial` (`plan`, `coupon`, `trial_end`, `days_left`), `eligible` per paid plan and the workspace's `redemptions`. |
| `POST /api/v1/billing/coupons/validate` `{code, plan?}` | any member | The coupon's public terms plus `eligible` and `reason` for this workspace. `404` unknown code. 20 lookups per workspace per minute. |
| `POST /api/v1/billing/coupons/redeem` `{code, plan}` | owner, admin | Apply the coupon: `400` when it does not cover the plan or ran out, `409` when the workspace redeemed it before or already pays for a subscription. |
| `GET|POST /api/v1/admin/coupons`, `GET|PATCH /api/v1/admin/coupons/{code}`, `POST /api/v1/admin/coupons/sweep` | operator | Create (`code`, `trial_days`, `plans` - empty = every paid plan, `auto_apply`, `max_redemptions`, `valid_until`, `description`), list with counters, see who redeemed a code, change its terms or disable it, run the expiry sweep now. |
The dashboard's *Plan and billing* page shows the running trial (days left, end date, *Subscribe now* when
Stripe is configured), marks the plans the promotion covers, and has a *Have a coupon code?* form; the operator
console manages codes at `/platform/admin/coupons`.
## 7. Organizations
Signup creates a personal workspace. An **organization** is a second kind of workspace with its own
members, roles, API keys, tenants, plan and usage; one user can belong to several and switches between
them in the dashboard (`/platform/dashboard/workspace`, `POST /api/v1/workspaces/{account_id}/switch`).
Creating one (`POST /api/v1/workspaces` with `name` and an optional `slug`) makes you its owner and
issues its first API key. A personal workspace that already carries a paid plan, keys and history can
instead become an organization in place: `POST /api/v1/workspace/convert` with `name` (and an optional
`slug`) changes only its kind - plan, subscription, keys, tenants and usage stay on the same row, the
browser session stays on it, and the user receives a fresh free personal workspace alongside (audit
`workspace.converted`). The *Members* page of a personal workspace offers exactly this, so the seats a
team or enterprise plan sells become usable without moving anything.
Roles are `member` < `admin` < `owner`:
| Action | Role |
|---|---|
| Route, read usage, savings and activity; read the roster and the teams | `member` |
| Manage the roster and the keys of a team they lead (below): mint, rename, rotate, revoke | `member` with the team `lead` role |
| Create and revoke API keys, change tenants, invite and remove members, create teams, rename the workspace, checkout, billing portal and cancellation | `admin` |
| Grant or remove `owner`, transfer ownership, configure organization SSO, delete the connection or the organization | `owner` |
API keys belong to the workspace and act with its full rights; the role model applies to signed-in
browser sessions. Members are invited by email (`POST /api/v1/workspace/invites`): the invitee gets a
message with the link when the deployment has a mail server, and the platform returns the same link once
to the inviter so it can be shared directly. `POST /api/v1/workspace/invites/{id}/resend` sends a pending
invitation again with a fresh link and a full validity period (the previous link stops working; audit
`member.invite_resent`); invitations that expired without being accepted are purged by the retention
sweep. Seats and tenants
are limited per plan (`max_members`, `max_tenants` in `GET /api/v1/info`). An organization always
keeps at least one owner. **Transferring ownership** is one call: `POST /api/v1/workspace/transfer` with the new
owner's `user_id` and the role the current owner keeps (`keep_role`: `admin` - the default -, `member`, `owner` to
share ownership, or `leave` to hand over and go), owner only, audited as `workspace.transferred`, both people
e-mailed; the crown button on **Members** does the same. There is no per-workspace "suspend" for a person:
**remove the member** (their sessions on the workspace end at once, the invitation can be sent again later), or
an operator disables the whole user account.
**Organization SSO** (enterprise) lets people sign in with the company identity provider:
`PUT /api/v1/workspace/sso` with `kind` (`github`, `google`, `microsoft`, `gitlab`, `linkedin` or `oidc` with an
`issuer`), `client_id`, `client_secret`, the allowed email `domains` and the `default_role` given to
anyone who signs in from one of those domains. The workspace needs a slug first; the sign-in page
shows the provider as `org-`. A `role_map` (`{"claim": "groups", "map": {"osr-admins": "admin"},
"require_match": false}`) derives the workspace role from the provider's group claim instead: the mapping is
re-applied at every sign-in (a user added to or removed from a mapped group changes role on the next login,
recorded as `member.role_sso` in the workspace audit; the last owner is never demoted), unmatched users get
`default_role`, and `require_match: true` refuses anyone outside the mapped groups. Entra `groups` / `roles` /
`wids`, Okta `groups`, Keycloak `realm_roles` and Cognito `cognito:groups` are read from the id token and
userinfo; ask the provider for the claim with the connection's `scopes` (`groups`).
**SCIM provisioning** (organizations) is the push side of the same identity provider: single sign-on lets a
person *in* when they sign in, SCIM 2.0 (RFC 7643 / 7644) lets Entra ID, Okta, OneLogin, JumpCloud and the like
create the account before the first sign-in and **remove it the moment HR closes it**. An owner mints the
provisioning token on **Workspace › Single sign-on** (*Provisioning (SCIM)*; `POST /api/v1/workspace/scim/token`,
shown once, valid a year, one live token per organization - minting again replaces it, `DELETE` revokes it) and
pastes it with the base URL `/scim/v2` into the provider's SCIM connector. The connector then calls:
| Call | What happens |
|---|---|
| `GET /scim/v2/ServiceProviderConfig`, `/ResourceTypes`, `/Schemas` | What is supported: users, PATCH, `filter`, no bulk, no passwords (sign-in stays with the provider). |
| `GET /scim/v2/Users` | The members as SCIM users; `filter=userName eq "..."` / `externalId eq "..."`, `startIndex`, `count` (up to 200). |
| `POST /scim/v2/Users` | Provision: the person's account is created if it does not exist (with a personal workspace like any signup; the address counts as confirmed because the provider vouches for it) and joins the organization with the SSO connection's `default_role` (`member` without one) - a `roles` entry of `admin` or `member` sets the role, `owner` is refused; the plan's seat limit applies as for an invitation; `409` when already a member. |
| `PATCH` / `PUT /scim/v2/Users/{id}` | `name`, `displayName`, `roles`, `externalId` and `active`: `active: false` **removes the membership and ends the person's sessions on the workspace** - the user and their other workspaces stay - and `active: true` restores it. Paths such as `name.givenName` and `emails[type eq "work"].value` are understood. |
| `DELETE /scim/v2/Users/{id}` | Deprovision, the same as `active: false` (`204`). The last owner is never removed (`409`). |
| `GET /scim/v2/Groups` | Empty: roles come from the SSO `role_map` at sign-in or from `roles` here; teams are not provisioned. |
Errors come back in SCIM's own shape (`urn:ietf:params:scim:api:messages:2.0:Error` with `status`, `detail`,
`scimType`). Every change is in the workspace audit trail with *identity provider (SCIM)* as the actor -
`member.provisioned`, `member.deprovisioned`, `member.role_scim`, `user.scim_updated` - and a deprovisioning files
the same *member removed* notice in the workspace inbox as a removal by hand. `GET /api/v1/workspace/scim`
(admin role) shows whether provisioning is on, the token's age and use and how many people came through it.
Not offered, by decision rather than omission: **SAML** (every supported identity provider - Entra, Okta,
Google, Keycloak, Cognito - also speaks OIDC, which is what the `oidc` connection kind uses; SAML-only
providers can front the platform through an OIDC bridge) and the OpenAI
**Batch API** (asynchronous batch files - the router prices and routes each request live; send them concurrently).
### Teams
A large customer does not want one admin minting every key. A **team** groups some of an organization's
members - a product squad, an agency's client desk, the data-science group - and gives that group its own
place in the workspace:
* **Tenant binding.** A team may be bound to one of the workspace's [tenants](#8-tenants). Keys minted for
the team are pinned to that tenant, so its constraints, budgets and usage roll-up apply, and the team's
spend shows on the tenant. `GET /api/v1/workspace/teams` reports each bound team's requests, tokens and
cost this month and today. A tenant bound to a team cannot be deleted until the team is unbound (`409`).
* **Leads.** A team member with the team role `lead` manages the roster (adds and removes members, not
other leads) and the team's keys - mints them (`POST /api/v1/keys` with `team: ""`; the key carries
`team` and is pinned to the team's tenant), renames, rotates and revokes them - without holding the
workspace `admin` role. Only admins make leads. This is the delegation that lets a customer run its own
people: the admin creates the team, binds it and names a lead; the lead does the rest. A team with keys
cannot be re-bound to another tenant (`409`; revoke its keys first); deleting a team detaches its active
keys (they keep working as the tenant, managed by admins from then on - `keys_detached` in the answer).
* **Attribution.** `GET /api/v1/workspace/members` lists every person's teams, `GET /api/v1/me` the
caller's own (`teams`), and the audit trail records `team.created`, `team.updated`, `team.deleted`,
`team.member_added` and `team.member_removed` (also notification events).
| Endpoint | Role | Notes |
|---|---|---|
| `GET /api/v1/workspace/teams` | `member` | Every team with `members`, `leads`, `keys` pinned to its tenant and `usage`; `mine` names the caller's own teams. |
| `POST /api/v1/workspace/teams` `{name, slug?, description?, tenant?, members?}` | `admin` | `slug` defaults to the name; `members` seeds the roster (`[{user_id, role}]`). `409` when the slug exists. |
| `GET /api/v1/workspace/teams/{slug}` | `member` | The team with its roster (team role and workspace role per person), its active `keys` (prefixes) and `can_manage`. |
| `PATCH /api/v1/workspace/teams/{slug}` `{name?, description?, tenant?}` | `admin` | `tenant: ""` unbinds. `409` while the team has keys and the tenant would change. |
| `DELETE /api/v1/workspace/teams/{slug}` | `admin` | Members stay in the workspace; the team's active keys are detached and keep working as the tenant. |
| `POST /api/v1/workspace/teams/{slug}/members` `{user_id, role?}` | `admin` or team `lead` | The person must already be a member of the workspace. Only admins set `role: "lead"`. |
| `DELETE /api/v1/workspace/teams/{slug}/members/{user_id}` | `admin` or team `lead` | A lead cannot remove another lead (but may leave). |
Leaving or being removed from the workspace drops every team membership; deleting the organization drops
its teams. Personal workspaces have no teams (`400`). The **Members** page carries the *Teams* card
(create, bind, roster, leads) and shows each member's teams; the **Tenants** page names the teams that run
under each tenant; the key dialog on **API keys** offers the teams the caller may mint for. Operators see
the teams and their rosters on the workspace's console page (`teams` on
`GET /api/v1/admin/accounts/{account_id}`), and the data export (`GET /api/v1/me/export`) carries them.
### Regions
The hosted platform runs as independent stacks per region - an EU stack and a US stack - each with its
own database, so a workspace's keys, usage and traces never leave the region it was created in. A stack
declares its region with `OSR_PLATFORM_REGION` (`eu`, `us`; unset = a single-region deployment that
enforces nothing). Every workspace created on a stack - at signup, `POST /api/v1/workspaces` or by an
operator - is pinned to that region (`region` on the account in `GET /api/v1/me` and `GET /api/v1/info`
reports the stack's `region` and `regions_enforced`), and API keys are minted where the workspace lives.
Using a key on the other stack answers `403` with `code: "region_mismatch"`, `region: ""` and
`detail: "this workspace is pinned to region ''; use the endpoint"`; browser sessions get the
same answer, which is how the dashboard tells you your workspace lives in the other region. Every
authenticated response carries `X-OSR-Region` with the answering stack's region, so a client can check
it is talking to the right one. Operators are not pinned, and public endpoints work on every stack.
Deploying two stacks = two deployments of the same template with
`OSR_PLATFORM_REGION` set to `eu` / `us` and DNS such as `eu.api.` / `us.api.` in front
of each. Moving a workspace is deliberate: an operator changes its `region` (`PATCH
/api/v1/admin/accounts/{account_id}` with `{"region": "us"}`, audited as `account.region_changed`; `""`
unpins it), and the data itself is exported from one stack and imported on the other by hand
(`GET /api/v1/me/export`). This is not the routing constraint of the same name: `constraints.region` on a
request (or a tenant's `region`) chooses *targets* deployed in a region; the workspace region chooses the
*stack* that accepts the key.
## 8. Tenants
Tenants (enterprise) apply a fixed set of hard constraints per customer, team or region without the
caller repeating them. Create a tenant, then send its slug in `X-OSR-Tenant` (or `osr.tenant`).
```bash
curl -s -X PUT "$OSR_URL/api/v1/tenants" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"slug": "eu-health", "config": {"region": "eu", "data_boundary": "private", "max_cost_per_1k": 0.01, "deny_targets": ["llm-frontier"]}}'
curl -s "$OSR_URL/api/v1/route" -H "Authorization: Bearer $OSR_API_KEY" -H "X-OSR-Tenant: eu-health" \
-H "Content-Type: application/json" -d '{"text": "Anonymise this discharge summary ..."}'
```
Tenant config keys: `region`, `data_boundary`, `max_cost_per_1k`, `max_latency_ms`, `preferred_max_latency_ms`,
`deny_targets`, `allow_targets`, `contains_pii`, `daily_budget_usd`, `monthly_budget_usd`, `log_payloads`,
`marketplace_discovery`, `data_protection`, `output_guard`, `allowed_ips`. Explicit
request constraints win over the tenant defaults; the workspace policy (next section) applies on top of both. Executed spend
is attributed to the tenant, so `GET /api/v1/governance` shows spend and budgets per tenant, and
`GET /api/v1/tenants` returns each tenant with `usage` (`month` and `today`: requests, tokens,
`cost_usd`), its `budgets` with what is used and remaining, and `keys` - the number of API keys pinned
to it. `GET /api/v1/usage` adds `by_tenant` (requests, tokens and cost per tenant slug; `null` is the
workspace's own traffic) and `?tenant=` narrows the daily series and the target and endpoint
breakdowns to one tenant; `GET /api/v1/audit?tenant=` keeps only that tenant's hash-chained
records, so a per-customer audit extract is one export away. Audit reads are bounded: each chain file is read
from its tail and the scan stops once `limit` matching records are in hand, and `next_before` in the answer is
the cursor for the following page (`GET /api/v1/audit?limit=100&before=`). The *Activity*, *Usage* and *Audit trail*
pages offer the tenant filter.
**Tenant-pinned keys** are the isolation boundary for a customer or team you hand a key to:
`POST /api/v1/keys` with `{"name": "acme", "tenant": "acme"}` mints a key whose every request runs
as that tenant - the header is not needed, naming another tenant answers `403`, and the key can
never act as the whole workspace. Its requests count against the tenant's budgets and appear under
the tenant in usage and audit. The boundary holds on every surface: the key routes, executes and reports
outcomes as its tenant; it reads only its tenant's rows on `/activity`, `/trace/{id}`, `/events`, `/usage`,
`/usage/report`, `/savings`, `/audit`, `/handoffs`, `/keys`, `/tenants`, `/me` and `/governance` (another tenant's
request id answers `404`, `?tenant=` answers `403`); every workspace-management call - keys, policy,
tenants, providers, members, channels, billing, listings and the other reads of the workspace - answers `403`
with `X-Key-Tenant: `. A tenant with active pinned keys cannot be deleted (`409`); revoke
them first. The *API keys* page offers the tenant on creation and shows the scope of every key.
**Fair share** (enterprise, `OSR_PLATFORM_FAIR_SHARE=true`) keeps one workspace or tenant from crowding out the
others on a shared deployment: the SDK's dominant-resource-fairness middleware watches requests, tokens and cost per
workspace and per tenant over a sliding window (`OSR_PLATFORM_FAIR_SHARE_WINDOW_S`, 300 s), steers whoever takes
more than `OSR_PLATFORM_FAIR_SHARE_SOFT` (1.5) x its share to cheaper targets and caps premium targets above
`OSR_PLATFORM_FAIR_SHARE_HARD` (3) x - the trace shows `fairshare.throttled` / `fairshare.capped` events.
`GET /api/v1/governance` -> `fair_share` reports the window, the counters and the dominant shares of the caller's
own workspace and tenants only; `controls.fair_share` says whether it is on. Operators see every tenant across
workspaces on `/platform/admin/tenants` and can edit a tenant's configuration or delete it there
(`PUT|DELETE /api/v1/admin/accounts/{account_id}/tenants/{slug}`).
## 9. Governance: workspace policy, budgets and controls
A **workspace policy** applies to every request of the workspace, whichever key or tenant sent it. It
takes the same keys as a tenant plus budgets, and needs the `admin` role:
```bash
curl -s -X PUT "$OSR_URL/api/v1/policy" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"config": {"data_boundary": "private", "deny_targets": ["llm-frontier"], "monthly_budget_usd": 250}}'
```
`GET /api/v1/policy` returns the policy, the accepted keys and the current budget lines; `DELETE`
removes it. Constraints merge deterministically: a boundary or region fills in when the request has
none, cost and latency caps take the stricter value, deny lists are merged and allow lists intersected.
Unknown keys, unknown target ids and an id present in both `allow_targets` and `deny_targets` are
rejected with `400`.
**Budgets** count executed spend (`execute: true`, `/v1/chat/completions`, MCP `ask`) - quotes and
route-only calls are free. `daily_budget_usd` resets at midnight UTC, `monthly_budget_usd` on the first
of the month. Once a workspace or tenant budget is exhausted, routed calls return `429 ... budget ...
exhausted` with `Retry-After` and the `X-Budget-*` headers until the period rolls over or the limit is
raised.
**Request logging** is off by default: the platform keeps the decision facts of every request (signals, ranking,
policy rejections, execution steps, tokens, cost) but no prompt or answer text. A workspace that wants to read
them next to each trace opts in with `log_payloads` in the policy (or per tenant): `redacted` stores the prompt
and the answer with PII masked first (`opensmartroute.security.Redactor`: e-mails, phone numbers, card and
ID numbers become ``-style placeholders) and credential shapes masked as well (API keys, cloud and
repository tokens, private keys become ``), `full` stores them as sent. Each text is cut to
`OSR_PLATFORM_REQUEST_LOG_MAX_CHARS` characters (`truncated` marks the row). The stored payload appears as
`payload` on `GET /api/v1/trace/{request_id}` and on the trace drawer of the *Activity* page, `GET /api/v1/activity`
marks the rows that have one (`logged`) and reports the mode in force (`logging`), the data export carries every
row (`request_logs`), the rows follow the usage retention, and `DELETE /api/v1/activity/logs` (admin) erases
them all while the activity rows, traces and outcomes stay. The governance posture flags `full` logging as a
warning and `redacted` as an informational finding.
**Data protection** decides what happens to personal and health data *before* the router or any provider sees the
text - the policy key `data_protection` (per workspace, or per tenant in the tenant's config, which wins):
```json
{"config": {"data_protection": {"mode": "mask", "entities": ["pii", "phi"], "restore": true}}}
```
| `mode` | What happens to a request that carries personal data |
|---|---|
| `detect` (default) | Nothing is changed; the request is marked as carrying personal data, so it may only route to targets whose catalogue entry allows it (`pii_allowed`) - a public model pinned by name answers `422`. |
| `mask` | Every detected value becomes a labelled placeholder (``, ``, ``) in the current turn **and** in the conversation history before anything is routed, so the request may run on any target; the same value gets the same placeholder throughout. Targets that are allowed to see personal data still receive the originals. With `restore: true` (default) the placeholders in the answer - streamed too - are turned back into the values for the client; `false` hands the client the placeholders. |
| `block` | A request that carries any of the chosen entities is refused with `422` before any spend, naming the kinds found (`mrn (1), email (2)`), never the values. |
| `off` | The detection is switched off entirely (not even the routing mark). |
`entities` chooses what counts: the groups `pii` (e-mail, phone, card, national ids, passport), `phi` (medical
record number, NPI, DEA, Medicare id, insurance member id, ICD-10 code, date of birth) and `financial` (card, IBAN),
or single names; the default is everything. Every response carries `X-OSR-Data-Protection: ` when something
was found, and the usage row's `facts.pii` records the mode, the entity counts and whether the text was masked
(`GET /api/v1/activity`, the trace drawer) - the values themselves are never stored. Under `mask` the stored
request log (`log_payloads`) keeps the masked text. The same `facts` object carries `cache: hit`, `fallback_from`
(the targets that failed before the answer came from another one) and `upstream_request_id` - the provider's own id
of the call (`x-request-id` at OpenAI, `request-id` at Anthropic, `apim-request-id` at Azure), what a support
ticket with the provider needs.
**Output guard** decides what happens to an answer *before* the customer sees it - the policy key `output_guard`
(per workspace, or per tenant, which wins): `{"mode": "redact", "pii": true, "secrets": true,
"injection_threshold": 0.7}`. Every executed answer, streamed or not, is screened for leaked secrets (cloud and API
keys, tokens, private keys), personal data and instruction injection. `flag` reports the finding and leaves the
text; `redact` replaces secrets and personal data by placeholders (``, ``) - a streamed answer
is held until it was screened, then sent; `block` refuses an answer that carries a secret or an injection with
`400` and redacts personal data otherwise; `off` (default) leaves answers alone. `pii: false` keeps personal data
in place (a support answer may quote an address) while secrets still go. The finding - kinds and counts, never the
values - is on the answer (the `output_guard` field of the response's `opensmartroute` block on chat
completions, `result.output_guard` on `/api/v1/route`),
in the `X-OSR-Output-Guard` header (`redact:secrets=1,pii=2`) and on the usage row as `facts.output_guard`.
Dashboard: *Governance › Controls*, next to the data protection.
**Allowed addresses** fence every routed request of the workspace: `allowed_ips` in the policy (addresses or CIDR
blocks, at most 32) refuses a request from anywhere else with `403` and `X-Policy-Limit: allowed_ips` before
anything is routed, judged on the client address past the trusted proxies - never a header the caller wrote. A
tenant's own `allowed_ips` replaces the workspace's for its requests. Management calls (the policy itself, keys,
billing) are not fenced, so a locked-out address can always be fixed; `allowed_ips` on one key (above) fences that
key everywhere.
**Marketplace discovery** decides which published skills and personas a plan may use *without* an install
(section 11): `marketplace_discovery` is `verified` (the deployment default unless the operator changed it -
only listings an operator marked verified), `all` (every published listing) or `off` (the workspace's installs
only). Discovered companions fill plan slots and never answer on their own; every decision names the listings
it used under `marketplace`, each with `installed: true|false`.
**Default objective.** `objective` in the policy (`{"quality": 1, "cost": 0.6, "latency": 0.1}`, any of
`quality`, `cost`, `latency`, `quality_floor`) is the objective of every request that sets none; a request's own
`objective` still wins. It is what promoting a shadow candidate writes (the *Evaluation* page, section above).
**Response cache.** `response_cache_ttl_s` (seconds, up to seven days) answers an identical chat request the
workspace already paid for from the store instead of a provider: the same model name, messages, options and tenant
hash to one key, the stored answer comes back with zero tokens and cost on its usage row, a `cache` entry
(`{"hit": true, "stored_at": ...}`) in the answer's routing block and an `X-OSR-Cache: hit | miss | bypass` header. A caller shortens the life of one answer with
`X-OSR-Cache-TTL: 60` (never lengthens it past the policy), skips the lookup with `X-OSR-Cache: bypass`, keeps an
answer out of the store with `X-OSR-Cache: no-store`, or sets `osr.cache: false` in the body. `GET /api/v1/cache`
reports hits, misses and what the hits saved; `DELETE /api/v1/cache` (admin) empties the workspace's entries. The
store is Redis when the deployment has one, else the database.
`response_cache_mode: "similar"` adds near-duplicate matching: every entry also keeps a *context key* (the request
with its last user message blanked - model, earlier turns, tools, sampling) and a signature of that message (hashed
character shingles of its normalised text, never the text), and a request that misses the exact key is compared
with the entries of the same context and served from the closest one whose similarity reaches
`response_cache_similarity` (0.5-1, default 0.85). That catches the rewordings that make up most repeats - case,
punctuation, a typo, a word - without an embedding call per request; the answer's `cache` entry says
`"match": "similar"` with the score, which `X-OSR-Cache-Similarity` carries too. It is surface similarity, not
meaning: two questions that differ in one decisive word can score high, so keep the threshold where your traffic
makes it safe and leave `exact` for workspaces where that cannot be tolerated. Both keys are in the policy dialog.
**Session affinity.** `session_affinity: true` keeps a conversation on the target that served it: a request that
carries a session (`context.session_id`, or the hosted `osr.conversation_id`) routes to the target of the previous
turn while that target stays executable, allowed by the policy and not behind an open breaker; an explicit `model`
in the request still wins. Agent loops stop flipping models mid-task and prompt caches on the provider side keep
their prefix.
**Aliases.** `aliases` maps the model names your clients already send to what the router understands - a target id,
a catalogue model, an offer (`vendor/model@provider`) or a preset (`osr/auto:cheap`): `{"gpt-4o": "auto",
"claude-3-5-sonnet": "llm-frontier"}`. Resolution is one step and happens before everything else on
`/v1/chat/completions` (`model` and `models`), so a code base migrates to the router without a search-and-replace
of model names.
**Answer judge.** `judge` turns on the LLM judge for a share of the executed answers:
`{"sample": 0.1, "target": "llm-frontier", "threshold": 6, "endpoints": ["chat"], "targets": ["llm-small"],
"max_per_day": 500}`. The request id decides the sample (reproducibly), the judge model grades the answer against
the prompt (0-10 with a reason), the grade lands in the `feedback` table as `quality` with `source: judge:`
- exactly where a hand-reported outcome lands, so the learners, the rankings and the replay comparison use it -
and the verdict with its reason stays on `GET /api/v1/evaluations/judge`. Judging is a metered call of its own
(`endpoint: judge`) and runs on a thread after the answer has left; `max_per_day` bounds it. Section 5 has the
endpoints.
`GET /api/v1/governance` is the compliance view of the workspace in one payload:
| Block | Contents |
|---|---|
| `controls` | What the deployment enforces: `input_guard`, `pii_redaction`, `steering_strip`, `output_guard` (`flag` / `redact` / `block` or `false`), `metrics`, `tracing`, `audit`, `health_breakers`, `decision_cache`, `learning` (`persisted` or `in-process`). |
| `audit` | Whether the hash-chained audit trail is on, its length and the result of the last verification. |
| `retention_days` | How long usage rows are kept before the background sweep purges them. |
| `policy`, `policy_keys`, `budgets`, `spend` | The workspace policy, its accepted keys, budget lines with used and remaining amounts, spend today and this month. |
| `quota` | The same payload as `GET /api/v1/quota`: requests used this month against the plan's quota, `state`, reset time and the overage so far. |
| `tenants` | Each tenant with its monthly spend and budget lines. |
| `catalogue` | Targets per `data_boundary`, targets that refuse PII, executable targets, providers, and which targets remain reachable under the policy. |
| `keys` | Hygiene of the active API keys: `active`, `unscoped`, `no_expiry`, `expired`, `limited` (budgets or allow lists), `stale` (unused for 90+ days), `tenant_pinned`. |
| `guard_activity` | The guard verdicts of the last 24 hours counted from the workspace's trace events - `guard.blocked` / `guard.flagged` / `guard.redacted` (input guard), `output_guard.flagged` / `output_guard.blocked`, `route.no_route`, `tenant.rejected`; `null` when tracing is off. |
| `posture` | The compliance findings a reviewer would raise, most severe first, each with a stable `code`, a `severity` (`critical` / `warning` / `info`), the reason, the dashboard page that fixes it and a `scope`: `workspace` for what the customer can change (no budget, no reach policy, unscoped / non-expiring / expired / stale keys, an organization without SSO on a plan that includes it or without a role map) and `deployment` for what only the platform operator holds (guards or audit off, output guard only flagging, chain broken, tracing off, unbounded retention). `score` (0-100; a critical costs 25, a warning 10, an info 3) counts the **workspace** findings only; `deployment` carries the platform's own findings with their own `score`, so a customer is never judged by a setting they cannot touch. |
`GET /api/v1/governance/report` (admin role) is the evidence bundle behind that view: the governance payload plus
the members and their roles, the SSO connection (client id and role map, never the secret), every active key with
its scopes, limits, expiry and last use, the platform's identity (brand, versions, region) and who generated the
report when; `?format=markdown` downloads it as a Markdown document for an audit file. Every generation is recorded
in the workspace's audit log (`governance.report`).
The dashboard renders the same data at `/platform/dashboard/governance` - the posture card links each finding to
the page that fixes it, the guard-activity table links into the events page filtered on the event, the key-hygiene
card into the keys page, and "Compliance report" downloads the Markdown evidence - with a policy editor for admins.
## 10. Observability and caching
The platform observes itself: every span, event, counter and alert below is produced, stored and served
by the platform - there is no external metrics system, tracing backend or alerting tool to run. Every
deployment exposes liveness and readiness probes and the counters without a key:
| Endpoint | Meaning |
|---|---|
| `GET /healthz` | Process is up: edition, version, target count, uptime. |
| `GET /readyz` | `200` when the database, catalogue, router and (enterprise) audit file all answer; `503` with the failing check otherwise. Also reports `database_dialect` (`postgres`), `redis` and `events` (the event bus) - those two are informational and never fail readiness. |
| `GET /metrics` | The counters in text exposition format: `osr_platform_http_requests_total{method,route,status}`, `osr_platform_http_request_duration_ms_*`, `osr_platform_http_in_flight`, plus the router's `osr_route_decisions_total`, `osr_route_latency_ms_*`, `osr_outcome_*` and `osr_decision_cache_*`. |
Responses carry `X-Request-Id` (send your own to correlate with your logs) and `Server-Timing`; the
API writes one JSON access-log line per request (`osr.platform.access`) with the route template,
status, latency and account - never the prompt. `GET /api/v1/stats` (enterprise) adds `http`,
`controls`, `cache` and `observability` snapshots; `/platform/dashboard/health` shows them live.
### Time series and alerts
| Endpoint | Returns |
|---|---|
| `GET /api/v1/telemetry/series?window=24h` | The workspace's metered traffic bucketed over `1h` (per minute), `6h` (5 min), `24h` (15 min), `7d` (3 h) or `30d` (per day): requests, failures, `error_rate`, p50 / p95 latency, cost and tokens per bucket, totals per target and per endpoint, and the window total. Computed from the platform's own usage records, so it agrees with the activity log to the row. Plans with `stats` also get `http`: the deployment's requests, 4xx, 5xx and p50 / p95 / p99 latency per bucket from the per-minute counters the process keeps for 24 hours. |
| `GET /api/v1/alerts` | Active conditions, most severe first, each with the dashboard page that shows or fixes it. Workspace rules on every plan: the monthly request quota at 80 % (`warning`), exhausted (`critical`) or in overage (`warning`, with the estimated bill); a daily or monthly budget of the workspace or a tenant at 80 % (`warning`) or exhausted (`critical`); 20 % or more of the last 15 minutes' metered requests failing (10+ requests); prompts or answers blocked by the guards in the last 24 hours (`warning`, `guard_blocked`) and answers the output guard flagged for secrets, PII or injection (`info`, `output_guard_flagged`) - both link into the events page filtered on the guard events. Deployment rules on plans with `stats`: a failing readiness check, 5 % or more 5xx over the last five minutes (20+ requests), p95 above 2 000 ms, an open or half-open circuit breaker per target, a routing-drift alarm in the last hour, a failed autopilot cycle, a routing SLM last fitted more than 30 days ago, and tracing being off. The money rules - a prepaid provider's balance (`provider_balance`), a model sold below cost or under the margin floor (`margin`), an offer priced above the vendor's list price (`above_list_price`), Stripe Tax and the Stripe account (`stripe_tax_unconfigured`, `stripe_account_inactive`) - are the operator's alone: they appear on the console (`GET /api/v1/admin/health`, `/api/v1/admin/finance/alerts`), never on a customer's dashboard - see *Usage pricing, credits and provider finance*. `status_monitor` (`critical`): the status page's sampler has recorded nothing for three intervals, so `/status` says unknown. `backup_stale` (`critical`, operator only): the nightly `osr-db-backup` job records every finished archive in the `backups` table; when the newest is older than 36 hours the console's Health page and the operator notifier say so - a deployment where the job never ran shows nothing. |
The overview shows the last 24 hours (traffic, latency) and the alerts; `/platform/dashboard/health` adds the
deployment series and the same alerts, refreshed with the page.
### Notifications: alert delivery
Alerts are evaluated by the API itself - no external scheduler. Every `OSR_PLATFORM_ALERTS_INTERVAL_S`
seconds (default 60; `0` turns the loop off and leaves *Evaluate now* to operators) one replica - the one
holding the `osr:alerts:leader` lease in Redis, or the only one without Redis - re-evaluates the
workspace rules for every workspace that was active in the last 15 minutes or has a channel or an open
alert, plus the deployment rules. Each condition becomes a **notification episode**: opened when the rule
starts firing, refreshed while it stays on (an escalation from `warning` to `critical` is delivered
again), resolved when it clears. Episodes land in an inbox with unread counts per severity, are published
to the `alerts` Kafka topic in cluster mode, and are delivered to the workspace's **channels**:
| Kind | Target | Delivery |
|---|---|---|
| `email` | An address | Through the platform mailer (Microsoft 365, ACS or SMTP; otherwise the outbox at `/platform/admin/mail`). |
| `webhook` | An `https://` URL | `POST` of `{event, alert, workspace, url}` with `X-OSR-Event` (`alert.firing`, `alert.escalated`, `alert.resolved`, `alert.test`), `X-OSR-Delivery` and, when the channel has a secret, `X-OSR-Signature: t=,v1=` = HMAC-SHA256 of `.`; verify like Stripe signatures. |
| `slack` | An incoming-webhook URL | Block Kit message with severity, rule, value against threshold and a link to the page that fixes it. |
| `teams` | An incoming-webhook URL | MessageCard with the same facts. |
| `pagerduty` | An Events API v2 routing key (32 characters) | `trigger` on firing and escalation, `resolve` when the alert clears, one `dedup_key` per episode so PagerDuty keeps a single incident; severity, rule, value and the dashboard link travel as custom details. |
A channel filters by minimum severity (`info`, `warning`, `critical`) and optionally by rule names, can be
disabled, records its last delivery and last error, and has *Send test*. Failed webhook deliveries are
retried on the following evaluation rounds up to three attempts. Up to 20 channels per workspace; creating
or changing them takes the `admin` role. Operators have the same inbox and channels for the **deployment**
scope (readiness, 5xx rate, p95, circuit breakers, drift, autopilot, SLM age, tracing) at
`/platform/admin/notifications`.
**Account notices.** The same inbox also carries what happened to your own account, so a security message
does not depend on a mailbox that may be filtered, forwarded or shared. Each one arrives as an entry with
`state: "notice"` (it never fires or resolves - it is simply read), a `rule` of `security.*`, and a link to
the page that acts on it; the *Account* filter on `/platform/dashboard/notifications` shows only these. Every
one of them is sent by e-mail as well, under the `security` category that cannot be switched off.
| Notice | When |
|---|---|
| `security.new_sign_in` | The account was opened on a browser and system it has not been seen on for 30 days. Carries the device, the method, the address and the time on your own clock. |
| `security.password_changed` / `security.password_reset` | The password was changed from the account page, or a reset link was used. Every other browser is signed out. |
| `security.two_factor_on` / `security.two_factor_off` | The second factor was switched on or off by you. |
| `security.two_factor_reset` | An **operator** reset the second factor (the usual reason is a lost phone) and signed every browser out. |
| `security.sessions_revoked` | An **operator** signed the account out of every browser. |
| `security.sign_in_attempts` | The per-address password limit tripped - wrong passwords in a row. Nobody got in. |
| `security.sign_in_method_added` / `security.sign_in_method_removed` | An identity provider was added to, or removed from, the ways into the account. |
| `security.account_disabled` / `security.account_enabled` | An **operator** closed or re-opened the account. |
A notice belongs to one person: other members of the organization never see it, it follows its owner from one
workspace to the next, and it is not in the operator console's deployment inbox. Notices are not delivered to
channels either - a workspace Slack room is not the place for one person's sign-in.
**Workspace notices.** The same mechanism records what happened to the *workspace*, under `workspace.*` rules
and `scope: "workspace"`, so the people who live in the dashboard see it even though only the owners are
mailed. Every member of the workspace sees these; they stay in the workspace they belong to.
| Notice | When |
|---|---|
| `workspace.key_created` / `workspace.key_revoked` | An API key was minted or revoked, with who did it and the key's prefix. Revocation is a `warning` - anything still signing with it now gets `401`. |
| `workspace.member_joined` | An invitation was accepted, with the role taken. |
| `workspace.member_role` | A member's role changed, from what to what and by whom. |
| `workspace.member_removed` | A member was taken off the workspace and their sessions here ended (`warning`). |
| `workspace.plan_changed` | The plan moved - upgrade, downgrade, cancellation or trial end - with the reason. |
| Endpoint | Purpose |
|---|---|
| `GET /api/v1/notifications?state=firing&unread=true&limit=100` | The workspace inbox (`items`) and unread counts per severity (`unread`). `state` is `firing`, `resolved` or `notice`; the reader's own account notices are included whichever workspace they are in. |
| `GET /api/v1/notifications/unread` | Unread counts only - the dashboard bell polls this. |
| `POST /api/v1/notifications/read` | `{"ids": [...]}` marks those read, `{"ids": null}` marks everything read. |
| `GET|POST /api/v1/notifications/channels` | List channels (with the rule names you can filter on) or create one: `kind`, `name`, `target`, optional `secret`, `min_severity`, `rules`, `enabled`. |
| `PATCH|DELETE /api/v1/notifications/channels/{id}` | Change or remove a channel. |
| `POST /api/v1/notifications/channels/{id}/test` | Deliver a test notification and return `ok`, `status`, `error`. |
| `GET /api/v1/notifications/deliveries` | The last delivery attempts (channel, event, attempt, status, error). |
| `GET|POST /api/v1/admin/notifications` and `.../channels`, `.../deliveries` | The operator (deployment scope) equivalents; `POST /api/v1/admin/notifications/evaluate` runs one round now and returns what it opened, resolved, delivered and failed. |
**Platform events.** Besides alerts, a `webhook`, `slack` or `teams` channel can subscribe to what happens in
the workspace: `events` on the channel is a list of event names or patterns (`"*"`, `"key.*"`), validated
against `event_types` in the channel listing. The event types are `request.completed` / `request.failed`
(one per metered request - high volume), `outcome.reported`, `handoff.pending` / `handoff.resolved` /
`handoff.cancelled`, `key.created` / `key.revoked` / `key.rotated` / `key.scoped` / `key.limited`,
`member.invited` / `member.joined` / `member.left` / `member.removed` / `member.role`,
`team.created` / `team.updated` / `team.deleted` / `team.member_added` / `team.member_removed`, `plan.changed`,
`billing.invoice_paid` / `billing.payment_failed` (as Stripe confirms the subscription invoice),
`account.deleted`, `support.opened` / `support.answered` / `support.replied` / `support.closed`,
`governance.report` and `workspace.reports`. Every event is something the platform recorded - it is queued in
the outbox on the request path (one insert) and delivered by the next evaluation round with the same envelope
and `X-OSR-Event` / `X-OSR-Delivery` / `X-OSR-Signature` headers as alerts; `GET /api/v1/notifications/deliveries`
shows the attempts. The same publications reach the Kafka topics below when a broker is configured, so an
external consumer sees the deployment-wide stream and a workspace only its own events.
### Shared state and replicas
The platform keeps its state in PostgreSQL: `OSR_PLATFORM_DATABASE_URL` is required, and every API replica
and domain service connects to the same database. The data volume only holds what is local to a process
(learner state, the audit chain, the SLM bundle, the autopilot cache). The hash-chained audit trail has one
chain **per process**: the API appends to `audit.jsonl`, each domain service to `audit..jsonl`
(`audit.routing.jsonl`, `audit.openai.jsonl`, `audit.mcp.jsonl`), because a hash chain has exactly one
writer; `/api/v1/audit`, the trace view and the governance status verify every chain on the volume and merge
the records (`chains` counts them, `problem` names the broken file). On a clustered deployment (Redis set,
several replicas of one role) each replica writes its own file, `audit...jsonl`, because a
file lock on a network share (Azure Files mounted `nobrl`, NFS) is not honoured across hosts and two replicas
appending to one file interleave and break the chain; the readers merge all of them and the go-live checklist
(*Audit trail*) shows whether every chain verifies. On a single-host deployment two processes of one role
share a chain file safely: every append takes an exclusive file lock and continues from the last line
on disk, so a second writer extends the chain instead of forking it. A chain that no longer verifies (a line
removed, a record edited, a fork written before 1.8.1) is a critical deployment alert (`audit_chain_broken`)
and every workspace's compliance posture shows it; a superadmin resolves it from the console's *Health* page
(`POST /api/v1/admin/audit/rotate` with a reason): the broken file is kept beside the new one as
`..archived`, never edited, and the new chain opens with a `chain.rotated` record naming the
archive, the problem, the reason and the operator - the rotation is itself on the record and in the operator
audit log. Spans and events are persisted by every
process that routes or executes (routing, openai, mcp) and read by the API, so a chat completion served by
the OpenAI service is traceable at `/api/v1/trace/{request_id}` like a `/route` decision. A deployment that runs several
replicas moves the remaining shared parts to services that are themselves containers - no managed cloud
database, cache or broker is required, and the same images run on a laptop, a Kubernetes cluster or Azure
Container Apps:
| State | Service | Setting |
|---|---|---|
| Accounts, users, keys, tenants, policies, usage, feedback, providers, marketplace, notifications | PostgreSQL | \OSR_PLATFORM_DATABASE_URL\ (required) |
| Spans and events (logs) - kept out of the main database so log volume never contends with it | PostgreSQL (own database, created on first start) | \OSR_PLATFORM_TELEMETRY_DATABASE_URL\ |
| Rate-limit windows, learner state (enterprise: every replica learns read-modify-write into one shared state and adopts what the others learned every `OSR_PLATFORM_LEARNER_REFRESH_S` seconds, default 30), `/v1/responses` conversation state | Redis | `OSR_PLATFORM_REDIS_URL` |
| Platform events as JSON topics `osr.`: `usage` (one per metered request), `feedback`, `training` (prompt text for the SLM service, only with `OSR_PLATFORM_TRAINING_EVENTS`), `admin` (account lifecycle and operator actions), `alerts`, `billing` (plan changes, invoices), `handoffs`, `support`, `sales` | Kafka (any Kafka-protocol broker) | `OSR_PLATFORM_KAFKA_BOOTSTRAP`, `OSR_PLATFORM_KAFKA_TOPIC_PREFIX` |
| Photographs operators upload for the public site (`/platform/admin/photos`): the bytes, under a content-addressed name with an immutable cache header, in a container with public read; the database keeps only the address. Without a container the bytes stay in the database and the API serves them. | Azure Blob Storage (REST, no SDK) | `OSR_PLATFORM_SITE_PHOTOS_URL` (the container URL; only an `https://` address on `*.blob.core.windows.net` or a sovereign cloud's blob endpoint is accepted, anything else is ignored - the start-up check and the console's Photographs page both say so, and the start-up check also warns when a container has neither a SAS nor a managed identity to sign with), signed with either `OSR_PLATFORM_SITE_PHOTOS_SAS` (a container SAS with read, write and delete) or, on Azure, the container app's managed identity - `OSR_PLATFORM_AZURE_CLIENT_ID` names the user-assigned identity, which needs *Storage Blob Data Contributor* on the account; the Azure template sets all of this up |
`GET /api/v1/admin/overview` -> `storage` says which mode a deployment runs (`{database, clustered, redis,
events, topics}`; `clustered` is true once Redis shares the rate limits, `topics` is the full list above as
the broker sees it) and the operator console shows it on
`/platform/admin`. The public `GET /api/v1/info` never names where state lives: that is an operator fact.
Event payloads carry the same fields as the activity log and never the prompt text. The
topic list is one constant (`cluster.TOPICS`): the Azure template creates exactly these event hubs for a
managed broker (Event Hubs does not auto-create topics), a self-hosted broker creates them on first use, and
a test fails when code publishes on a topic that is not in the list.
An authenticated broker - Event Hubs, Confluent Cloud, any SASL or TLS endpoint - is reached with
`OSR_PLATFORM_KAFKA_SECURITY_PROTOCOL` (`SASL_SSL`, `SASL_PLAINTEXT`, `SSL`), `OSR_PLATFORM_KAFKA_SASL_MECHANISM`
(`PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`) and the `OSR_PLATFORM_KAFKA_SASL_USERNAME` /
`OSR_PLATFORM_KAFKA_SASL_PASSWORD` pair; for Event Hubs that is the literal user `$ConnectionString` and the
namespace connection string. Such a gateway hangs up on a connection left idle for a few minutes - and so does the
network in front of a self-hosted broker (the internal ingress of Azure Container Apps after four) - while the
driver would hold one for nine, so the clients, whatever the protocol, release theirs first after
`OSR_PLATFORM_KAFKA_CONNECTIONS_MAX_IDLE_MS` (default 180000, three minutes; `0` leaves the question to the
driver). Without it a quiet topic finds the socket already gone on its next read: no event is lost - the producer
retries and the consumer rebuilds - but the log fills with `Connection lost: socket disconnected` and
`ConnectionResetError: [Errno 104] Connection reset by peer` under *Error receiving network data*, once per idle
connection, every few minutes in every service.
Both shared services degrade rather than stop the API. When Redis is unreachable the rate limiter answers
from its per-replica window and stays off Redis for a five-second cool-down before trying again (`/readyz`
`redis` shows `ok`, `latency_ms` or the error); the shared learners keep learning in memory - a refresh that
cannot read Redis leaves the replica's state untouched instead of blanking it, and the next successful save
catches up. A request never waits for Kafka: `emit` only enqueues, one sender thread per process hands the
events to the producer (queue bounded at 10 000 events; beyond that the newest are dropped and counted as
`events.overflow`). Events are acknowledged by every in-sync replica (`acks=all`), retried with back-off and
dropped - counted and reported as `events.dropped` / `queued` / `last_error` - only when the broker stays
unreachable. The SLM service consumes `training` and `feedback` with manual offset
commits after each ingested batch (at-least-once: a record is never committed before it was processed) and
reports the consumer group, commits, errors and per-topic lag under `consumer` in `GET /model`.
The self-hosted Compose stack
starts PostgreSQL with the platform; the Azure template provisions it as a container app with internal TCP
ingress next to the API and web app.
A deployment that predates 1.1 kept its state in a `platform.sqlite3` file on the data volume. Import it
once with `osr-platform-import-sqlite /data/platform.sqlite3 --database-url postgresql://...` (ships
with the platform API; run it inside the API container -
safe to re-run; rows already in PostgreSQL are left alone), then remove the file.
In that stack the routing SLM is its own service too: the API publishes each routed prompt to the
`training` topic (opt-in, `OSR_PLATFORM_TRAINING_EVENTS`) together with the decision it took and the ranked
propensities of the top candidates (`target_id`, `ranked`); the SLM service remembers prompts, decisions and
workspace in `autopilot/requests.jsonl` on the shared volume - a restart or a deploy loses nothing - pairs
prompts with the outcomes reported through `POST /api/v1/feedback` (feedback rows) and distils the ensemble's
ranking on every routed prompt that has no outcome yet (decision rows, newest `OSR_SLM_DECISION_ROWS_MAX`),
trains a challenger on a schedule, keeps it only when it beats the champion on a held-out split, and writes
the promotion to the shared volume from which every API replica reloads it (a timer every
`OSR_PLATFORM_SLM_RELOAD_S` seconds in every process that routes, idle replicas included). Synthetic seed rows are added only
while real rows are scarce (`OSR_SLM_SYNTHETIC_PER_TEMPLATE` rows per prompt template - 8 in the shipped stacks,
about 500 rows and a 100-row holdout, so the gate can tell two architectures apart before real traffic
arrives); `GET /model` reports `prompts_remembered`, `decisions_remembered`, `outcomes` and the
cycle's `sources` (feedback, decisions, datasets, synthetic). The trainer builds every challenger with the
query encoder `OSR_SLM_ENCODER` names (`attention` in the shipped stacks - a transformer block over the token
sequence on top of the hashed features, about five milliseconds per prediction; `hashed` for the bag-of-features
encoder alone): when it differs from the champion's the challenger is a fresh model with the configured
architecture and the gate decides, so the served model changes encoder only where it wins the holdout;
`GET /model` shows `encoder`. `OSR_SLM_EMBEDDER` adds a real embedding model to every challenger as frozen dense
features (`openai:@`, any OpenAI-compatible `/embeddings`): the Azure stack points it at the
embedding model the deployment already serves for `embed-small` (its model server's `/v1`, nothing leaves the
environment; `azd env set OSR_SLM_EMBEDDER none` keeps the hashed and attention features only, any other value names
an endpoint of your own), Compose takes it from the same variable (`openai:qwen3-embedding:4b@http://ollama:11434/v1`
with the `llm` profile). The SLM then learns its head on what a sentence transformer knows a prompt means, while the
hashed features keep the surface cues; the promoted file names the embedder, so every API replica rebuilds the same
client on reload (`OSR_SLM_EMBEDDER_TIMEOUT_S` 5 s in the API, 120 s for the trainer's batches) and a request the
endpoint cannot embed in time is scored by the other strategies alone. Both start from the warm-start model the API image ships at `/app/models/slm.json`
(`OSR_PLATFORM_SLM`) - the attention model bundled inside the SDK package, trained on the example catalogue, its
labelled dataset and follow-up conversations and measured on a held-out seed - so a fresh deployment routes with
a learned strategy from the first request and the first promotion has a champion to beat. Operators see the
model in service, the evidence ingested, every cycle and a
prompt probe on `/platform/admin/slm` (`GET /api/v1/admin/slm`, `/slm/reports`, `POST /slm/cycle`, `/slm/predict`).
**The labelled floor.** The holdout a promotion is judged on is self-agreement: the router's own decisions on the
deployment's traffic. On a deployment whose traffic is one kind of job (production's was its own blog and SEO
writers) the served model promoted itself twenty times on holdouts of 0.92-0.97 and routed 23 of the 70
hand-labelled reference rows acceptably - and, heard in the ensemble, pulled it from 66 to 39 of 70. The SDK
package now ships that labelled set next to the bundled model, and every promotion is held to it
(`OSR_SLM_LABELLED_FLOOR`, on): a challenger that routes fewer of the applicable labelled rows acceptably than the
champion, by more than `OSR_SLM_LABELLED_TOLERANCE` (5 %) of them, is refused whatever its holdout says, and the
report carries both scores under `labelled`. At start the service holds the champion it loads from the volume to
the same floor against the packaged model and adopts the packaged model when the champion fell below it (the
replaced model stays as `slm.prev.json`; `GET /model` -> `floor` says what happened). Rows count only when the
deployment's catalogue holds their targets; with fewer than `OSR_SLM_LABELLED_MIN_ROWS` (20) such rows the
floor is off, so a catalogue of your own is judged by its own evidence alone.
**Rehearsal.** A floor alone would leave the loop stuck: a challenger fitted on one workspace's traffic learns that
traffic perfectly (31 holdout wins, 0 losses) and forgets the reference set (44 of 70), so the floor refuses it -
every cycle, for ever, and nothing is ever promoted. The applicable labelled rows therefore also *train* every
challenger, replicated until they are about `OSR_SLM_LABELLED_REHEARSAL` (0.25) of its training rows (0 = off);
the holdout stays the deployment's own rows. The same challenger then keeps 69-70 of the 70 reference rows while
it learns the traffic, and is promoted; the report's `sources` count the rows under `labelled`.
The same cycle trains the model's **slot models**: the outcomes credited to a plan's persona, skill and tool
slots (`role` on the `feedback` topic - the platform's own assessment of every composed answer carries one per
companion, `POST /api/v1/feedback` takes one) become one small encoder per role, saved in the same file and
promoted with the primary, so the ensemble's SLM also votes on which persona or skill a request wants - a
marketplace listing that gathered a record included. A role trains from `OSR_SLM_SLOT_MIN_ROWS` (20) rows on;
each cycle report lists the roles under `slots` (rows, accuracy, skill, or why the role was skipped) and
`GET /api/v1/learning` shows the served slots under `slm.slots`.
Workspaces do not all route alike, so the service also trains **per-workspace models**: once a workspace has
`OSR_PLATFORM_SLM_WORKSPACE_MIN_ROWS` (100; `0` = off) rows of its own - outcomes joined to prompts
through the `training` topic plus its routed prompts' decision rows, attributed by `account_id` - each shared cycle also trains a challenger for it,
starting from the shared champion and seeing only that workspace's rows. It is promoted to
`autopilot/slm..json` under the same gates (a ten-row holdout at least, a significant win over the
model it replaces - the shared champion the first time), and the API replicas pick the file up with the
shared one and serve it for that workspace's requests only; everyone else keeps the shared model. Each of these
models is a multi-megabyte bundle every replica holds in memory and every cycle trains in series, so a
deployment keeps at most `OSR_PLATFORM_SLM_WORKSPACE_MAX_MODELS` (20; `0` = no cap) of them - the workspaces
with the most evidence.
`GET /api/v1/learning` lists them under `workspace_slms`, the service's `/model` under `workspaces`, and
`POST /predict` takes an `account_id` to rank with a workspace's model.
### Services
The API is one image that can run as one process or as several: the documentation JSON (`docs`), the
public model catalogue and rankings (`rankings`), the marketplace (`marketplace`), provider management
(`providers`), signup and account links (`onboarding`), identity and workspaces (`accounts`: sign-in,
sessions, organizations, keys, tenants, policy, governance, usage), Stripe billing (`billing`), the
operator console API (`admin`), the MCP endpoint (`mcp`), the OpenAI-compatible proxy (`openai`), the
routing core with its trace, event and learning reads (`routing`) and the routing SLM (`slm`) each have
their own entry point and port. `/api/v1/info`, `/api/v1/status` and `/api/v1/estimate` always stay
with the API. The API remains the single public origin and forwards a domain's paths to its
service when `OSR_PLATFORM__URL` is set (streamed completions are relayed as they arrive; a service
that stays silent longer than `OSR_PLATFORM_PROXY_TIMEOUT_S`, default 300 s - a cold self-hosted model
loading its weights - answers `503`); the
answer carries `X-OSR-Service` naming the process
that served it. The topology is an operator fact: the public `GET /api/v1/info` does not list it; operators see
the map and live health per service on `/platform/admin/services` (`GET /api/v1/admin/services`).
The Compose stack and the Azure template
deploy every service as its own container next to the gateway.
### Traces and events
The router records every request as a tree of **spans** (`request`, `route`, `plan`, `execute`) with
**events** inside them (`route.signals`, `route.policy`, `route.rank`, `route.fallback`, `guard.*`,
`execute.step`, `learn.outcome`, `cache.hit`, ...). The newest ones stay in an in-memory buffer (1000
by default, `OSR_OBSERVABILITY_MEMORY_EVENTS`) and every one of them is also written to the platform
database by the platform's own telemetry store (`OSR_PLATFORM_TELEMETRY_STORE`, on by default) - off
the request path, from a background writer, so tracing never slows routing. Events and traces therefore
survive restarts, can be read by time range and are purged after `OSR_PLATFORM_TELEMETRY_RETENTION_DAYS`
(14). They are readable per workspace:
| Endpoint | Returns |
|---|---|
| `GET /api/v1/trace/{request_id}` | The activity row plus every span and event of one request, in time order, with `trace_id`, `spans` and the root `duration_ms`; `outcomes` (the feedback reported for the request), `audit` (its hash-chained records, plans with `audit`) and `payload` (the stored prompt and answer when the workspace's `log_payloads` policy kept them, else `null`). `source` says whether the rows came from the buffer or the store. `404` when the request is not the workspace's; `events: []` once the request is older than the telemetry retention. |
| `DELETE /api/v1/activity/logs` | Erases every prompt and answer stored under the `log_payloads` policy (admin role); activity rows, traces and outcomes stay. Returns `deleted` and the `logging` mode still in force. |
| `GET /api/v1/events?name=route.*&kind=&level=&request_id=&since=&until=&limit=200` | Events of the workspace's requests (oldest first, `limit` up to 2000; `name` may end with `*`; `since` / `until` are epoch seconds, default the last 6 hours). Deployment-wide events without a request id (breaker transitions, autopilot) are included on plans with `stats`. `source`, `persisted`, `retention_start` and `retention_days` describe the store. `404` when tracing is off entirely. |
| `GET /api/v1/events/stream?name=&kind=&level=&request_id=&max_s=3600` | The same events pushed as they are recorded, as server-sent events (`text/event-stream`): one `event: ` / `data: {…}` frame per event, a `: ping` comment every 15 seconds while quiet, and the stream ends after `max_s` seconds (default and ceiling one hour) - reconnect to continue. Every replica reads the shared store, so the stream sees every replica's events. Same filters and `404` as the list endpoint. |
| `GET /api/v1/status` | Public readiness: the `/readyz` checks plus edition, versions, uptime, controls and tracing state; `503` while a check fails. `Cache-Control: no-store`. The public status page of the site (`/status`) polls it every 15 seconds underneath the board below, so a browser that cannot reach the API says so at once. |
| `GET /api/v1/status/board` | The public status page (`/status`): `overall` (`operational`, `degraded`, `partial_outage`, `major_outage`, or `unknown` when the platform's own checks have not run for three intervals - stale checks are never shown as operational), `monitor` (`enabled`, `last_sample_at`, `stale`, the last pass's `error`), every **component** the platform samples itself - the API's readiness, the database, the cache, the event bus, mail delivery, each remote domain service, the web app and every enabled model provider - with its last check (`status`, `latency_ms`, `checked_at`, `detail`), `uptime_30d_pct` / `uptime_90d_pct` and the per-day `history` (checks, failures, latency), the `incidents` (`open`, `recent` = the last 30 days) with their updates, and `served` / `served_90d`: metered requests per UTC day and the share the platform failed with a 5xx. One replica samples every `OSR_PLATFORM_STATUS_SAMPLE_S` seconds (default 60; 0 = off) and keeps `OSR_PLATFORM_STATUS_HISTORY_DAYS` (default 90) of roll-up; raw samples live a week. The web app is checked at `OSR_PLATFORM_WEB_URL` - or at `OSR_PLATFORM_WEB_PROBE_URL` when the public address does not resolve from the API's own network (compose sets `http://web:3000`). A core component (`api`, `database`) or a domain service that fails three checks in a row **opens an incident on its own** (`automatic: true`, severity `critical` for a core component) and resolves it after three healthy checks; a provider down degrades the banner but never opens one - the router fails over. Cached 10 s (`ETag`). |
| `GET /api/v1/status/incidents?days=90&limit=50`, `GET /api/v1/status/incidents/{id}` | Incidents opened or active in the window, newest first, each with `severity` (`minor` / `major` / `critical`), `status` (`investigating` / `identified` / `monitoring` / `resolved`), `components`, `started_at`, `resolved_at` and every update. |
| `GET /api/v1/status/feed` | RSS 2.0 of incident updates (one item per update) - what "subscribe to status updates" points at. |
| `GET /api/v1/services` | What this deployment offers as a service, computed from its state at the time of the call (no key): every first-party capability - routing, the OpenAI-compatible proxy, quotes, plans, learning, the routing SLM, your own providers, human hand-off, the MCP server, traces, savings, governance, tenants and audit, alerts, usage digests, support, the marketplace, the model catalogue, the blog, the newsletter, single sign-on, billing, the CLI, self-hosting, the browser extension and desktop app - with `status` (`live` works now, `limited` runs with a stated restriction such as no model provider or no mail transport, `off` not on this deployment or edition), `detail` (the reason), `endpoint` (`METHOD /path`, always a real route), `access` (`public` / `key` / `operator`), `plan` (the cheapest plan that includes it), `docs`, `page` and `facts` (small live figures from the same stores the dashboards read). The site renders it at `/services`; the welcome mail and every newsletter issue link to it. Cached for a minute (`ETag`). Nothing about the platform's own editorial use of the router is included. |
| `GET /api/v1/facts` | The public numbers of this deployment, each section carrying its **class** so no page can put a reference-catalogue count, an imported index and a live counter in one sentence by accident (no key): `version` (platform, SDK and the build label), `catalogue` (class `catalogue`: the reference model index - `models`, `vendors` - indexed, not necessarily executable here), `storefront` (class `live`: `models` sold by the token, `endpoints`, `executable_models`), `deployment` (class `live`: `targets`, `executable_targets`, `kinds`, `providers`, `vendors`, whether the routing `slm` is loaded and the `autopilot` runs), `activity` (class `live`: lifetime `requests`, `requests_24h`, `workspaces`, `workspaces_routing`, `executed_requests`, the measured `saved_usd` / `saved_pct` against `baseline`) and `marketplace` (class `imported`: `indexed` listings and `publishers_indexed` in total, `installs`, `reviews`, and `native` / `imported` / `sources` once the manifest scan has run - `null` before). `classes` spells each class out in the words the site prints next to the number. The site renders it at `/whats-live` together with the services catalogue and the provider table. Cached for a minute (`ETag`). |
`POST /api/v1/route` responses include `trace_id` and the `X-OSR-Trace-Id` header when the request was
traced, and `GET /api/v1/activity` marks rows whose spans are kept (in memory or in the store) with
`traced: true` and summarises the feedback reported for each request in `outcome` (`POST /api/v1/feedback`
is stored per request, so the loop from decision to result is visible per row). A caller's **W3C `traceparent`**
header is honoured: the request's spans join the caller's trace (`X-OSR-Trace-Id` returns that trace id), and a
well-formed client `X-Request-Id` (8-64 characters of `A-Za-z0-9._:-`, not yet used by the workspace) becomes
the decision's `request_id` - what `/feedback` and `/trace/{request_id}` take. The TypeScript SDK, the browser
extension and the desktop app send both on every call by default (`trace: false` on the SDK client turns it
off), so every request of every client is one traceable row; the extension's side panel and the desktop's answer
line link *answered by* to the request's trace on the dashboard. Attributes hold ids,
numbers and short labels; the request text is represented only by its digest and length. The dashboard
shows the history at `/platform/dashboard/events` (window of 1 hour to 7 days, search by request or trace id),
opens a trace from any activity row (with a *How it was routed* story: what the request looked like, who
was allowed to answer, how the candidates ranked with their expected quality, price and latency, *why* the
winner beat the runner-up strategy by strategy with each vote's rationale, the plan slots filled or
skipped, the decision and the execution steps with tokens and cost) and adds a *Trace* tab to the
playground. `GET /api/v1/trace/{request_id}` also returns `usage`: tokens (prompt / completion), the metered
cost, the target's list price and what the same tokens would have cost on the savings baseline (the dearest
model the workspace itself ran in the last 30 days, or the catalogue's dearest when it ran none - the same rule
as `GET /api/v1/savings`; `usage.baseline.source` says which), so every request shows what it saved.
The deployment-wide figures the site shows (`GET /api/v1/stats/public`: `requests`, `tokens`, `cost_usd`,
`success_rate`, `top_targets` and the `savings` block, plus `internal` for the deployment's own jobs) are read
from a lifetime ledger (`platform_counters`, one row per target and scope) that every metered request adds to
and nothing subtracts from: the retention sweep, a workspace's own `retention_days` and a deleted workspace
remove request rows, never what has already been counted. `since` is the oldest counted request. The ledger
seeds itself from the request rows a deployment still has the first time it is read, so an upgrade starts from
today's totals rather than from zero. Only the windows (`requests_24h`, `tokens_24h`, the latency figures) and
`accounts_routing` follow the rows that exist.
Public catalogue reads (`/api/v1/info`, `/api/v1/models*`, `/api/v1/rankings`, `/api/v1/llms*`,
`/api/v1/catalogue`, `/api/v1/stats/public`) return an `ETag` and `Cache-Control: public, max-age=60,
stale-while-revalidate=60`; send `If-None-Match` to get `304` when nothing changed. Operators tune this
and the decision cache with:
| Setting | Default | Effect |
|---|---|---|
| `OSR_PLATFORM_METRICS_PUBLIC` | `true` | `false` restricts `/metrics` to requests carrying `X-Admin-Token`. |
| `OSR_PLATFORM_TRUSTED_PROXIES` | `private` | Which `X-Forwarded-For` hops are the deployment's own proxies (ingress, web proxy, gateway) when working out a caller's address for per-IP limits and audit records: `private` skips every RFC 1918, loopback and link-local hop, `none` uses the TCP peer only, or list CIDRs. Entries a client wrote itself are never trusted. |
| `OSR_PLATFORM_GEOIP_HEADER`, `OSR_PLATFORM_GEOIP_SERVICE`, `OSR_PLATFORM_GEOIP_TOKEN` | unset | Where sign-ins come from: a request header the ingress sets with the country code, and / or an IP geolocation service (a preset or a URL template with `{ip}`) with its key. See [Sign-in locations and alerts](#sign-in-locations-and-alerts). |
| `OSR_PLATFORM_NETWORK_DATA_RETENTION_DAYS` | `365` | Days the address and the place of sessions and account events are kept before the retention sweep removes them (the events stay); `0` keeps them as long as the row. |
| `OSR_PLATFORM_EVAL_GATES_INTERVAL_S` | `21600` | How often every workspace's quality gates are routed again and judged (six hours); `0` = only on request and after a policy change. See [Evaluation](#evaluation). |
| `OSR_PLATFORM_HTTP_CACHE_MAX_AGE_S` | `60` | `Cache-Control` lifetime of public reads; `0` sends `no-store`. |
| `OSR_PLATFORM_STATS_ROLLUP_S` | `120` | How often the public aggregates that scan the request table (lifetime latency, workspaces routing, per-target traffic of the week) are recomputed; every replica and service reads the one stored copy, a stale copy is served while one process refreshes it. `0` computes on every read. |
| `OSR_PLATFORM_ROUTE_CACHE_TTL_S` | `0` | Reuse a routing decision for identical requests within this window (`0` = off). |
| `OSR_PLATFORM_CHAT_TOOL_MIN_CONFIDENCE` | `0.5` | A built-in tool (calculator, web search) answers `/v1/chat/completions` only when the router is at least this confident it fits the request; below it the chat is routed again without tools, and a tool is never the fallback for a failed model. When every runnable model's breaker is open the chat answers `503` with `Retry-After`. `0` lets tools compete like any target. |
| `OSR_PLATFORM_FAIR_SHARE` | `false` | Enterprise: dominant-resource fair share across workspaces and tenants (`_WINDOW_S` 300, `_SOFT` 1.5, `_HARD` 3 tune it; section 8). |
| `OSR_PLATFORM_OUTPUT_GUARD` | `flag` | Response-side guard on every model / agent answer (`opensmartroute.security.OutputGuard`): `flag` records leaked secrets (API keys, private keys, JWTs), personal data and instruction injection in the trace and under the `output_guard` entry of the answer's metadata block; `redact` also replaces secrets and PII with placeholders; `block` refuses the answer (`400`); `off` disables it. Under `redact` / `block` a streamed answer is buffered and screened before its first token leaves. |
| `OSR_PLATFORM_RETENTION_DAYS` | `365` | Usage rows older than this are purged every six hours; `0` keeps everything. |
| `OSR_PLATFORM_REQUIRE_EMAIL_VERIFICATION` | `true` | Hold every account at the confirmation step until the signed-in user - or, for an API key, an owner of the workspace - has confirmed the e-mail address (`403`, `X-Email-Verification: required`, everywhere but `/me`, `/onboarding`, `/auth/*`, `/legal`). `false` only on a stack that is thrown away afterwards. |
| `OSR_PLATFORM_SIGNUP_TEST_DOMAINS` | `false` | Accept signups from domains reserved for testing (`example.com`, `*.test`, `*.local`, ...). The smoke, load and e2e suites need it on their throw-away stacks; production keeps it off so test users never land in the customer database - the operator's own doors (`POST /api/v1/admin/users`, `POST /api/v1/admin/accounts`, members added by e-mail) refuse those domains too, and the retention sweep removes anything on such a domain that still got in (audited as `test_data.purged`, actor `system`). Operators list what a run left behind with `GET /api/v1/admin/test-data` and remove it at once with `DELETE /api/v1/admin/test-data` (superadmin, audited). |
| `OSR_PLATFORM_REFERRAL_ENABLED` | `true` | Run the referral programme: a workspace's link earns both sides credits (section 6, *Referrals*). |
| `OSR_PLATFORM_REFERRAL_REWARD_CENTS` | `500` | Credits (cents of the wallet) the referrer earns per qualified referral. |
| `OSR_PLATFORM_REFERRAL_WELCOME_CENTS` | `500` | Credits the invited workspace starts with once it qualifies. |
| `OSR_PLATFORM_REFERRAL_MAX_REWARDS` | `20` | Rewards one referrer may earn; `0` = no cap. |
| `OSR_PLATFORM_CAPTCHA_PROVIDER` | `off` | A bot challenge on the sign-up form: `turnstile` (Cloudflare) or `hcaptcha`. The form renders the provider's widget and the API verifies its token with the provider before anything else runs; a missing or failed token is refused (`400`, `X-Signup-Refused: captcha`). Needs the two keys below. |
| `OSR_PLATFORM_CAPTCHA_SITE_KEY` | - | The provider's public site key (published on `GET /api/v1/info` as `captcha`). |
| `OSR_PLATFORM_CAPTCHA_SECRET` | - | The provider's secret the API verifies tokens with; never leaves the API. |
| `OSR_PLATFORM_UNVERIFIED_ACCOUNT_TTL_DAYS` | `14` | A user who never confirmed the address is removed after this many days, with the personal workspace only they belonged to (the retention sweep); `0` keeps them. The platform's own `*.local` service workspaces are exempt. |
| `OSR_PLATFORM_REQUEST_LOG_MAX_CHARS` | `20000` | Longest prompt / answer kept per request under a `log_payloads` policy (longer texts are cut and marked `truncated`). |
| `OSR_PLATFORM_TELEMETRY_STORE` | `true` | Persist every span and event to the platform database (the events page, traces and `since` / `until` queries outlive the in-memory buffer). |
| `OSR_PLATFORM_TELEMETRY_DATABASE_URL` | unset | A separate PostgreSQL database for the telemetry store (e.g. `postgresql://osr:pw@host:5432/osr_telemetry`), so span/event volume never contends with the main database; created on first start when missing. Unset keeps them in the main database. |
| `OSR_PLATFORM_TELEMETRY_RETENTION_DAYS` | `14` | Persisted spans and events older than this are purged with the same sweep; `0` keeps them. |
| `OSR_PLATFORM_LOG_EXPORT_TICK_S` | `30` | How often the accounts owner looks for request-log export destinations whose interval has passed and ships their next batch (one replica per tick); `0` = only on *Export now*. |
### Request-log export to your own storage
The usage rows and the kept payloads stay queryable through the dashboard and the API; a team that runs its own
data lake or SIEM wants the same rows where its other logs are, continuously, without polling
`GET /api/v1/activity`. A **log export** does that: `POST /api/v1/workspace/log-export` names a destination and
the platform writes every new row there in batches of up to 2 000, on the destination's own `interval_s`
(60 s and up), with a cursor so nothing is sent twice and nothing is skipped when a batch fails (the error is
recorded, the cursor stays, the next pass retries). Destinations: `azure_blob` - a container URL
(`https://.blob.core.windows.net/`) with a container SAS as `credential` (or the API's
managed identity when it is empty), one JSONL blob per batch under `/YYYY/MM/DD/--.jsonl` - and `webhook` - a `POST` of the JSONL batch (`application/x-ndjson`, `Authorization: Bearer
`, `X-OSR-Export-Batch`) to an HTTPS URL: Splunk HEC behind a shim, Datadog Logs intake, a Kafka
REST proxy, Logstash, a function. Every line is one request with its metering fields (`account_id`, key,
member, tenant, project, endpoint, target, provider and upstream model, tokens, cost and charge, latency,
status); `include_payloads: true` adds `prompt` and `response` from the kept payloads (redacted when the policy
says so, `payload_redacted` marks it); `start_from` exports from a past timestamp (default: from now on). The
credential is stored encrypted like a provider key and never returned (`credential_set` only). `GET` lists the
destinations with cursor, batch and row counts and the last error; `PATCH` renames, re-points, rotates or clears
the credential (`clear_credential`), changes the interval, pauses (`enabled: false`); `POST .../{id}/run` ships
the next batch now and reports what went; `DELETE` removes it. Feature `byok` (Pro and up), up to five
destinations; writes take the `manage` scope and are audited (`log_export.created`, `log_export.deleted`).
**Traces for your observability tool.** Two more kinds ship the same rows shaped as traces, with the outcomes
reported so far and the answer judge's grade as scores: `langfuse` - `url` is the Langfuse host
(`https://cloud.langfuse.com`, a regional or self-hosted one), `credential` is `:`; one
`POST /api/public/ingestion` per batch with a `trace-create` per request (input / output when payloads are
included, user, tags, metadata), a `generation-create` (model, token counts, cost, latency, level) and a
`score-create` per outcome - and `otlp` - `url` is an OTLP/HTTP traces endpoint (the standard `v1/traces` path:
Arize Phoenix at app.phoenix.arize.com, Arize AX at otlp.arize.com, a collector), `credential` a
bearer token or `header=value;header=value` pairs (Arize: `space_id=...;api_key=...`); spans carry OpenInference
attributes (`openinference.span.kind` `CHAIN` for the routing decision and `LLM` for the generation,
`llm.model_name`, `llm.token_count.prompt|completion|total`, `input.value` / `output.value`, `session.id`,
`user.id`, `osr.*` fields and `osr.score.`), which Arize Phoenix, Arize AX and TruLens read natively and
every OpenTelemetry backend stores as attributes. Trace and span ids derive from the request id, so a client that
forwards `X-Request-Id` into its own tracing can join the two. [docs/AGENT_ENGINEERING.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/AGENT_ENGINEERING.md)
places this next to the other evaluation tools.
### Monitoring: what is always on, and Application Insights on Azure
Every deployment monitors itself without another tool: the telemetry store above, `GET /metrics` (Prometheus text
exposition: HTTP counters and latency per route, routing decisions, outcomes, cache), `GET /readyz` (database,
catalogue, router, audit path, plus Redis / event bus / mail health), `GET /api/v1/status`, the deployment alerts and
the operator console's *Health* page (`GET /api/v1/admin/health`). On **Azure** the platform additionally exports
to **Application Insights** - the Azure template and the Marketplace offer create a workspace-based resource next
to the Log Analytics workspace and hand every platform container its connection string:
| Setting | Default | Effect |
|---|---|---|
| `APPLICATIONINSIGHTS_CONNECTION_STRING` | unset | The connection string of the Application Insights resource (the variable every Azure SDK reads; set by the Azure template). `OSR_PLATFORM_APPINSIGHTS_CONNECTION_STRING` is the same thing under the platform's own name. |
| `OSR_PLATFORM_APPINSIGHTS` | `auto` | `auto` exports when a connection string is present **and** the process runs on Azure (Container Apps and App Service announce themselves through their environment); `on` exports wherever the string is set - a self-hosted stack reporting into an Azure subscription; `off` never. |
| `OSR_PLATFORM_CLOUD` | detected | `azure` names the cloud explicitly for hosts that do not announce it (an AKS node, a VM); the Azure template sets it. |
| `OSR_PLATFORM_APPINSIGHTS_SAMPLE_RATE` | `1` | Share of operations exported - a request together with its spans and events is kept or dropped as a whole, so traces stay complete. Lower it before the ingestion bill does. |
| `OSR_PLATFORM_APPINSIGHTS_EVENTS` | `true` | Also export the tracer's events (`route.decision`, `guard.*`, ...) as `customEvents`; spans always go as `dependencies`. |
| `OSR_PLATFORM_APPINSIGHTS_LOGS` | `warning` | Lowest log level forwarded as `traces` (logged exceptions become `exceptions`); `off` forwards none. |
| `OSR_PLATFORM_APPINSIGHTS_SQL` | `true` | Every SQL statement a process runs for a request as a `dependencies` row of type `SQL` under that request - the statement text with its placeholders, never the bound values - so a slow table shows under the requests that waited for it. `false` leaves the database out. |
| `OSR_PLATFORM_APPINSIGHTS_SQL_MIN_MS` | `0` | Report only successful statements at least this slow (failed ones are always reported); `0` reports every statement. |
| `OSR_WEB_APPINSIGHTS` | `auto` | The browser's share, served by the web containers: `auto` / `on` export when the container has a connection string (the Azure template gives the two web apps the same one), `off` never. The page asks `/telemetry/config` once and posts straight to the resource's ingestion endpoint. |
| `OSR_WEB_APPINSIGHTS_SAMPLE_RATE` | `1` | Share of browser sessions kept (0..1), decided once per session so a session is whole or absent. |
What arrives, all under the request's operation id so the end-to-end view lines up: `requests` (every HTTP request a
process answered - route template, status, latency, workspace id as the authenticated user, the caller's user agent
and the page it came from as origin and path, never a query string; a refused request carries the reason the
platform gave, the message of a `400` or the fields a `422` named), `dependencies` (every
router span: `route`, `plan`, `execute`, provider calls, with duration and outcome - and every SQL statement the
request ran, type `SQL`, named by verb and table, a failed one carrying the error's type and message), `customEvents` (the routing events with their
attributes - ids, numbers and short labels, never prompt text), `traces` and `exceptions`. Each process reports as
its container app (`cloud_RoleName` = `osr-api`, `osr-routing`, ...). What a replica does to check itself - the
readiness probe routes a sentence through the engine on every check - is not exported. Delivery is batched off the request path with
bounded retries; the console's *Health* page shows the export (`monitoring`: endpoint, sent, dropped, SQL statements
tracked, last error) and nothing about it can fail a request. Nothing is exported from a host that is not Azure
unless you say `on`.
**The browser is in the same picture.** The web app reports to the same resource (`cloud_RoleName` = `osr-web` /
`osr-platform`): one `pageViews` row per navigation named by its route shape (ids and tokens reduced to `*`), the
Core Web Vitals (`web.LCP`, `web.INP`, `web.CLS`, `web.FCP`, `web.TTFB`) as `customMetrics`, uncaught errors and
rejections as `exceptions` - and a page that failed to render as one too, filed by the error boundary that caught
it (`boundary` = `root`, `marketing`, `docs`, `auth`, `dashboard`, `console` or `global`) with the reference the
person was shown on the error page (`digest`, the same one the server log carries, else the page's operation id),
so "quote the reference to support" finds the exception - every API call as a `dependencies` row of type `Ajax`,
a dead address as a `page.not_found` event with the path and the page that linked to it, and the moments of the
user flow as `customEvents` - `sign_up`, `login`, `begin_checkout`, `playground.answered`, `outcome.reported`,
`evaluation.run`, `evaluation.replayed`, the marked calls to action - with the workspace as the user and one
session id per tab. Correlation is by request id: every call carries a browser-minted `X-Request-Id` and a W3C
`traceparent`; the API answers under that id (or its routing decision's id, which the response header carries
back), files its request under it with the browser's span as parent, and its router spans and SQL statements
under the same operation - so the end-to-end view reads page view -> call -> request -> route / execute / SQL,
and a slow page can be followed to the statement that made it slow. The web **server** is in it too: every call a
server-rendered page makes to the API (`cloud_RoleName` the web app, `client_Type` *Server*, the replica as the
instance) is a `dependencies` row of type `Http` under its own request id - memoised per path, so one row per
refresh rather than per visitor - and a call that got no answer or a 5xx is also one line in the container log
(`[upstream] GET /api/v1/services -> TimeoutError ... request_id=...`, at most one per path per half minute), so a
public page that rendered without its data is visible from the web tier whether or not a browser was involved.
Nothing a person typed travels: URLs are
reduced to the allow-listed query keys, the only ids are request, session and workspace. The marketing pages'
Google tag is separate (section 14, consent-gated); the Application Insights copy is the deployment's own
operations telemetry and runs on the product surfaces too.
### Configuration preflight
At start-up every process checks its own configuration for the mistakes an install makes before the first request
- placeholder passwords from `.env.example`, a missing session secret on a multi-replica stack, a public URL still
on `localhost` or plain `http`, e-mail confirmation required with no mail transport, a providers or targets file that
is not in the container, an SLM service without an event bus, Application Insights forced on without a connection
string - and logs each finding with the variable that fixes it. The same list is on the operator console's
*Health* page (`GET /api/v1/admin/health` -> `configuration`, each with `severity`, `code`, `message`, `fix`) and
`python -m osr_platform.app --check` prints it and exits `1` on an error, so a Compose or pipeline step can gate
on it: `docker compose -f platform/docker-compose.yml --env-file platform/.env run --rm api python -m osr_platform.app --check`.
The preflight never touches the network; reachability of PostgreSQL, Redis and the event bus is what `GET /readyz`
reports once the process runs.
**Go-live checklist.** The preflight reads the environment; the *Go live* page of the console
(`GET /api/v1/admin/readiness`) reads the deployment as a buyer meets it. Every promise the public site makes is
an item - the Stripe account can take payments (activation finished, read live), paid plans can be bought (a
recurring price for every sold plan), overage is invoiced (the metered price
on each interval), tax is computed at checkout (Stripe's tax settings, read live), e-mail reaches people (the mail
transport and its last error), inquiries and tickets reach an operator (the recipients), models answer (executable
LLM targets minus providers that cannot be paid), the terms and privacy notice have a published version naming the
text the pages ship (the *Terms and privacy* page shows the shipped version per document and its publish form
arrives prefilled from the checklist), the public URLs are https, operators use two-factor, the status page is
sampled, every audit chain verifies, the storefront's customer prices are not above the vendors' list prices (a
mark-up rule without contract rows shows a buyer a surcharge, not a saving) - each with `status` (`ok`, `action`, `blocked`, `info`), the `promise` in the customer's words, `where` the
site says it, the deployment's `state`, the `fix` and the console page that shows it. `ready` is false while any
item is blocked; the verdict (*Not ready to sell*, *Ready to sell, with caveats*, *Ready to sell*) also heads the
console overview. Read it before selling and after every configuration change; nothing is remembered between calls.
### Learning: the routing SLM and the autopilot
Every deployment learns from `POST /api/v1/feedback`: the bandit, IRT, Bradley-Terry, LinUCB and Markov
strategies update on each outcome (persisted under the data volume in the enterprise edition). A
deployment can also serve a **routing SLM** - the small routing model `osr slm train` / `osr improve`
write to a file - as one more strategy in the ensemble, and run the **autopilot** in-process so the
platform's own feedback keeps retraining it:
| Setting | Default | Effect |
|---|---|---|
| `OSR_PLATFORM_SLM` | none | Path of the model file, or `bundled` for the routing SLM shipped inside the SDK package; it joins the ensemble as the `slm` strategy (weight `OSR_WEIGHTS_SLM`, default 1.0). |
| `OSR_PLATFORM_AUTOPILOT` | `false` | Run `SelfImprover` cycles on a thread: outcomes and remembered prompts become rows, a challenger is trained on a split and promoted only when it beats the champion on the holdout. The promoted model is written to `DATA_DIR/autopilot/slm.json` and outranks the mounted file after a restart. |
| `OSR_PLATFORM_AUTOPILOT_OFFLINE` | `true` | No catalogue refresh, dataset download or web search from the API process; `false` also refreshes `OSR_PLATFORM_AUTOPILOT_SOURCES` (public dataset presets, comma separated) and reads the article feeds in `OSR_SLM_FEEDS` (vendor blogs, release notes; the same variable `osr serve --autopilot` honours), so a model announced in a post is counted (`mentions`) and, once the catalogue prices it, becomes a candidate (`trending`). |
| `OSR_PLATFORM_AUTOPILOT_INTERVAL_S` | `OSR_SLM_AUTOPILOT_INTERVAL_S` (3600) | Schedule; drift alarms from the outcome stream trigger a cycle early (`OSR_SLM_AUTOPILOT_DRIFT_*`, rate-limited by `OSR_SLM_AUTOPILOT_MIN_GAP_S`). |
| `OSR_PLATFORM_AUTOPILOT_MIN_ROWS` | `20` | A cycle needs at least this many rows before it trains a challenger. |
| `OSR_PLATFORM_SLM_WORKSPACE_MIN_ROWS` | `100` | In the SLM service: a workspace with this many feedback rows of its own gets its own challenger and, when it wins, its own promoted model (`slm..json`); `0` keeps one shared model. |
| `OSR_PLATFORM_SLM_WORKSPACE_MAX_MODELS` | `20` | Most per-workspace models a deployment trains and serves (the workspaces with the most evidence); `0` = no cap. |
| `OSR_PLATFORM_LEARNING_FEEDBACK_DAYS` | `30` | Window the per-target outcome statistics of `GET /api/v1/learning` are computed over. |
`GET /api/v1/learning` (any plan) returns the strategy ensemble with its weights, the learner state
(persistence, whether it is `shared` through Redis, drift resets, quarantined state, and `store_degraded` /
`store_errors` when the shared store could not be read or written - the learners keep learning in memory
meanwhile), per-target outcome statistics over every recorded outcome of the last
`OSR_PLATFORM_LEARNING_FEEDBACK_DAYS` days (`n`, `success_rate`, `mean_quality`, `mean_cost_usd`,
`mean_latency_ms` and the pairwise `wins` / `losses` - the whole deployment, not the replica that answered),
the learned
`leaderboards` (enterprise edition: `irt` abilities from graded outcomes and `preference` Bradley-Terry
strengths from `preferred_over` comparisons, best first, each with how many observations back it), the
SLM in service (rows, sources, encoder, calibration, fit history - never the weights) and the autopilot
status with its recent champion-vs-challenger reports. `controls.slm` / `controls.autopilot` appear in
`GET /api/v1/governance` and `GET /api/v1/info`. Operators trigger a cycle with
`POST /api/v1/admin/autopilot/cycle` (`X-Admin-Token`; 404 without an autopilot). Cycles are traced
as an `autopilot.cycle` span with `learn.improve` / `learn.promote` events, so they appear in
`GET /api/v1/events` and on `/platform/dashboard/events`. The dashboard renders all of it at
`/platform/dashboard/learning`.
## 11. Marketplace
The marketplace (`/marketplace`) is where agents, skills, personas, prompts, tools, model profiles and
stack templates are shared. Every listing is a manifest the SDK, CLI and this API understand; installing
records it on your account and returns the manifest and ready-made snippets, and `registry://`
references in a stack file resolve to `GET /api/v1/registry/{slug}/manifest`. Browsing is public;
installing, rating and publishing need an account. See the [marketplace guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MARKETPLACE.md) for
finding, installing, publishing and the review lifecycle; the endpoints are in the
REST API reference (rendered at `/docs/api`) under *Marketplace*.
### Installed listings and discovery in routing
What a workspace installs joins its routing pool at once - on every API replica, because each one checks the
workspace's install stamp before trusting its cached target list - and competes for the plan's persona, skill
and tool slots or, as a primary, for the answer (`GET /api/v1/targets` lists installs with `source:
marketplace`). Beyond installs, every plan **discovers** published skills, personas and prompts that match the
request: the request's domains and salient words are searched in the listings' name, summary, tags and
domains, a listing must work in one of the request's domains, and the best-rated matches (bodies up to 8,000
characters, at most `OSR_PLATFORM_MARKETPLACE_DISCOVERY_LIMIT` of them) are offered to the router as plan-slot
companions that never answer on their own. A prompt is offered as an instructions-only skill (`prompt:`)
when it takes its input through at most one `{{variable}}` - the placeholder stands for the user's message; a
prompt with several variables is a template for a person to fill and stays out. Which listings qualify is a
trust decision:
| Mode | Discovered |
|---|---|
| `verified` (default) | Listings an operator marked verified - the seeded skills, personas and prompts of the deployment are; *Verified* on `/platform/admin/marketplace` or `POST /api/v1/admin/registry/{slug}/flags` adds one. |
| `all` | Every published listing, the harvested catalogue included. |
| `off` | Nothing; the workspace's installs only. |
`OSR_PLATFORM_MARKETPLACE_DISCOVERY` sets the deployment default, the `marketplace_discovery` policy key
overrides it per workspace (section 9). Agents and tools are never discovered - they execute, so they need an
install. Each searched term is cached for ten minutes per replica; the router's strategies and quality floor
decide whether a discovered companion is used, and the decision reports it under `marketplace` with
`installed: false`.
| Setting | Default | Meaning |
|---|---|---|
| `OSR_PLATFORM_MARKETPLACE_DISCOVERY` | `verified` | Deployment default for what a plan may discover: `off`, `verified`, `all`. |
| `OSR_PLATFORM_MARKETPLACE_DISCOVERY_LIMIT` | `12` | Companions offered to the router per request (0 disables discovery). |
### Rankings by task type and kind
Two public boards rank what this deployment observes, aggregated across every workspace and
anonymised - no workspace id, request id or prompt text ever appears in them:
- `GET /api/v1/rankings/targets?kind=&domain=&window_days=30&limit=50`
ranks routing targets by the outcomes workspaces reported for them (`POST /api/v1/feedback`), optionally
within one task domain. The rank key is the Wilson lower bound (95 %) of the reported success rate, so a
target with three perfect outcomes never outranks one with two hundred good ones; each row also carries
the routed requests and their `share` of the domain, the HTTP success rate, mean reported quality, the
p50 latency and the mean cost per request. Targets with fewer than `min_samples` reported outcomes
(`OSR_PLATFORM_RANKINGS_MIN_SAMPLES`, default 20) are listed as `provisional` after the ranked rows.
`GET /api/v1/rankings/targets/domains` lists the task domains routed in the window with their request
counts; domains thinner than the threshold are left out.
- `GET /api/v1/rankings/listings?kind=&sort=installs|rating|newest&limit=50` is the marketplace
board: published listings by installs, by a Bayesian average rating (the mean shrunk towards 3.5 by
the weight of five ratings, so one five-star review does not top the chart) or by publication date;
publishers appear by display name only. `GET /api/v1/rankings/listings/kinds` counts published
listings, installs and reviews per kind.
Both answer with `ETag` and `Cache-Control` like the model board and are served by the `rankings`
domain. The site renders them at `/rankings/targets` and `/rankings/marketplace`, next to `/rankings`.
## 12. MCP server
The platform is also a [Model Context Protocol](https://modelcontextprotocol.io) server, so an IDE
or agent (VS Code, Cursor, Claude, Windsurf) can route, quote and explain through it. `GET /mcp`
describes the server without a key; `POST /mcp` accepts JSON-RPC 2.0 messages with the same
`Authorization` header as the REST API. Tools: `route`, `estimate`, `recommend`, `explain`,
`list_targets`, `feedback`, `ask` (pro plan and above; routes and executes) and `marketplace_search` /
`marketplace_get`. Routed calls are metered like `/api/v1/route`. (`marketplace_get`'s `install` flag is for
the local `osr mcp` server, which owns its catalogue; on the platform, install a listing from the marketplace
page or `POST /api/v1/registry/{slug}/install`.)
Clients that only speak stdio use the CLI as a bridge:
```json
{"mcpServers": {"opensmartroute": {"command": "osr", "args": ["mcp", "--url", "https:///mcp", "--api-key", "osr_..."]}}}
```
The endpoint is a full Streamable HTTP transport and an OAuth 2.1 resource server: `initialize` hands out a
signed `Mcp-Session-Id` (`DELETE /mcp` ends it), `Accept: text/event-stream` on `POST /mcp` streams progress
and log notifications before the result, `GET /mcp` with the same header is the server's notification stream
(`tools/list_changed`, `resources/list_changed`), a browser `Origin` must be the deployment's own, and a
client without a key is pointed at `/.well-known/oauth-protected-resource/mcp` so it can sign the person in
at the organization's identity provider; the token is verified against the issuer's keys and mapped to the
platform user, and a provider whose credential is `obo:` is then called with a token exchanged **on
behalf of** that person (Entra on-behalf-of or RFC 8693). Variables: `OSR_PLATFORM_MCP_OAUTH_ISSUERS`,
`OSR_PLATFORM_MCP_OAUTH_AUDIENCES`, `OSR_PLATFORM_MCP_OBO_CLIENT_ID` / `_CLIENT_SECRET` / `_TOKEN_URL`,
`OSR_PLATFORM_MCP_OAUTH_EMAIL_MATCH`, `OSR_PLATFORM_MCP_STREAM_MAX_S` - all defaulted from the `microsoft`
single sign-on provider.
Per-client configuration, the tool reference and the SDK side are in [MCP.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MCP.md).
### MCP gateway: your own servers' tools in the routing pool
The other direction: a workspace connects the MCP servers it already runs - an internal ticketing server, a
database agent, a vendor's hosted server - and their tools join the workspace's routing pool as `tool` targets
named `:`. A chat completion or a `/route` decision may then pick one the way it picks a model,
under the same policy, trace and metering; a plan may give one a tool slot next to the model that answers.
`POST /api/v1/workspace/mcp-servers` connects a Streamable-HTTP server (`name` - the tool id prefix -, `url`, an
optional bearer `credential` stored encrypted like a provider key and never returned, `extra_headers`, the
`approval` mode): the server is initialised and its tools listed at once and kept, so routing never waits on
`tools/list`; `POST .../{id}/sync` refreshes the list; `GET .../{id}/tools` shows the tools as the calling key
sees them (whether each executes, its `readOnlyHint`, the description risk the SDK's injection guard measured -
tools that read like instructions to the model are dropped); `PATCH` re-points, rotates the credential, changes
the mode, pauses; `DELETE` disconnects. Approval modes: `auto` executes every tool the router picks (agent
mode); `read_only` executes only tools the server marks read-only - the others are routable, so a decision may
name them and a plan may show them, but the caller runs them itself; `manual` lists and ranks only. Per key,
the `mcp_tools` limit (`server:tool` or `server:*`, section 1) narrows what a key may reach. Feature `byok`
(Pro and up), up to 20 servers of up to 200 tools each; connecting takes the `admin` role, every change is
audited (`mcp_server.connected`, `mcp_server.removed`).
## 13. Dashboard
The site has two halves on one hostname. The **website** - landing page, `/docs`, `/models`, `/vendors`,
`/rankings`, `/pricing`, `/marketplace`, `/compare`, `/roi`, `/estimate`, `/extension` - needs no account.
The **platform**
is everything under `/platform`: sign-in and sign-up (`/platform/login`, `/platform/signup`, password
recovery, invitations, `/platform/cli/authorize`), the dashboard (`/platform/dashboard/...`), the operator
console (`/platform/admin/...`), the playground (`/platform/playground`) and the marketplace publish flow
(`/platform/marketplace/publish`). On the hosted service the two are separate deployments; the older
paths without the prefix (for example /dashboard/keys, /login or /auth/callback) redirect permanently to their
`/platform` counterpart with the query string intact, so bookmarks, emailed links and OAuth redirect URIs
registered before the split keep working.
Signed-in users have a dashboard for the data the API exposes:
| Page | What it shows |
|---|---|
| `/platform/dashboard` | A guided tour opens on the first arrival from signup (or with `?tour=1`) and replays from *Take the tour* on the getting-started card: the sidebar, the figures and how each is computed (the *i* on every card), the checklist, the quick start, the header tools and the account menu. Requests, tokens and spend for the last 7 days; the monthly quota bar with a warning banner from 80 % (`GET /api/v1/quota`); *Ask about your usage* - the window's findings in plain words with a question box (`GET /api/v1/insights`, `POST /api/v1/insights/ask`); the last 24 hours as a traffic and latency series (`GET /api/v1/telemetry/series`) and the active alerts (`GET /api/v1/alerts`). On a phone a bottom bar holds Home, Activity, Usage, Keys and the full menu. |
| `/platform/dashboard/usage` | Per-day, per-target, per-endpoint and per-app usage (`GET /api/v1/usage`; the *By app* table appears once a client named itself with `X-OSR-App`); CSV export of the daily series and every breakdown. |
| `/platform/dashboard/savings` | Routed cost against a baseline model (`GET /api/v1/savings?baseline=`). Without `baseline` the report uses the dearest model the workspace's executed requests actually ran on (`baseline_source: executed`) and falls back to the catalogue's dearest only when none of them is priced (`catalogue`); `baseline_name` is the display name. Pick the model the team used before routing and keep it for the pilot. CSV export per day and per target for finance. |
| `/platform/dashboard/impact` | What routing did for the workspace in one report (`GET /api/v1/usage`, `/savings`, `/governance`, `/usage/opportunity`): provider cost saved against the baseline, requests routed and their success rate, personal data protected by the guard (24 h verdict counts) and the governance score with its findings; the same figures per API key (a person in a team workspace - CSV export), per tenant with its data boundary (enterprise) or per client app, and the spend observed outside the router. |
| `/platform/dashboard/apps` | The browser extension and the desktop app for this workspace: install buttons from the published release manifests (`/extension/release.json`, `/desktop/release.json`), whether each client is installed or connected, what each client sent through the router in 30 days (`by_app` of `GET /api/v1/usage`), whether the extension shares observations, and the CLI / SDK sign-in. |
| `/platform/dashboard/spend` | Two tabs (`?tab=observed|bill`). *Observed by the extension*: what the browser extension observed the team spend on chat sites and provider consoles (`GET /api/v1/usage/opportunity`, `GET /api/v1/usage/observed`): observed spend against the router's quote for the same requests, the models with the widest gap and the target the router would have picked, breakdowns by model, site and task, the per-call history (never the text) and a CSV export; an install prompt when nothing has been observed. *Your provider bill*: drop the invoice PDF a model provider already sends, or its usage export, or paste either (`POST /api/v1/usage/imports/upload`, `POST /api/v1/usage/imports`, *Preview only* for `?dry_run=true`), see the bill against the router's quote for the same requests - the saving and its monthly projection, the share of the bill the router could quote, day by day, by provider and by model with the target the router would have picked, the notes on what was assumed - keep it, download it as CSV, Excel or PDF, and the list of every kept analysis (`GET` / `DELETE /api/v1/usage/imports/{id}`). |
| `/platform/dashboard/prompts` | The prompt registry (*Prompt registry* under section 4): every prompt with its versions and change notes, labels moved by a click to deploy a version, a new version or a new prompt (text, or chat with examples) and a preview that renders a version with the variables filled in. |
| `/platform/dashboard/memory` | What the routed chat keeps in mind for you (*Conversations and memory* under section 4): the long-term memory switch, every remembered fact with its kind and how often it shaped an answer - pause, forget or add one; the workspace's knowledge graph (*Knowledge graph and context engineering*) - its switch and context budget, the facts in plain words with search and delete, a fact added by hand, *forget that ...*, a preview of the context placed for a message, the export and the link predictor; and the conversations the platform holds for the desktop app, the browser extension and the playground, with where each was started; delete one or all. |
| `/platform/dashboard/activity` | Request-level activity log: endpoint, target, domain, complexity, tokens, reported outcome - never the prompt text; filters by endpoint, target, API key and tenant (`GET /api/v1/activity?tenant=`). Rows still in the tracer buffer open a span waterfall with the decision story (signals, admission, ranking, why the winner, plan, decision, execution and what crossed the boundary: PII redaction or restoration, prompt compaction, output-guard findings), outcome and audit records; the trace can be downloaded as JSON and shared as a link (`?request_id=` opens the drawer). A request whose prompt was kept offers *Add to dataset* (a row of a kept evaluation dataset, new or existing) and *Judge this answer* (a metered judge call whose grade becomes the outcome). CSV export of the loaded rows. |
| `/platform/dashboard/events` | The router's event history for the workspace (`GET /api/v1/events`, persisted by the platform): a window of 1 hour to 7 days, spans and events with level and stage filters, search by request id / trace id / name, most frequent names, click-through to the request trace, CSV export. The **Live** refresh option follows `GET /api/v1/events/stream` and prepends events as they happen. |
| `/platform/dashboard/evaluation` | Measure the router on your own prompts without executing anything (`POST /api/v1/evaluations`, section 5 *Evaluation*). Four tabs (`?tab=dataset|replay|datasets|judge`): *Labelled dataset* - drop or paste the JSON-lines file `osr eval` reads (or load the catalogue's sample dataset), pick the cost weight, compare with the baselines, sweep the cost weight; *Replay your traffic* - re-route the prompts kept under the request-logging policy through today's router; *Kept datasets* - the datasets saved from logged requests (`GET /api/v1/evaluations/datasets`), each with *Evaluate* on today's pool, *Download JSONL* and *Delete*, and *From your traffic* to save a window of kept prompts (by outcome, target and size); *Answer judge* - the judge in force, what it graded per answering target over the window and the latest verdicts with their reasons (`GET /api/v1/evaluations/judge`). The report shows accuracy with its interval, realised quality or confidence, the price of the mix against always using the dearest admitted target, calibration, the baselines on the same rows, where the rows went, accuracy by domain, the misroutes to read, the reliability bins and every decision (searchable, filterable, CSV); a replay shows what would move, the projected cost, whether accepted answers are kept and rejected ones moved, the then / now mixes and every request with a link to its trace. The newest 50 runs are kept and listed below the form. |
| `/platform/dashboard/learning` | How the router learns (`GET /api/v1/learning`): strategy weights, outcomes per target, the IRT ability and Bradley-Terry preference leaderboards (enterprise edition), the routing SLM (rows, sources, the targets it ranks, accuracy history, calibration) and the autopilot (schedule, drift, cycles, a retraining-in-progress indicator, champion vs challenger; operators can run a cycle). Leaderboards, fits and cycles export as CSV. |
| `/platform/dashboard/keys` | Create, name and revoke API keys; per-key scopes, expiry and limits (daily budget, requests per minute, allowed targets - a bare catalogue id such as `openai/gpt-4.1` means every endpoint of that model - the monthly spend limit that caps what the key is charged, the headers every request must carry (`required_headers`, refused with 428 before anything is spent) and the connected MCP servers' tools the key may reach (`mcp_tools`, `server:tool` or `server:*`)). |
| `/platform/dashboard/mcp-servers` | Your own MCP servers' tools in the routing pool (`/api/v1/workspace/mcp-servers`, every plan with `byok`): connect a Streamable-HTTP server (name, URL, optional bearer token, extra headers, approval mode) and its tools join the pool as `name:tool` targets at once; the list shows each server's health from its last sync, its tool count (click for the tools as the router sees them for your key - target id, whether it executes, read-only, a flagged description), the approval mode (*auto* - every tool runs; *read-only* - only tools the server marks read-only execute; *manual* - listed and ranked, never run) and *Sync*, with enable / disable, a change of approval, clearing the token and removal (confirmed) in the row menu. |
| `/platform/dashboard/providers` | Bring your own model providers (every plan of the default catalogue), in three tabs. *Your providers*: connect Azure OpenAI, Amazon Bedrock, Google Vertex AI / Gemini, OpenAI, Anthropic or any OpenAI-compatible endpoint from a preset picker - the preset's placeholders (resource, region, project) are asked for as fields and the URL is assembled - with the workspace's key, extra headers, timeout and retries, and check the connection on save; each provider card shows health with the time of the check, the masked key, its settings, the models the endpoint reported (click one to map it) and the targets mapped to it; *Edit* changes every stored field and replaces or removes the key; enable, disable or remove (confirmed). *Targets*: every model and tool the router can choose from with where it executes for this workspace (yours / platform / not executable), searchable and filterable, with *Map*, *Change* and *Unmap* per row; the mapping dialog takes the model, an optional system prompt and JSON request defaults. *Included with the platform*: the deployment's own providers with health and the targets each one runs. |
| `/platform/dashboard/tenants` | Tenant constraints and budgets (enterprise), in two tabs: *Tenants* (one card per tenant with its constraints in words, this month's usage, budgets and pinned keys; create and edit in a sectioned dialog, delete after confirmation) and *Usage and setup* (totals and the `X-OSR-Tenant` curl example). |
| `/platform/dashboard/governance` | Five tabs (`?tab=overview|controls|budgets|data|audit`). *Overview*: spend today and this month, quota, reachable targets, the workspace policy in readable rows with *Edit* / *Remove*, compliance posture with linked findings and score. *Controls*: controls in force (including the output guard mode), guard activity of the last 24 hours and, where the deployment runs fair share across tenants (`OSR_PLATFORM_FAIR_SHARE`), the *Fair share* card - the window and the soft / hard multiples, how many requests were steered or capped, and each tenant's requests and dominant share in the current window. *Budgets*: workspace and tenant budgets against their spend. *Data*: the response cache (`GET /api/v1/cache` - live entries, hits served, the window in force; *Clear* for an admin) and the request-log export (`/api/v1/workspace/log-export` - destinations with rows sent, last run and state; add an Azure Blob container or an HTTPS endpoint with its credential, prefix, interval and whether prompts and answers ride along; *Export now*, pause / resume, remove). *Audit and keys*: API-key hygiene, audit status and retention, catalogue data boundaries. The policy dialog groups its fields into where requests may run, cost and latency limits, budgets, privacy and *how requests are answered* - model aliases, the response-cache window, conversation stickiness (`session_affinity`), the default objective and the answer judge - offers the executable targets as chips, previews what the policy will do in sentences, and edits every policy key - the response cache (window, and whether rewordings count), the providers block (try first / only / never, tie-breaker, price ceilings, no data collection, zero retention only), data protection and allowed addresses - so nothing is lost across a save. "Compliance report" download (`GET /api/v1/governance/report?format=markdown`) in the header. |
| `/platform/dashboard/health` | Deployment readiness (`GET /api/v1/status`), the active alerts, HTTP traffic and latency percentiles over time (`GET /api/v1/telemetry/series`), live counters, routing decisions and latency, circuit breakers, decision cache (enterprise). |
| `/platform/dashboard/notifications` | The alert inbox (firing and resolved episodes, mark read), delivery channels (email, webhook, Slack, Teams) with severity and rule filters and *Send test*, and the delivery log. The bell in the header shows the unread count. |
| `/platform/dashboard/audit` | Hash-chained decision trail with chain verification (enterprise); export as CSV or as the JSONL the offline verifier reads. |
| `/platform/dashboard/billing` | Four tabs, one question each (deep-linkable with `?view=overview|plans|credits|invoices`; a return from Stripe opens the right one). **Overview** - a running coupon trial (days left, end date, *Subscribe now*), the subscription (state, trial end, renewal, portal, cancel), the monthly quota card (used, limit, state, reset date, overage so far), what the plan includes and the **Service level** card (`GET /api/v1/sla`: availability and support first-response this month against the plan's commitments). **Plans** - the plan cards with a monthly / annual switch (plans the promotion covers start a free trial without payment; plans sold through sales link to the sales form) and the **Pay as you go** card (switches to the prepaid plan without Stripe), the coupon-code form, and on deployments without Stripe or a promotion the operator command that sets a plan. **Credits** - the **Credits** card (balance, pending charges, last top-up, *Buy credits* through Stripe Checkout with the deployment's credit packs and the purchase fee, auto top-up, the ledger and the monthly statement) and the **Usage and charges** table (`GET /api/v1/billing/usage`: provider cost and the workspace's charge by day, model, provider, key or tenant, CSV). **Invoices** - every invoice Stripe issued, with the hosted page and the PDF, and a plain explanation when the deployment issues none. |
| `/platform/dashboard/referrals` | *Refer a team*: the workspace's referral link and code with copy, e-mail and share buttons, an *Invite by e-mail* form (twenty a day, a line in your words), the figures (signed up, rewarded, credits earned), the invited workspaces by initials with each one's status and reward, how the programme works, and - for a workspace that came through a referral - who invited it and when its welcome credit arrives. | `GET /api/v1/referrals`, `POST /api/v1/referrals/invite` |
| `/platform/dashboard/support` | Support tickets of the workspace: open one (subject, category, priority, optional request id), the confirmation with its reference, the list by status, and the conversation of a ticket - reply (reopens a resolved ticket), close. E-mailed links (`?ticket=OSR-00042`) open the ticket directly. |
| `/platform/dashboard/feedback` | Feedback the workspace sent from the platform's pages (bugs, ideas, improvements, praise) with the status trail of each (received, being reviewed, planned, in progress, shipped or declined), the conversation, the screenshots and the page details that went with it; add to an open item. The *Send feedback* button in the header opens the form from any page with screen capture, paste or upload and the browser diagnostics. E-mailed links (`?item=FB-00012`) open the item directly. |
| `/platform/dashboard/workspace` | The current workspace in tabs: *General* (name, slug, region, plan), *Single sign-on* (owners of an organization: set up or edit the connection in three steps - provider, credentials next to the redirect URI, who gets in - and share the direct sign-in link; admins see it) and *Leave or delete*. "New organization" opens a dialog and shows the organization's first API key once. |
| `/platform/dashboard/members` | Members, roles and pending invitations of an organization in one list - search by name or e-mail, filter by role or by joined / invited. |
| `/platform/dashboard/listings` | Marketplace listings you published and everything you installed. |
| `/platform/dashboard/integrations` | SDK snippets with your key (the TypeScript client from npm, the OpenAI Python client against `/v1`, curl) and MCP configuration for VS Code, Cursor, Windsurf, Claude Code and Claude Desktop. |
| `/platform/dashboard/content` | The content publishing service of the workspace (docs/PUBLISHING.md): write a post from a brief, the posts with *Publish*, the feeds you follow with *Fetch now*, the destinations your posts are delivered to (signed webhook, WordPress, Ghost) with *Send a test*, the editorial profile and the delivery log with *Retry*. |
| `/platform/dashboard/account` | Your sign-in identities, password, browser sessions, e-mail preferences and agreements (which version of the terms and the privacy notice you accepted, when). |
Every table with an *Export CSV* button downloads exactly the rows shown (UTF-8 with BOM, RFC 4180 quoting,
cells that start with `=`, `+`, `-` or `@` are prefixed so spreadsheets do not run them as formulas). A failed
read shows what it means - session expired, feature not on the plan, budget exhausted with the amounts, rate
limit with the retry delay, API unreachable - with a *Retry* button; an expired session returns to sign-in and
back to the page afterwards.
Lists read the same way everywhere. A list the API returns whole - API keys, members and invitations, the audit
trail, hand-off tickets, support tickets, feedback items, events, the learning leaderboards, usage per tenant and
per target, the observed-spend history, the impact report's per-key and per-tenant tables, your marketplace
listings and installs, and in the console operators, coupons, leads, offers, digests, listings, operator runs,
the price book, wallets and provider invoices, growth keywords, campaigns, search queries and posts, a
provider's model mappings, ranking runs and the members, keys, tenants, ledger and audit history of a
workspace or user - gets a search box, one or two filters (role, status, source) and a count above the rows
once it is longer than a handful, and a pager under the rows
(10 / 25 / 50 / 100 per page, "11-20 of 42") once it is longer than a page; short lists show none of it.
Lists the API pages itself (activity, the console's users, workspaces and audit log) keep their *Load more*
or server-side pages. Forms show the essentials and fold the rest behind one **More options** row - a
support ticket asks for the subject and what happened, with category, priority and the request id behind the
row; a coupon asks for the code, the trial length and the plan, with the redemption cap, the deadline, the
description and auto-apply behind it. The closed row states what is set inside, so nobody has to open it
to check.
The header of every dashboard page carries the same tools: a search (Ctrl K) that jumps to any dashboard page by
its name or by what people call it ("token", "upgrade", "invite"), then to models and documentation; the
notification bell; the theme; *Send feedback*; help; the playground; the docs. The month's quota figure links to
plan and billing. A long page introduction folds behind *More about this page* so the page's own content stays
above the fold.
Secondary sections of a long page - the usage heatmap, the health page's HTTP routes and raw snapshot, the stdio
bridge and tool list under Integrations, the account signals under Savings - are foldable cards: closed by default,
the title, a one-line summary and the action stay in view, and the browser remembers what each person opened.
Every dashboard and console page ends with the same footer - edition, plan (or version and operator in the
console), the guides, status, support and legal links - and a *Back to top* control; a floating one appears after
half a screen of scrolling.
### How the Impact report counts
The Impact page is the number a finance lead, a team lead or an auditor takes away, so every figure on it is
this workspace's own metering, never an estimate from a model, and every one can be reproduced from the API.
| Figure | Computed from | Notes |
|---|---|---|
| Provider cost saved | `GET /api/v1/savings?days=N&baseline=` | The provider cost of every *executed* request (`execute=true`, the platform's own key or a BYOK provider) against the list price of the baseline model for the same tokens. Decisions that were not executed carry no provider cost and add nothing, which is why a workspace that only asks for routes sees a flat line. |
| Requests routed and success | `GET /api/v1/usage?days=N` | `requests` and `ok`; success is `ok / requests`. |
| Data protected | `GET /api/v1/governance` -> `guard_activity` | The guard verdicts of the last 24 hours of trace events: redactions, blocks, boundary refusals and injection stops. Older verdicts are in `/platform/dashboard/events`. |
| Governance score | `GET /api/v1/governance` -> `posture` | The posture score and its findings, the same as `/platform/dashboard/governance`. |
| Per API key | `GET /api/v1/usage` -> `by_key`, names from `GET /api/v1/keys` | In a team workspace one key per person makes this the per-person view. A key that was deleted keeps its rows under its id. |
| Per tenant | `GET /api/v1/usage` -> `by_tenant`, boundaries from `GET /api/v1/me` | Enterprise edition; every row shows the tenant's region and data boundary so the report doubles as evidence that the boundary held. |
| Per client app | `GET /api/v1/usage` -> `by_app` | `browser-extension`, `desktop`, or the API, SDK and playground (no app header). |
| Spend outside the router | `GET /api/v1/usage/opportunity` | What the extension observed on provider consoles and chat sites that did not go through the router, and what routing it would have cost. |
Rows are isolated per workspace: the API key (or browser session) that reads the page fixes the account, and
every query above is filtered by that account before anything else. Operators see a workspace's figures only
through the console (`/platform/admin/workspaces/{id}`).
The public pages
`/models` (every routing target with request, token, cost, latency and health statistics; also
`GET /api/v1/models`, and the reference LLM catalogue with vendor prices and context windows at
`GET /api/v1/llms`), `/vendors` (one page per LLM vendor with prices, context windows and benchmarks),
`/rankings` (targets ordered by observed quality per domain, and the model leaderboard: models ranked by
published benchmarks - the Artificial Analysis indexes of the model listing, the Hugging Face Open LLM
Leaderboard, Chatbot Arena and Hub downloads - combined into one index; the platform refreshes those
sources three times a day, `GET /api/v1/rankings/models` returns the board with day-over-day movement
and `GET /api/v1/rankings/models/{vendor}/{model}` one model's rank history), `/roi` (estimated
saving against always calling one model), `/compare` (alternatives side by side) and `/marketplace` need
no account; `/platform/playground` (route a request in the browser - *Get an answer* streams the reply token by
token from `POST /v1/chat/completions` with **Stop** keeping what arrived, *See the decision*,
*Compare models*, *Embeddings, voice and images* - `POST /v1/embeddings`, `POST /v1/moderations`,
`POST /v1/images/generations`, `POST /v1/audio/speech` and `POST /v1/audio/transcriptions` from the browser, each
with the target that answered, the request id and the equivalent `curl` - and copy the equivalent `curl`) asks for the free sign-in first, because decisions are
metered on the workspace. Under every decision and every chat answer the playground asks *Was this the right
choice?* - **Good** or **Poor** is one click and lands as `POST /api/v1/feedback` (quality 0.85 / 0.20), the
first outcome of the checklist; the activity page keeps the full form for an exact score and a better target.
Machine-readable companions: `/llms.txt` (documentation map for AI assistants, llmstxt.org;
every documentation page is also served as Markdown at `/docs/.md` and advertised with a
`text/markdown` alternate link), `/llms-full.txt` (the whole documentation as one Markdown file),
`/feed.xml` (Atom feed of releases), `/sitemap.xml` and `/openapi.json` (the complete OpenAPI 3.1
document of the API - the committed snapshot the REST reference renders, not the subset one gateway
process happens to serve).
### Operator console
The people who run a deployment sign in at `/platform/admin/login` with an operator **username and password**
(`POST /api/v1/admin/auth/login` returns an `osr_op_` session token valid for twelve hours; send it as
`Authorization: Bearer`). In the browser the console does not keep that token where a script can read it:
right after sign-in it moves into an **HttpOnly, SameSite=Strict cookie** scoped to `/api/v1/admin` (the web
tier's `POST /api/admin-session` verifies the token against the API first), and the web proxy injects it as
the Bearer credential only for admin calls that carry the console's own marker header - a cross-site form or
navigation cannot set custom headers, so the cookie never rides them. Sign-out revokes the API session and
clears the cookie. Automation keeps sending the token as a plain `Authorization: Bearer` header, which always
takes precedence. The first operator comes from the environment: set
`OSR_PLATFORM_ADMIN_USERNAME` and `OSR_PLATFORM_ADMIN_PASSWORD` and the API creates it - or resets its
password - at start-up as a `superadmin`. The static `OSR_PLATFORM_ADMIN_TOKEN` (`X-Admin-Token`)
keeps working for automation and counts as a superadmin. Operators are separate from workspace users.
**Two-factor authentication.** Every operator can add an authenticator app (TOTP, RFC 6238: six digits every
30 s, one step of drift accepted) under *Operators › Two-factor authentication*: `POST /api/v1/admin/auth/mfa/setup`
(current password) returns the secret and the `otpauth://` URI, `POST /api/v1/admin/auth/mfa/enable` confirms a
code and returns eight single-use **recovery codes** (shown once, stored hashed). From then on
`POST /api/v1/admin/auth/login` answers `mfa_required` with a five-minute signed `challenge` instead of a token,
and `POST /api/v1/admin/auth/mfa/verify` with the challenge and a code (or a recovery code) issues the session.
`DELETE /api/v1/admin/auth/mfa` (password + code) switches it off; a superadmin resets a locked-out operator with
`PATCH /api/v1/admin/operators/{id}` `{"mfa_reset": true}` (their sessions end). `OSR_PLATFORM_OPERATOR_MFA_REQUIRED`
= `off` (default), `superadmin` or `all` makes enrolment mandatory: an operator without a factor gets a session
that answers only the `/api/v1/admin/auth/*` endpoints (`403`, `X-MFA-Required: enrol` elsewhere) and the console
shows the enrolment card until it is done; the factor cannot be switched off while the policy covers the account.
The static admin token is automation and is never asked for a factor. Audit: `operator.mfa_enabled`,
`operator.mfa_disabled`, `operator.mfa_recovery_used`, `operator.mfa_reset`; failed codes appear on the `admin`
event topic as `operator.login_mfa_failed`.
| Page | What it does | Endpoints |
|---|---|---|
| `/platform/admin` | A guided tour of the console opens after the first sign-in through the form (or with `?tour=1`) and replays from *Take the tour*: the grouped navigation, the eight deployment figures (each with an *i* saying what is counted), search, sign-in configuration, storage and scaling, the operator guide. Deployment counters: users, workspaces, keys, tenants, operators, sessions, requests, errors and provider cost of the last seven days, plan mix, sign-in configuration, and outgoing mail at a glance (transport, health, queued, retrying, gave up, skipped by opt-out). | `GET /api/v1/admin/overview`, `GET /api/v1/admin/plans` |
| `/platform/admin/analytics` | The whole deployment over 7, 30 or 90 days: requests and failures per day with mean latency, cost per day, endpoint share, workspaces per plan, signups per day, the busiest workspaces and the targets that carried the traffic. *Before the signup: the visitor journey* counts distinct anonymous visitors of the public site through visited, looked at the price or the docs, opened the signup form, started it, signed up - with the step conversion, the signup attempts that failed and why (address already registered, terms not accepted, rate limit, rejected, server error), the landing pages and referrers with their signup rate and the calls to action clicked. *What people told us* aggregates the one-question surveys - what holds visitors back (pricing and signup pages) and what is in the way of a workspace that has not routed yet (dashboard) - with the answers in their own words. The *Activation funnel* card follows the workspaces that signed up in the range through the same stages as their getting-started checklist - signed up, e-mail confirmed, first request routed, a target answered, an answer reported as good, came back on another day, on a paid plan - with the share of the cohort, the conversion from the previous stage and the median time from signup to each stage; the funnel per acquisition source (UTM source of the signup); and *Stuck before the first request*, the newest workspaces that signed up more than a day ago and never routed, linked to their console page, each with what it said is in its way (or what the platform can see: unconfirmed e-mail, no key, failed calls, no executable model) and marked when its getting-started reminder went out. *Send reminders now* runs the reminder sweep on demand. | `GET /api/v1/admin/analytics?days=`, `GET /api/v1/admin/analytics/journey?days=`, `GET /api/v1/admin/analytics/activation?days=`, `POST /api/v1/admin/analytics/activation/nudge` |
| `/platform/admin/health` | This replica of the routing process: readiness checks, HTTP traffic and latency series (1 h / 6 h / 24 h), the last five minutes, circuit breakers per target, the deployment alerts in force and the alert evaluator; auto-refresh. *Status page incidents*: what `/status` tells customers - open an incident (title, severity with what the banner will say, the affected components, the first update), post updates and resolve, *Edit* the title, severity or components of an open one in place (the banner and the feed follow), see what the monitor opened on its own. | `GET /api/v1/admin/health?window=`, `GET|POST /api/v1/admin/status/incidents`, `POST /api/v1/admin/status/incidents/{id}/updates`, `PATCH /api/v1/admin/status/incidents/{id}` |
| `/platform/admin/activity` | Every metered request across workspaces, newest first: workspace, endpoint, target, status, latency, tokens, cost, request id; filter by endpoint, failures, workspace, target (click a target) or request id; load older pages; CSV export. *Trace* on a row opens the request's trace as the workspace sees it (decision story, spans, usage, outcomes; read-only, audited as `activity.trace`); `?request_id=` opens it directly - the search page and the workspace page link there. | `GET /api/v1/admin/activity?limit=&before=&account_id=&endpoint=&target=&failed=&request_id=`, `GET /api/v1/admin/trace/{request_id}` |
| `/platform/admin/users` | The user directory: search by email or name, filter by status, email confirmation and sign-in method (password or SSO / key only), sort by joined, last sign-in, email or name, export the matching rows as CSV. Select rows for a bulk action: disable, enable, mark the email verified, sign out everywhere, send a password reset, delete (each user is reported as done or failed). Create one with a personal workspace, plan, optional password and first key (shown once; without a password a reset link is issued so the person chooses one). Per user: workspaces and roles, linked identities, sessions, lifecycle history; rename, disable, reset password or send a reset link, re-send or force the email confirmation, sign out everywhere, *Reset two-factor* for a person who lost the authenticator and the recovery codes (`mfa_reset`; sessions end, they are told by e-mail), delete (workspaces they alone belonged to are disabled and their keys revoked); an *E-mail* card with the person's optional categories and newsletter status - switchable on their behalf after a bounce or a complaint, audited as an operator action - and the last messages addressed to them. | `GET|POST /api/v1/admin/users` (`q`, `disabled`, `verified`, `method`, `sort`, `order`, `format=csv`), `POST /api/v1/admin/users/bulk`, `GET|PATCH|DELETE /api/v1/admin/users/{id}`, `DELETE /api/v1/admin/users/{id}/sessions`, `POST /api/v1/admin/users/{id}/reset-link`, `POST /api/v1/admin/users/{id}/verify-link`, `GET /api/v1/admin/users/{id}/audit`, `GET /api/v1/admin/users/{id}/email`, `PUT /api/v1/admin/users/{id}/email-preferences` |
| `/platform/admin/workspaces` | The workspace directory: search, filter by kind, plan and status, sort by name, plan or created, export CSV; create an organization for an owner email. Per workspace: plan, name, slug, disable, delete; members with roles (add by email, change role, remove - the last owner stays); API keys (mint, revoke); tenants (create or replace a validated configuration, delete); policy (in words), SSO and usage over a window of 7, 30 or 90 days (`?days=` on the usage read); *Connected beyond providers* - the workspace's MCP servers with health and approval mode, its request-log export destinations with rows sent and state, the response cache in force and what the answer judge graded in the last seven days (`integrations` on `GET /api/v1/admin/accounts/{id}`, read-only). *Open dashboard* (superadmins) mints a one-hour browser session on the workspace as its owner and switches to the workspace portal, which shows an *Operator view* banner with the way back; the view sees every plan feature (statistics, audit, tenants) whatever the workspace's plan, while quotas stay the workspace's; the owner sees the session (provider `operator`) in their session list and the audit log records `workspace.open`. A *Getting started* card shows the workspace's own checklist state (the setup steps and first-value milestones of `GET /api/v1/onboarding`, as `onboarding` on the detail), whether it is stuck before its first request and when its getting-started reminder went out; *Send reminder* sends that reminder to the owners now, whatever the workspace's age, once (`409` afterwards). | `GET|POST /api/v1/admin/accounts` (`q`, `kind`, `plan`, `disabled`, `sort`, `order`, `format=csv`), `GET|PATCH|DELETE /api/v1/admin/accounts/{id}`, `.../usage`, `.../members[/{user_id}]`, `.../keys[/{key_id}]`, `.../tenants/{slug}`, `POST /api/v1/admin/accounts/{id}/session`, `POST /api/v1/admin/accounts/{id}/nudge` |
| `/platform/admin/tenants` | Every tenant across workspaces with its configuration. | `GET /api/v1/admin/tenants` |
| `/platform/admin/leads` | Product-qualified leads: workspaces scored on volume, tenants, EU or private data boundaries, agentic traffic, several teams and a company email domain; CSV export. | `GET /api/v1/admin/pql?days=&qualified_only=` |
| `/platform/admin/sales` | Sales inquiries from the *Talk to sales* form: queues (open, new, contacted, qualified, won, lost), search by reference, e-mail or company, the requester's message and workspace, status, assignee and internal notes; won deals are applied on the workspace page. | `GET /api/v1/admin/sales/inquiries?status=&q=`, `GET|PATCH /api/v1/admin/sales/inquiries/{key}` |
| `/platform/admin/support` | The support desk: queues (needs an answer, waiting for the requester, resolved, closed), search by reference, subject, e-mail or request id, desk statistics (unassigned, median first response, oldest open) and the recipients of the notices; per ticket the whole conversation including internal notes, reply to the requester (e-mailed, optionally setting the status), add an internal note, change status with a note for the requester, priority, category, take or unassign. | `GET /api/v1/admin/support/tickets` (`status`, `q`, `account_id`, `assigned_to`, `priority`, `limit`, `offset`), `GET|PATCH /api/v1/admin/support/tickets/{key}`, `POST /api/v1/admin/support/tickets/{key}/messages` |
| `/platform/admin/feedback` | The feedback desk: queues (open, new, reviewing, planned, in progress, shipped, declined), filter by kind, search by reference, title, e-mail, page or request id, the figures (open and unanswered, median first answer, kinds and pages of the last 30 days); per item the conversation including internal notes, the screenshots, the browser details (viewport, last API calls with request ids linked to the request log, console errors), answer the person (e-mailed, optionally setting the status), add an internal note, change status with a note for the person, kind, take or unassign. | `GET /api/v1/admin/product-feedback` (`status`, `kind`, `q`, `account_id`, `assigned_to`, `page`, `limit`, `offset`), `GET|PATCH /api/v1/admin/product-feedback/{key}`, `POST /api/v1/admin/product-feedback/{key}/comments`, `GET /api/v1/admin/product-feedback/{key}/attachments/{attachment_id}` |
| `/platform/admin/plans` | The plan catalogue - monthly quota, overage rate, per-minute limit, keys / tenants / members, annual price, trial, support tier, feature matrix on this edition - and the workspaces on each plan. A superadmin edits a plan in place (quotas, prices, overage, trial, SLA, support tier, features, description): the change is checked, stored as a run-time override that wins over `OSR_PLATFORM_PLAN_OVERRIDES`, in force at once for every workspace on the plan and adopted by every process; *customised* marks such plans, *env* the fields the environment sets, and *Reset* returns a plan to the deployment's definition. | `GET /api/v1/admin/plans`, `PATCH|DELETE /api/v1/admin/plans/{name}`, `GET /api/v1/admin/overview` |
| `/platform/admin/billing` | Billing operations: the Stripe configuration in force (mode, price keys, meter, intervals each plan is sold for - secrets never shown), live subscriptions per plan with an MRR estimate, past-due / incomplete / unpaid / canceled counts, the overage reported to Stripe this month and the last overage sync; every workspace with a billing history (past due first) with this month's requests against the quota, the overage still to report, Stripe dashboard links and *Sync* (re-read the subscription from Stripe after a missed webhook or a change made in the Stripe dashboard - a subscription Stripe no longer has sends the workspace to free); *Report overage now*; the webhook events processed, filtered by type. The workspace page has the same *Sync from Stripe*. | `GET /api/v1/admin/billing`, `GET /api/v1/admin/billing/subscriptions` (`state`, `plan`, `q`, `limit`, `offset`), `GET /api/v1/admin/billing/events` (`kind`, `limit`), `POST /api/v1/admin/billing/accounts/{id}/sync`, `POST /api/v1/admin/billing/overage/sync` |
| `/platform/admin/finance` | The money view (*Usage pricing, credits and provider finance*): usage revenue, provider cost and gross margin over 7 / 30 / 90 days with breakdowns by provider, model, workspace and plan; *Margin watch* - the `margin`, `above_list_price` and `provider_balance` alerts, or the line saying every priced request covered its cost; the pricing rules and the provider price book (add a rule - each kind explained next to its parameters - or a contract price with a reason, end one through a dialog that records the reason, seed the catalogue's list prices); the price-book versions, each with *What it froze* - the prices and rules the version pinned and the raw snapshot, so a disputed charge can be read against the rules in force when it was computed; the credits wallets with *Debit now*; provider prepaid balances and low-balance lines, provider invoices (record, reconcile against expected spend, *Mark paid*). Reads for every operator; writes for the `finance` and `provider_manager` roles. | `GET /api/v1/admin/analytics` (`finance`), `GET /api/v1/admin/finance/alerts`, `GET|POST /api/v1/admin/finance/prices`, `DELETE .../prices/{id}`, `POST .../prices/seed`, `GET|POST .../rules`, `DELETE .../rules/{id}`, `GET .../versions`, `GET .../versions/{version}`, `POST .../quote`, `GET .../wallets`, `POST .../wallets/sync`, `GET|POST .../accounts/{id}/ledger`, `PUT .../accounts/{id}/billing-mode`, `GET .../providers`, `PUT .../providers/{id}`, `GET .../providers/{id}/reconcile`, `GET|POST .../provider-invoices`, `PUT .../provider-invoices/{id}` |
| `/platform/admin/coupons` | Coupon codes that grant a paid plan for a number of days without payment: create (days, plan, cap, deadline, auto-apply), enable or disable, make one the promotion, see who redeemed it, end expired trials now. | `GET|POST /api/v1/admin/coupons`, `GET|PATCH /api/v1/admin/coupons/{code}`, `POST /api/v1/admin/coupons/sweep` |
| `/platform/admin/slm` | The routing SLM service: model in service, prompts and outcomes ingested, improvement cycles (champion vs challenger), run a cycle, probe the model with a prompt. | `GET /api/v1/admin/slm`, `GET /api/v1/admin/slm/reports`, `POST /api/v1/admin/slm/cycle`, `POST /api/v1/admin/slm/predict` |
| `/platform/admin/offers` | Models by the token: every offer (one catalogue model on one provider endpoint) with its upstream id, list prices, data policy and status; filter by model or provider, pause / resume or remove an offer, list a provider's recognised models in one go, add a single offer with its region and data policy. Customer prices are set on the Finance page; the storefront (`/models?view=offers`) and `GET /v1/models` follow every change at once. | `GET|POST /api/v1/admin/offers`, `PATCH|DELETE /api/v1/admin/offers/{vendor}/{model@provider}`, `POST|DELETE /api/v1/admin/providers/{id}/offers` |
| `/platform/admin/providers` | Model providers: add an OpenAI-compatible endpoint from a preset, check it (latency, upstream model list and how many of those models the reference catalogue recognises), **List models** (offer every recognised upstream model by the token under its `vendor/model` id - the storefront and `GET /v1/models` pick them up at once), map catalogue targets to upstream models with an optional system prompt, disable or delete; shows the mounted file, what is attached on this replica and the targets that cannot execute yet. | `GET|POST /api/v1/admin/providers`, `GET|PATCH|DELETE /api/v1/admin/providers/{id}`, `POST /api/v1/admin/providers/{id}/check`, `POST|DELETE /api/v1/admin/providers/{id}/offers`, `GET|POST /api/v1/admin/offers`, `PUT|DELETE /api/v1/admin/providers/models/{target_id}` |
| `/platform/admin/services` | Where each domain runs (in the API or as its own service) with a live health probe of every remote service. | `GET /api/v1/admin/services` |
| `/platform/admin/model-servers` | The self-hosted [model servers](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MODELS.md) behind the on-prem and specialist targets, inspected live: for each server its kind (the OpenSmartRoute model server with engine, device, memory budget and store usage, or Ollama), the models in memory (size, pinned or expiring, slots busy and waiting), the models in its store with the targets they serve, *mapped but not stored* and *stored but cold* warnings; pull a model (followed as a job with progress), load and pin, unload, delete (refused while a target maps to it), the engine log of a loaded model. | `GET /api/v1/admin/model-servers`, `GET /api/v1/admin/model-servers/{id}`, `GET /api/v1/admin/model-servers/{id}/logs?model=`, `POST /api/v1/admin/model-servers/{id}/pull`, `GET /api/v1/admin/model-servers/jobs/{job_id}`, `POST /api/v1/admin/model-servers/{id}/load`, `POST /api/v1/admin/model-servers/{id}/unload`, `DELETE /api/v1/admin/model-servers/{id}/models/{model}` |
| `/platform/admin/growth` | The SEO / SEM engine (section 17): signup funnel by channel, activation and retained-value cohorts (first execution, accepted outcomes, time to first value, week-two / month-two return, denominators shown), keywords (add, pause, delete; coverage, Search Console impressions and position), the writer and IndexNow state, LinkedIn (connect a profile or company page, pick the page to post as, post now, every post with its permalink or LinkedIn's refusal), search campaigns with their router-written ads, budgets and metrics (enable, pause, rewrite the ad, delete), CSV export for Google Ads Editor and Microsoft Advertising, *Push to Google Ads*, *Run a cycle now* or single steps, Search Console queries; the *Lifecycle campaigns* card (trial ending, upgrade offer, release news - due, handled, last run, the coupon the offer quotes; *Run now* for all or one). | `GET /api/v1/admin/growth`, `GET /api/v1/admin/growth/funnel`, `GET /api/v1/admin/growth/cohorts`, `GET /api/v1/admin/growth/runs`, `POST /api/v1/admin/growth/cycle`, `GET|POST /api/v1/admin/growth/keywords`, `PATCH|DELETE /api/v1/admin/growth/keywords/{term}`, `GET /api/v1/admin/growth/queries`, `GET /api/v1/admin/growth/campaigns`, `PATCH|DELETE /api/v1/admin/growth/campaigns/{cid}`, `POST /api/v1/admin/growth/campaigns/{cid}/ads`, `GET /api/v1/admin/growth/campaigns/export`, `POST /api/v1/admin/growth/campaigns/sync`, `GET|POST /api/v1/admin/growth/indexnow`, `GET|POST /api/v1/admin/growth/social`, `POST /api/v1/admin/growth/social/linkedin/connect`, `GET /api/v1/admin/growth/social/linkedin/organizations`, `PATCH|DELETE /api/v1/admin/growth/social/linkedin`, `GET /api/v1/admin/campaigns`, `POST /api/v1/admin/campaigns/run` |
| `/platform/admin/referrals` | The referral programme across workspaces: the terms in force, how many workspaces hold a link, signups through one, rewarded and capped, credits paid; the recent referrals with both workspaces (linked to their console pages), status and what each side received; *Qualify now* runs the qualification pass without waiting for the finance pass. | `GET /api/v1/admin/referrals?limit=`, `POST /api/v1/admin/referrals/sweep` |
| `/platform/admin/search-listings` | Search engines and directories (section 17): IndexNow, Google Search Console and Bing Webmaster submissions with the index verdict per key page, *Submit now*; the listing kit (router-written tagline, descriptions at the directories' lengths, categories, tags, links, logos, facts - each with a copy button, *Rewrite*); the directory register with status, listing address, last check and the visits and signups each listing brought - open the submission page, e-mail the kit, mark submitted, paste the listing's address, check, skip, add your own directory. | `GET /api/v1/admin/growth/search-engines`, `POST /api/v1/admin/growth/search-engines/submit`, `GET|POST /api/v1/admin/growth/directories`, `PATCH|DELETE /api/v1/admin/growth/directories/{id}`, `POST /api/v1/admin/growth/directories/check`, `POST /api/v1/admin/growth/directories/{id}/check`, `POST /api/v1/admin/growth/directories/{id}/submit`, `GET|POST /api/v1/admin/growth/kit`, `GET /api/v1/admin/growth/kit/email` |
| `/platform/admin/social` | Syndication (section 19): the channels every published article is shared on - LinkedIn, X, Facebook, Threads, Discord, Telegram, Mastodon, Bluesky, Reddit, Dev.to, Hashnode, Medium, Blogger, Tumblr, WordPress, Ghost, webhooks - each with its credentials (sealed, never shown again), posting schedule (at once or a window of weekdays and hours in a time zone, a minimum gap, a daily cap), message template, hashtag count, draft mode for blog platforms and re-share cadence; *Send a test*, enable / disable, edit, remove; share any post on chosen channels now or at a chosen time with your own text; the share log with each network's answer and permalink, *Retry* and *Cancel*; *Send what is due*. | `GET /api/v1/admin/social`, `GET /api/v1/admin/social/networks`, `POST /api/v1/admin/social/preview`, `GET|POST /api/v1/admin/social/channels`, `PATCH|DELETE /api/v1/admin/social/channels/{channel_id}`, `POST /api/v1/admin/social/channels/{channel_id}/test`, `GET /api/v1/admin/social/shares`, `POST /api/v1/admin/social/share`, `POST /api/v1/admin/social/shares/{share_id}/retry`, `DELETE /api/v1/admin/social/shares/{share_id}`, `POST /api/v1/admin/social/run` |
| `/platform/admin/marketplace` | Marketplace moderation: the review queue and every other status, approve, reject or unpublish with a note, featured and verified flags, the imported catalogue per source and the scheduled refresh (run it now). | `GET /api/v1/admin/registry?status=`, `POST /api/v1/admin/registry/{slug}/approve`, `POST /api/v1/admin/registry/{slug}/reject`, `POST /api/v1/admin/registry/{slug}/unpublish`, `POST /api/v1/admin/registry/{slug}/flags`, `GET /api/v1/admin/registry/import`, `POST /api/v1/admin/registry/import/refresh` |
| `/platform/admin/notifications` | Deployment alerts as an inbox (readiness, error rate, latency, breakers, drift, autopilot, SLM, tracing), operator channels (email, webhook, Slack, Teams), the delivery log and *Evaluate now*. | `GET /api/v1/admin/notifications`, `POST /api/v1/admin/notifications/read`, `POST /api/v1/admin/notifications/evaluate`, `GET|POST /api/v1/admin/notifications/channels`, `PATCH|DELETE /api/v1/admin/notifications/channels/{id}`, `POST /api/v1/admin/notifications/channels/{id}/test`, `GET /api/v1/admin/notifications/deliveries` |
| `/platform/admin/reports` | The report scheduler: state, deployment recipients, daily / weekly / monthly workspace counts, every digest sent (kind, window, recipients); *Send a digest now* for the deployment or one workspace to the configured recipients or an address; *Run the scheduler now*. | `GET /api/v1/admin/reports`, `POST /api/v1/admin/reports/send`, `POST /api/v1/admin/reports/tick` |
| `/platform/admin/automation` | The job board (section 18): every schedule the deployment runs for itself with its cadence, owning service and the state its component reports, the writers' crew (sources, house skills, marketplace picks, prompt size cap), the operator-triggered runs with their outcome and *Run now* per job (bypasses the lease and cadence, audited as `autonomy.run`). | `GET /api/v1/admin/autonomy`, `POST /api/v1/admin/autonomy/{job}/run` |
| `/platform/admin/newsletter` | Subscribers (filter, active only, add by hand, remove, CSV export) and campaigns: draft in plain text, live preview of the branded email, send a test to one address, send once to every active subscriber; sent campaigns stay as records. | `GET /api/v1/admin/newsletter`, `GET /api/v1/admin/newsletter/subscribers.csv`, `POST /api/v1/admin/newsletter/subscribers`, `DELETE /api/v1/admin/newsletter/subscribers/{id}`, `POST /api/v1/admin/newsletter/campaigns`, `GET|PATCH|DELETE /api/v1/admin/newsletter/campaigns/{id}`, `POST .../campaigns/{id}/test`, `POST .../campaigns/{id}/send` |
| `/platform/admin/operators` | Operator accounts (superadmins create, disable, promote and delete them; the last enabled superadmin cannot be removed) with each one's two-factor state and *Reset 2FA*, your own two-factor authentication (set up an authenticator, recovery codes, turn off), your own password and sessions. Superadmins open any operator's console sessions and revoke one or all of them - a compromised console account is locked out at once. | `GET|POST /api/v1/admin/operators`, `PATCH|DELETE /api/v1/admin/operators/{id}`, `GET|DELETE /api/v1/admin/operators/{id}/sessions`, `POST /api/v1/admin/auth/password`, `POST /api/v1/admin/auth/mfa/setup`, `POST /api/v1/admin/auth/mfa/enable`, `DELETE /api/v1/admin/auth/mfa`, `POST /api/v1/admin/auth/mfa/verify`, `GET /api/v1/admin/auth/sessions`, `POST /api/v1/admin/auth/logout` |
| `/platform/admin/audit` | Deployment-wide account-lifecycle log: signups, sign-ins and failed attempts, password and email changes, invitations, roles, removals, deletions and every operator action (actor `operator`). Filter by action (the menu lists every action with its count), actor type, time window, country, user, workspace or free text over actor, IP, place and details; each row shows where it came from; page back with *Load older*; export the matching events as CSV. | `GET /api/v1/admin/audit-log` (`action`, `actor_type`, `actor_id`, `user_id`, `account_id`, `q`, `country`, `after`, `before`, `limit`, `format=csv`) |
| `/platform/admin/mail` | Every email the platform composed with its transport (Microsoft 365, ACS, SMTP or outbox - a malformed setting is called out with the reason), delivery state and counters (composed, awaiting delivery, retrying, failed, skipped because the recipient opted out); each row carries its category (security, account, usage reports, alerts, support, product news); filter by recipient, kind, category or delivery; open a message to see the rendered HTML and plain text, its `List-Unsubscribe` headers, copy its link, resend it (with a transport) or delete it; the *Categories and opt-outs* table counts stored mail and users who switched each optional category off; send a test message to prove the transport. Without a transport this is the delivery channel. | `GET /api/v1/admin/mail` (`to`, `kind`, `unsent`, `failed`, `limit`, `offset`), `GET|DELETE /api/v1/admin/mail/{id}`, `POST /api/v1/admin/mail/{id}/resend`, `POST /api/v1/admin/mail/test` |
| `/platform/admin/settings` | The effective configuration grouped (sign-in, storage, services, learning, marketplace, mail, billing, observability) with the variable that sets each value; secrets show only whether they are set. The manageable settings (below) are edited in place by a superadmin - a switch for booleans, a bounded number, a choice, a list or text - with *Reset* back to the deployment's value, who changed it and when, a *restart* mark on the ones a background job read at start-up, and *Changed only* / *Editable only* filters. Retention: request rows, persisted telemetry, audit log and outbox sizes, and *Run the sweep now*. Test data: the users and workspaces signed up with a reserved testing domain (`example.com`, `localhost`, `*.test`) that a smoke, load or e2e run left behind, whether the deployment accepts such signups, and *Delete all test data* (superadmin). | `GET /api/v1/admin/settings`, `PUT|DELETE /api/v1/admin/settings/{name}`, `GET /api/v1/admin/retention`, `POST /api/v1/admin/retention/purge`, `GET|DELETE /api/v1/admin/test-data` |
| `/platform/admin/legal` | Terms of service and privacy notice (section 14): the version in effect and an announced one per document, the history with effective dates and summaries, how many users accepted; publish a new version (document, version, what changed, effective date with a warning under the 30-day notice, notify every user) and resend a notice. | `GET /api/v1/admin/legal`, `POST /api/v1/admin/legal/versions`, `POST /api/v1/admin/legal/versions/{version_id}/notify`, `GET /api/v1/admin/users/{user_id}/legal`, `POST /api/v1/admin/users/{user_id}/legal/accept` |
| `/platform/admin/photos` | The photographs of the public site: every picture the pages show with its caption, crop focus, photographer and licence, and the pages it appears on. Replace a picture (JPEG, PNG, WebP, GIF, BMP or TIFF up to 24 MB, at least 1200 px on the long side; stored at 2400 px as WebP; the caption, photographer, source page and licence are all optional, and a credit shows under the picture only when a photographer is named), move the focus with two sliders over a live preview, correct the caption or the credit, or put the shipped picture back; the pages show the change on their next load (the browser and the web server each keep the list of changes for ten seconds), and each card links the pages the picture is on and the audit log keeps who changed what. Slots marked *Placeholder* (the page-not-found and error pages) ship a brand gradient with no credit and are listed first so a deployment can put its own photograph there. The public manifest and the replaced pictures are read by every visitor without a key. Where the pictures are kept: with an object-storage container configured (`OSR_PLATFORM_SITE_PHOTOS_URL`, the Azure template provisions one) an upload goes to the container under a content-addressed name with a year-long immutable cache header, the row keeps its address and the pages embed that address, so no picture request reaches the API or the database and the site's image optimiser caches each size for a year; without a container the picture stays in the database and the API serves it. The page says which applies, and when pictures uploaded before the container was configured still sit in the database, *Move them to storage* moves them (idempotent, audited). The public picture address redirects to the storage URL only when that URL is inside the configured container; a row that names any other host is served as the shipped picture instead. The form checks the source page and the file size before the upload starts and says what to fix. The review of this path, and what it changed, is docs/reviews/SITE_PHOTOS.md (`docs/reviews/SITE_PHOTOS.md`). | `GET /api/v1/admin/site/photos`, `PUT /api/v1/admin/site/photos/{key}` (multipart: `file`, `alt`, `focus`, `photographer`, `page`, `license`, `clear`), `DELETE /api/v1/admin/site/photos/{key}`, `POST /api/v1/admin/site/photos/offload`; public `GET /api/v1/site/photos`, `GET /api/v1/site/photos/{key}` |
| `/platform/admin/search` | Support search from the header box: users by email or name, workspaces by name, slug or email, API keys by prefix, operators, tenants by slug and request ids - each hit links to its page. | `GET /api/v1/admin/lookup?q=` |
Every `/api/v1/admin/*` call is written to the access log with `operator:` (or
`operator:admin-token`) as the actor. The console is served by the web app but every action is an API
call, so the same administration works from scripts with the static token. Operators keep both portals
open at once: the console's account menu leads to the workspace dashboard (a superadmin opens any
workspace from `/platform/admin/workspaces`) and the dashboard's account menu leads back to the console
while an operator session exists.
### Runtime configuration
Every setting comes from the process environment (`OSR_PLATFORM_*`; the deployment guides in
[QUICKSTART_PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_PLATFORM.md) and `platform/README.md` list them).
The tunable subset - sign-in and signup policy, the operator two-factor policy, plan and pricing defaults,
spending bounds and reservations, digest and support recipients, newsletter and blog publishing, the growth
engine (posts per day, auto-publish, writer targets, keywords, IndexNow, the paid-search budget, LinkedIn
sharing), marketplace review and the harvest cadence, rankings refresh, the router's decision cache, the
`/v1/responses` conversation store, the routing model's reload and per-workspace training thresholds, provider
reload and keep-warm, the gateway timeout, the finance, overage and trial sweeps, the launch coupon, retention
windows, the trusted proxies, the output guard and the Application Insights export - can also be managed from the
console without a redeploy. `GET /api/v1/admin/settings` marks those entries `editable` with their
`kind` (`bool`, `int`, `float`, `str`, `list`, `choice`), bounds and choices, whether a change is `live`,
the `source` of the effective value (`default`, `env` or `override`), the environment's own value and
who changed it when. `PUT /api/v1/admin/settings/{name}` with `{"value": ...}` (superadmin) checks the
value against the setting's kind and bounds (400 with the reason; 404 for a setting that is not manageable
at run time), stores it in `settings_overrides`, applies it to the process that took the call at once and
writes `settings.updated` to the audit log; `DELETE` removes the override (`settings.reset`) so the
environment's value is in force again. The overrides travel as one document: `GET /api/v1/admin/settings/export`
(superadmin) returns `{"overrides": {name: value}}` - keep it in version control, and apply it to another
deployment with `PUT /api/v1/admin/settings` `{"overrides": {...}, "replace": false}`; every value is checked
first and one refused value stops the whole import with the list of problems (400), otherwise the process
reconfigures once and the reply names what `changed`, what was `unchanged`, what was `removed` (`replace: true`
drops the overrides the document does not name) and which changes wait for a `restart`; one `settings.imported`
audit entry carries the same lists. The console's *Overrides as code* card does both. An override wins over the
environment, survives restarts and
reaches every other process - API replicas and the domain services - on their next reload
(`OSR_PLATFORM_PROVIDERS_RELOAD_S`, 30 s by default, the cadence console-managed state is re-read at).
Settings that are read per request (`live`) take effect immediately; the ones a background job captured
when it started (the retention sweep, the alert and digest schedulers, the feed poller, `/metrics`
access) are stored and marked *restart* - they take full effect after the next deploy or restart.
Secrets, storage URLs, Kafka, the service topology and every other value stay in the deployment and
are read-only in the console.
The plan catalogue is managed the same way: `PATCH /api/v1/admin/plans/{name}` (superadmin) changes a
plan's `requests_per_month`, `requests_per_minute`, `max_keys`, `max_tenants`, `max_members`,
`retention_days`, `activity_days`, `activity_events_per_day`, `trial_days`, `trial_needs_card`, `price_usd_month`,
`price_usd_year`, `overage_usd_per_100k`, `sla_uptime_pct`, `contact_sales`, `support`, `features` or
`description` (400 names an invalid field or value; 404 for an unknown plan), stores the change as the
`plan_overrides` override (merged per plan onto `OSR_PLATFORM_PLAN_OVERRIDES`), rebuilds the catalogue every
process shares so the workspaces on the plan get the new quota on their next request, and writes
`plans.updated` to the audit log; `DELETE /api/v1/admin/plans/{name}` (`plans.reset`) removes a plan's
run-time changes. `GET /api/v1/admin/plans` lists each plan with its `overrides` and `env_overrides`.
## 14. Privacy and data handling
* Prompt text is routed in memory and, when execution is requested, forwarded to the configured
provider. The platform stores request metadata (endpoint, target, domain, complexity, token and cost
counts), not the prompt or the answer.
* `profile` values are hashed before they reach the learners.
* PII detection runs on every request; `contains_pii` together with a tenant `data_boundary` keeps
sensitive requests on private or on-prem targets. Details: [security model](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SECURITY.md).
* On a deployment that exports to Application Insights the web pages report their own operation facts to the
operator's resource (section 10 *Monitoring*): page views by route shape, Core Web Vitals, uncaught errors, API
calls by method, path and status, and the named moments of the flow - with the workspace id as the user and a
per-tab session id, never a prompt, an answer, a key or a free-text field. It is first-party operations data of
the deployment, held in the operator's own Azure subscription, not an advertising or marketing service.
* The public web pages load Google Analytics 4 in consent mode: no analytics or advertising cookies
until the visitor accepts the banner, IP addresses truncated, nothing on the dashboard. The choice is
stored in `localStorage` (`osr-consent`) and can be changed on the privacy page. Besides page views the
tag records Core Web Vitals, clicks on outbound links and calls to action, code-snippet copies and the
sign-up / sign-in events. Operators of a
self-hosted web image leave `NEXT_PUBLIC_GA_MEASUREMENT_ID` empty to ship no tag at all.
* The site also keeps its own, first-party **visitor journey** (`POST /api/v1/journey`): page views on the
public pages, calls to action and the signup form's progress for a random id that lives in the browser tab's
`sessionStorage` (`osr-journey`, gone when the tab closes) and is stored hashed with the deployment secret. No
cookie, no IP address, no e-mail; a path keeps none of its query string but a few harmless keys (`plan`,
`tab`), a referrer only its origin. A signup joins the visitor to the new workspace so the console can count
how many visitors sign up and from which pages; the rows are deleted after 400 days and a workspace's rows go
with the workspace. The one-question surveys (`POST /api/v1/journey/answer`: *what is holding you back?* for a
visitor, *what is in your way?* for a signed-in workspace) store the chosen reason and the person's optional
line; a workspace's answer is replaced by its next one and audited as `journey.answered`.
### Where a request's data goes
The same picture a security reviewer asks for first: what the hosted platform holds, what passes through it, and
what leaves it. Solid arrows carry request text; dashed arrows carry metadata only (ids, counts, prices, places).
```mermaid
flowchart LR
subgraph you["Your side"]
B["Browser extension / desktop app///`, one page per
published pair - the strongest models against their nearest rivals from other vendors and their closest sibling,
with verdicts, the specification side by side, what three monthly workloads cost on each and the tiered rule that
uses both) and *best models for a task* (`/models/best-for/`: coding, agents, reasoning, long documents,
high-volume chat, vision, structured output, self-hosting, summarisation, customer support, translation, free -
each ranking the models that meet the task's requirements by the index that measures it, with the best value and
the cheapest, and the rule that sends the hard requests to the strongest and the rest to the best value). The
glossary covers the terms people search for around the product - routers, gateways, semantic routing, prompt
injection, PII redaction, observability, context windows, structured output, reasoning models, Agent Skills, A2A,
the routing trace and the learning loop - each with a definition an assistant can quote, FAQ structured data and
links to the pages that do the thing.
**Indexing.** A published URL is pushed to the IndexNow engines (Bing, Yandex, Seznam, Naver and the others that
share the protocol) the moment it is public - posts published within a minute go as one batch - and, with Google
Search Console or Bing Webmaster Tools connected, the sitemap is submitted to them again right after (at most once
an hour), so Google re-reads it for the new address instead of waiting for its next crawl. The site also serves a
Google News sitemap of the posts published in the last two days at `/blog/news-sitemap.xml` (title, publication
date, keywords and the lead image of each), named in `robots.txt` and submitted next to the sitemap index. Each sweep also submits, once, every address the site's own
sitemap lists apart from the marketplace - the pages, the documentation, the comparisons, the model and vendor
pages - so a new page is known to those engines without waiting to be crawled; the key is derived from the
deployment secret and served by the
site at `/indexnow-key.txt` (`GET /api/v1/growth/indexnow-key`), so no configuration is needed
(`OSR_PLATFORM_GROWTH_INDEXNOW=false` turns it off). Google has no equivalent push for ordinary pages; it reads
the sitemap: `/sitemap.xml` is an index whose first file lists the site's pages, documentation, model and vendor
pages, followed by one file per thousand blog posts (the archive included) and one per five thousand marketplace
listings, each with the real modification date where one exists. Every submission and its status is listed in
the console (`GET /api/v1/admin/growth/indexnow`; `POST` re-submits given URLs or every public post).
**Search engines, beyond IndexNow.** The console page `/platform/admin/search-listings` (*Search and directories*)
shows what every engine received and what Google says is indexed. With Google Search Console connected
(`OSR_PLATFORM_GSC_SITE` + `OSR_PLATFORM_GSC_CREDENTIALS`) the sitemap index is **submitted** to the property
once a day (`sitemaps.submit`; the service account needs owner access for that - a refusal is recorded with its
reason and retried on the next cycle) and the **URL Inspection API** is asked about the pages that matter - the
home page, pricing, documentation, models, marketplace, downloads, the blog and comparisons, then the newest
posts - twenty a cycle, each read again after a week, so the console shows a verdict (`PASS`, `NEUTRAL`,
`FAIL`), the coverage state and the last crawl per address instead of assuming a submitted page was found. With
a **Bing Webmaster Tools** API key (`OSR_PLATFORM_BING_WEBMASTER_API_KEY`, *Settings > API access* in Webmaster
Tools) the sitemap goes to Bing's `SubmitFeed` daily as well, and every address IndexNow accepted is also sent to
Bing's `SubmitUrlBatch` within the site's daily quota (read first; what does not fit waits for the next cycle).
Three steps of the cycle - `search_engines`, `index`, `directories` - run this; `POST
/api/v1/admin/growth/search-engines/submit` runs them now.
**Directories.** The same page keeps the register of the directories a product like this is listed in - Product
Hunt, AlternativeTo, G2, Capterra, SaaSHub, SourceForge, OpenAlternative, LibHunt, the awesome-MCP list, the MCP
Registry, Glama, Smithery, There's An AI For That, Futurepedia, Toolify, TopAI.tools, BetaList, Indie Hackers,
Crunchbase, Wikidata - and of the stores and registries the packages are published on (PyPI, npm, the Visual
Studio Marketplace, the Chrome Web Store, Edge Add-ons, Firefox Add-ons, Artifact Hub, the Azure Marketplace);
operators add their own rows and skip the ones that do not apply. The **listing kit** is written once through the
router (`growth-directories`; the documentation's own wording until a writing model is executable): a tagline
under 60 characters, descriptions at 160, 300 and 1,000, categories and tags, next to the links, the logos and
the facts every form asks for, each with a copy button (`GET /api/v1/admin/growth/kit`; `POST` rewrites the
texts; `/kit/email` is the plain-text version). Nothing fills in a third party's form on your behalf: a row
links to the directory's submission page, and where a directory accepts listings by e-mail the kit is sent from
the platform's mailbox and the row marked submitted (`POST /api/v1/admin/growth/directories/{id}/submit`). From
then on the platform **verifies** the listing itself: paste the listing's address (or leave the directory's search
page the register knows) and every cycle reads it for the site's address - daily while pending, weekly once live -
marking the row *live*, or *missing* when a listing disappears; a package registry is live when its address
answers. Next to every row stand the visits and signups whose referrer is that directory (the visitor journey and
the signup attribution), so an operator sees which listings pay. Everything is audited
(`growth.directory_submitted`, `growth.directory_updated`, `growth.search_engines_submitted`, `growth.kit_written`).
**Social.** The engine keeps the company's LinkedIn presence going without anyone writing posts. Connect the
deployment's LinkedIn app once from the console (*Connect LinkedIn* on `/platform/admin/growth`;
`OSR_PLATFORM_LINKEDIN_CLIENT_ID` and `OSR_PLATFORM_LINKEDIN_CLIENT_SECRET` are the app's credentials, the
redirect URL to register on the app is `/api/v1/growth/linkedin/callback`) - as your own profile
(*Share on LinkedIn* product, scope `w_member_social`) or as a company page (*Community Management API*,
`w_organization_social` + `r_organization_admin`; after consent the console lists the pages the member
administers and you pick the one to post as, or name its id up front). The OAuth tokens are sealed at rest
with the deployment secret, renewed with the refresh token when LinkedIn issues one, and never shown to the
console; a fixed token works too (`OSR_PLATFORM_LINKEDIN_ACCESS_TOKEN` + `OSR_PLATFORM_LINKEDIN_AUTHOR`, a
`urn:li:organization:` or `urn:li:person:`). Then, every `OSR_PLATFORM_LINKEDIN_INTERVAL_S` (4 h;
0 = only from the console) the engine publishes one post: the newest public blog post not shared in the last
30 days first, then the documentation pages in rotation. The commentary is written *through the router* (the
same writer targets and skill as the guides; a plain title + summary + link when no model is executable),
escaped for LinkedIn's *little text* format, capped at 3,000 characters with up to five hashtags, and the
URL travels as an article card (title, description). A blog post is also shared the moment it is published -
by the engine, the content curator or an operator - unless something was shared in the last hour
(`OSR_PLATFORM_LINKEDIN_SHARE_ON_PUBLISH=false` turns this off; the cadence then picks it up first). Every
post, success or LinkedIn's refusal, is recorded with its permalink (`GET /api/v1/admin/growth/social`) and
audited (`growth.social_posted`, `growth.social_failed`, `growth.social_connected`, ...); one replica posts
at a time (Redis lease) and a refusal never resets the cadence. `POST /api/v1/admin/growth/social` posts now:
the engine's pick, a given blog `slug`, or a given `text`. `OSR_PLATFORM_LINKEDIN_API_VERSION` is the
`LinkedIn-Version` header (YYYYMM). The same app signs people in (`Continue with LinkedIn`, section 1) unless
`OSR_PLATFORM_LINKEDIN_SSO=false`. When the syndication service (section 19) has an enabled LinkedIn channel that
shares automatically, the engine's own LinkedIn sharing stands down (`delegated: true` on
`GET /api/v1/admin/growth/social`) so a post is never shared twice; a LinkedIn channel without a token of its
own posts with the connection made here.
**Advertising.** Commercial, comparison and navigational keywords with a landing page - and informational ones once a
guide covers them - are grouped by landing page into one search campaign per network (`OSR_PLATFORM_ADS_NETWORKS`, `google,microsoft`)
with phrase-match keywords and a responsive search ad whose ten headlines (<= 30 characters) and four
descriptions (<= 90) the router writes from the documentation. A Thompson-sampling allocator spreads
`OSR_PLATFORM_ADS_DAILY_BUDGET_USD` across the *enabled* campaigns in proportion to how often each one has the
highest signups-per-click rate (Beta posterior over the network's clicks and conversions plus the signups
attributed to the campaign's `utm_campaign`), with a 5 % floor so every campaign keeps being explored. Campaigns
are always exportable as Google Ads Editor and Microsoft Advertising bulk CSV
(`GET /api/v1/admin/growth/campaigns/export?network=`). With the Google Ads REST credentials
(`OSR_PLATFORM_GOOGLE_ADS_DEVELOPER_TOKEN`, `_CLIENT_ID`, `_CLIENT_SECRET`, `_REFRESH_TOKEN`, `_CUSTOMER_ID`,
optional `_LOGIN_CUSTOMER_ID` for a manager account) and a budget above zero, the engine creates and updates the
Google campaigns itself (`POST /api/v1/admin/growth/campaigns/sync` does it now) and pulls their impressions,
clicks, cost and conversions back. Safety: a new campaign is `draft` until an operator enables it, nothing is
pushed while the daily budget is 0, and campaigns are created **paused** in Google Ads unless
`OSR_PLATFORM_GOOGLE_ADS_ENABLE_CAMPAIGNS=true` - three switches an operator flips on purpose. The loop is
closed in both directions. Every campaign carries a tracking suffix (`utm_source`, `utm_medium=cpc`,
`utm_campaign=`, `utm_term={keyword}`) - set on the Google campaign and written into the bulk CSV
addresses - so a click lands with the campaign named and the signup's first-touch attribution counts towards it;
the allocator's reward is those signups plus the network's own conversions. With a conversion action
(`OSR_PLATFORM_GOOGLE_ADS_CONVERSION_ACTION`, its resource name or id) every signup that arrived through a Google
click id is uploaded back to Google Ads as an offline conversion, dated, once (the `conversions` step; a refused
batch stays pending for the next pass), so *maximize conversions* bids on real signups instead of a landing-page
tag the platform cannot place. And a sync pushes more than status and budget: the ad group's keywords are
reconciled with the campaign's list (added and removed to match), and copy rewritten in the console after the
last push replaces the ad upstream (new ad created, the old one removed).
The cycle - Search Console, keywords, guides, snippet repairs, IndexNow, the search-engine sitemaps and URL
submissions, the index inspection, the directory checks, LinkedIn, campaign planning, metrics,
budget, sync - runs every `OSR_PLATFORM_GROWTH_INTERVAL_S` (6 h; 0 = only from the console) in the process that
owns the documentation, one leader per Redis lease; every step is isolated so one failing integration never
stops the others, and every run is recorded (`GET /api/v1/admin/growth/runs`). `POST /api/v1/admin/growth/cycle`
runs it now, optionally only some `steps`. Everything the engine does is in the audit log
(`growth.guide_written`, `growth.keyword_added`, `growth.campaign_updated`, `growth.campaigns_synced`, ...).
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/v1/growth/indexnow-key` | The IndexNow key as `text/plain` (the site serves it at `/indexnow-key.txt`). Public. |
| `GET` | `/api/v1/growth/linkedin/callback` | Where LinkedIn returns the operator's browser after consent (the app's redirect URL); stores the connection and goes back to the console. Public. |
| `GET` | `/api/v1/admin/growth` | Engine status (writer, IndexNow, Search Console, advertising mode, LinkedIn, keyword counts, snippet audit, last run) and the 30-day funnel. |
| `GET` | `/api/v1/admin/growth/funnel?days=` | Signups per channel with verified / active / paid counts. |
| `GET` | `/api/v1/admin/growth/cohorts?weeks=` | Signup-week cohorts: first execution and first accepted outcome (time to each, non-completers), activation within seven days, week-two and month-two retained value with explicit denominators, paying as a plan state. |
| `GET` | `/api/v1/admin/growth/runs` | The recorded cycles, newest first. |
| `POST` | `/api/v1/admin/growth/cycle` | Run a cycle now; `{steps: [...]}` limits it. |
| `GET`, `POST` | `/api/v1/admin/growth/keywords` | List (with counts, `?status=`) and add (`term`, `intent`, `landing`, `priority`). |
| `PATCH`, `DELETE` | `/api/v1/admin/growth/keywords/{term}` | Change intent, status, landing or priority; remove. |
| `GET` | `/api/v1/admin/growth/queries?days=` | Search Console queries aggregated over the window. |
| `GET` | `/api/v1/admin/growth/campaigns` | Campaigns with their ad, budget and metrics. |
| `PATCH`, `DELETE` | `/api/v1/admin/growth/campaigns/{cid}` | Enable or pause, set the budget or landing page; remove. |
| `POST` | `/api/v1/admin/growth/campaigns/{cid}/ads` | Write the responsive search ad again through the router. |
| `GET` | `/api/v1/admin/growth/campaigns/export?network=` | Google Ads Editor or Microsoft Advertising CSV. |
| `POST` | `/api/v1/admin/growth/campaigns/sync` | Create or update the Google campaigns now. |
| `GET`, `POST` | `/api/v1/admin/growth/indexnow` | Recent submissions; submit URLs (or every public post) again. |
| `GET` | `/api/v1/admin/growth/search-engines` | IndexNow, Google Search Console and Bing Webmaster: sitemap submissions, URL counts, the index verdicts per page, the directory counts. |
| `POST` | `/api/v1/admin/growth/search-engines/submit` | Submit the sitemaps and the pending addresses now (`force` resubmits today's sitemaps; `inspect: false` skips the index read). |
| `GET`, `POST` | `/api/v1/admin/growth/directories` | The register with each listing's status, last check and referrals; add a directory of your own. |
| `PATCH`, `DELETE` | `/api/v1/admin/growth/directories/{id}` | Change the status, the listing or submission address, notes; remove a directory you added (built-ins are skipped, not deleted). |
| `POST` | `/api/v1/admin/growth/directories/check` | Re-read every listing now. |
| `POST` | `/api/v1/admin/growth/directories/{id}/check` | Re-read one listing now. |
| `POST` | `/api/v1/admin/growth/directories/{id}/submit` | E-mail the kit to the directory (or record that its form was submitted). |
| `GET`, `POST` | `/api/v1/admin/growth/kit` | The listing kit; rewrite its texts through the router. |
| `GET` | `/api/v1/admin/growth/kit/email` | The kit as the plain-text e-mail a submission sends. |
| `GET`, `POST` | `/api/v1/admin/growth/social` | LinkedIn status (author, cadence, connection, next due) and the recorded posts; post now (`slug`, `text`). |
| `POST` | `/api/v1/admin/growth/social/linkedin/connect` | Start the consent flow (`author` = a page id or URN, or `pages: true` to pick one afterwards); returns the URL to open and the redirect URL to register. |
| `GET` | `/api/v1/admin/growth/social/linkedin/organizations` | The company pages the connected member administers. |
| `PATCH`, `DELETE` | `/api/v1/admin/growth/social/linkedin` | Post as another page or person (`author`); forget the stored connection. |
## 18. Autonomy: the platform's schedules route through the router
Everything the platform does for itself on a schedule - the blog posts written from the news feeds, the daily
newsletter, the SEO guides, the responsive search ads, the LinkedIn posts, the model rankings, the marketplace
harvest, the routing autopilot, the usage digests and the alert evaluation - is a **job** on one board, and every
job that produces text is a customer of the router in the same way a workspace is. The editorial and growth
writers do not pick a model: they call `route(plan=True)`, so the decision composes a persona, one or more
skills and the model around the job's own house skill (`editorial-writer` for the blog and newsletter,
`seo-writer` for guides, ads and social posts), and they offer the router a **crew** of companions as
`extra_targets` - the deployment's own Agent-Skills packages, whatever the
editorial / growth workspace installed from the marketplace, and the best-rated published marketplace skills,
personas and prompts whose listing matches the job's topics (a prompt without `{{variables}}` becomes an
instructions-only skill; the OpenSmartRoute [Marketplace](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MARKETPLACE.md) harvests them daily). The composed
system prompt is what the plan executor builds from those slots, every slot is credited with an
outcome, and once the job has parsed what came back it **grades** the run: a publishable article, ad or post is
a high-quality outcome (`source: response:grade`) for the model *and* its companions, an unusable answer a
failure. Those grades reach the learners, the per-target statistics on `/platform/dashboard/learning`, and - with
`OSR_PLATFORM_TRAINING_EVENTS` - the SLM service's training feed, so the routing SLM is trained on the
platform's own traffic exactly as on a customer's. The plan a post was written with travels with it
(`routing.plan` on `GET /api/v1/blog/{slug}` and in the console).
Two things distinguish the platform's own writers from a customer's request. Their material is public - the
news feeds, the documentation, the site's own pages - so a support address quoted in a documentation passage
is not a customer's PII and does not pin the job to on-prem models (`RequestConstraints.contains_pii=False`).
And the router judges the **assignment, not the quoted material**: a news story about jailbreaks or a security
page that spells out an injection is what the writer is asked to write *about*; judged on the whole prompt, the
signals read it as an attack and the policy would leave the job with no admissible target (seen live on the
search-ad writer). The route decision is therefore made on the prompt without its quoted block, while the
model receives the full prompt.
**The product pages' walk-throughs are answered by the router too.** The desktop and extension pages play a
person typing a draft into an app and the answer landing; the quotes on those pages were always real
(`POST /api/v1/estimate`), and the answers now are as well. The `showcase` job answers each of the walk-through
drafts through the same path as every other job - routed with a plan among the executable models, run, metered on
the platform's own `showcase` workspace, graded - and `GET /api/v1/showcase` serves them anonymously with the
target, model, tokens, price and latency each one really cost. A draft with personal data is answered in the form
the app sends it (redacted, ``). The answers are re-written every `OSR_PLATFORM_SHOWCASE_REFRESH_S`
(a day; 0 = only from the board) and kept in `/showcase.json`; until the first run - or on a deployment
without an executable model - the endpoint says so (`answer: null`, `error`) and the page shows that rather than
an invented answer.
| Setting | Default | Meaning |
|---|---|---|
| `OSR_PLATFORM_AUTONOMY_CREW` | `true` | Offer the crew to the router. `false` routes with the house skill alone. |
| `OSR_PLATFORM_AUTONOMY_CREW_SIZE` | `12` | Marketplace picks per job (best-rated first; the catalogue is searched once an hour per job). |
| `OSR_PLATFORM_AUTONOMY_CREW_MAX_CHARS` | `8000` | Longest companion body the writers load into a prompt; long documentation skills stay out of a blog post's context. |
| `OSR_PLATFORM_AUTONOMY_SLOT_MIN_CONFIDENCE` | `0.5` | A persona or skill the router proposed joins the prompt only at or above this slot confidence (the house skill always does): a doubtful companion next to the house skill can push a small model into an unusable draft. |
| `OSR_PLATFORM_SHOWCASE_REFRESH_S` | `86400` | How often the product pages' walk-through answers are written again through the router (0 = only from the board). |
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/v1/showcase` | The walk-through drafts with the answer the router wrote for each and what it cost (anonymous, cached five minutes). |
| `GET` | `/api/v1/admin/autonomy` | Every job (`content`, `newsletter`, `growth`, `social`, `syndication`, `showcase`, `rankings`, `marketplace`, `autopilot`, `reports`, `alerts`) with its cadence, owning service, state and writer (model, skill, crew offered, grades); the crew's summary; the recent operator runs. The console shows it on `/platform/admin/automation`. |
| `POST` | `/api/v1/admin/autonomy/{job}/run` | Run one job now, bypassing its lease and cadence (audited as `autonomy.run`); 404 for a job this deployment does not run (no autopilot, harvest off). |
## 19. Syndication: every article on your social networks, chat channels and blogs
Everything the blog publishes - the posts the curator writes from the news feeds, the SEO guides the growth
engine writes, anything an operator publishes by hand - goes out to the **channels** an operator sets up on
`/platform/admin/social` (`osr_platform.social`; `OSR_PLATFORM_SOCIAL=false` removes the service). A channel is
one account on one network:
| Group | Networks | What goes out |
|---|---|---|
| Social networks | LinkedIn (a profile or a company page), X, Facebook page, Threads, Mastodon, Bluesky, Reddit (a subreddit) | A message - title, summary, link and hashtags from the post's tags by default, or the channel's own template with `{title}`, `{summary}`, `{url}`, `{hashtags}`, `{brand}`, `{why}` - fitted to the network's limit without ever cutting the link (X counts a link as 23 characters), with the article attached as a link card where the network has one. |
| Chat channels | Discord (a webhook), Telegram (a bot in a chat or channel) | The message plus an embed or link preview with the title, summary, link and lead image. |
| Blog platforms and sites | Dev.to, Hashnode, Medium, Blogger, Tumblr, WordPress, Ghost, any site through a signed webhook | The article itself (Markdown or HTML as the platform wants it, the tags, the lead image) with the blog page as the **canonical URL**, published or left as a **draft** there (`post_status`). |
Each network lists the credential fields it needs (`GET /api/v1/admin/social/networks`: an access token and an
author URN for LinkedIn, the four OAuth 1.0a values *or* an OAuth 2 user token for X, a page id and page token for
Facebook, a webhook URL for Discord, a bot token and chat id for Telegram, an instance and token for Mastodon, a
handle and app password for Bluesky, a script app and account for Reddit, an API key for Dev.to, a token and
publication id for Hashnode, an integration token for Medium, a blog id and OAuth client with refresh token for
Blogger, the OAuth 1.0a values for Tumblr, a user and application password for WordPress, an Admin API key for
Ghost, a URL and optional signing secret for a webhook) and the console shows the form for the network picked.
Credentials are sealed with the deployment secret (`OSR_PLATFORM_SECRET_KEY`), never returned - a channel only
says which fields are set - and a PATCH merges new values into the stored ones (an empty value removes a field).
A LinkedIn channel may leave its token empty and post with the connection made on the Growth page (section 17).
URLs a channel will call (webhooks, instances, sites) must be public endpoints unless the deployment allows
private ones (`OSR_PLATFORM_BYOK_ALLOW_PRIVATE`).
**Schedules.** Every channel has one. `immediate` shares a post the moment it is published; `window` waits for
the next allowed weekday and time of day in the channel's time zone (`days`, `start`, `end`, `timezone` - an IANA
name); `gap_minutes` keeps two posts on the same channel at least that far apart and `max_per_day` caps a
channel's posts per local day. The slot of a new share is computed after the last post sent *or planned* on that
channel, so three articles published within a minute land, say, Monday 09:00, 09:30 and 10:00 instead of all at
once, and the fourth on Tuesday when the cap is three. A channel may also keep a **cadence** (`cadence_hours`):
when nothing went out on it for that long and nothing is queued, the newest public post not shared there in the
last 30 days is queued - a steady drumbeat between articles. An operator can share any published post on any
channels **now** (waiting for each network's answer) or **at** a chosen time, with their own text instead of the
composed message, cancel a queued share, retry a failed one and send a clearly marked **test post** to a channel.
**Delivery.** One replica per `OSR_PLATFORM_SOCIAL_INTERVAL_S` (60 s; 0 = only from the console) sends the shares
whose time has come (Redis lease) and queues the cadences; a publication also kicks a pass at once so an
`immediate` channel posts within seconds. A share the network did not answer, failed on its side or rate-limited
is retried after 1, 5, 30 and 120 minutes; one it refused (a bad token, a rejected post) is marked **failed**
with the network's message. Every attempt is on the share log with the permalink of what went out
(`GET /api/v1/admin/social/shares`), every outcome in the audit log (`social.shared`, `social.share_failed`,
`social.channel.added`, `social.channel.changed`, `social.channel.removed`, `social.share_requested`,
`social.run`). At most `OSR_PLATFORM_SOCIAL_MAX_CHANNELS` channels (30).
The same networks are **destination kinds of the workspace content service** ([Content publishing](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PUBLISHING.md)):
a workspace adds its own LinkedIn page, X account or Dev.to profile on `/platform/dashboard/content` with the
same credential fields and schedule, and its published posts reach its own accounts on its own schedule.
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/v1/admin/social` | Everything on one page: the scheduler, the network catalogue with its fields, the channels with their schedules and next slot, share counts, the newest shares, the posts to share. |
| `GET` | `/api/v1/admin/social/networks` | The networks with the credential fields each needs and its limits. |
| `POST` | `/api/v1/admin/social/preview` | The message a share would carry on a network for a post (`network`, `slug`, `template`, `hashtags`). |
| `GET`, `POST` | `/api/v1/admin/social/channels` | The channels (fields set, never values); add one (`network`, `name`, `credentials`, `schedule`, `template`, `hashtags`, `post_status`, `cadence_hours`, `auto_share`). |
| `PATCH`, `DELETE` | `/api/v1/admin/social/channels/{channel_id}` | Change a channel (credentials merge; `enabled`, `auto_share`); remove it with its share log. |
| `POST` | `/api/v1/admin/social/channels/{channel_id}/test` | Post a test message now and return the network's answer. |
| `GET` | `/api/v1/admin/social/shares` | The share log (`channel_id`, `status`, `slug`, `limit`). |
| `POST` | `/api/v1/admin/social/share` | Share a post (`slug`) on every enabled channel or `channel_ids`: now, or queued for `at`; optional `text`. |
| `POST` | `/api/v1/admin/social/shares/{share_id}/retry` | Try a failed, cancelled or waiting share again now. |
| `DELETE` | `/api/v1/admin/social/shares/{share_id}` | Withdraw a queued share (a sent one leaves the log only). |
| `POST` | `/api/v1/admin/social/run` | Run a scheduler pass now: due shares and cadences. |
## Related
* REST API reference (rendered at `/docs/api`) - every endpoint, parameter and schema.
* [Marketplace](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MARKETPLACE.md) - find, install and publish agents, skills, personas, prompts and templates.
* [User guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md) - targets, rules, plans and learning.
* [Deploy](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md) - run the same router on your own infrastructure with `osr serve`.
---
# Platform journeys: plans, features, navigation, onboarding and offboarding
This document walks through the hosted platform the way a person meets it: what you can do before you sign
up, what happens the moment you do, what each plan unlocks, where every feature lives in the navigation, how a
workspace grows into a team, and how a person, a team or a whole organization leaves cleanly. Every page,
endpoint and price named here exists on the platform today; the reference behind each one is the
[platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md), the fastest route to a first request is the
[platform quickstart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_PLATFORM.md), and the REST API reference is at `/docs/api`.
```mermaid
flowchart LR
V["Visitor`, `/platform/cli/authorize` (the device sign-in
approval page the CLI, the browser extension and the desktop app open), `/platform/email/preferences?token=...`
(the switches behind every optional e-mail, no sign-in needed) and `/platform/logout`. Links minted before the
`/platform` prefix existed redirect to their new home with the query string intact.
### The dashboard sidebar
The dashboard is everything under `/platform/dashboard`. The header carries the **workspace switcher**
(personal workspace and every organization you belong to), the **notifications bell** with the unread count,
the theme toggle, the **assistant** and the account menu (docs, support, sign out; the way back to the console
when an operator session exists). The sidebar is grouped as follows.
| Section | Page | Path | What you do there | Plan |
|---|---|---|---|---|
| Overview | Overview | `/platform/dashboard` | Requests, tokens and spend of the last 7 days, the quota bar, the 24-hour traffic and latency series, active alerts, the getting-started checklist until it is complete, *What's new* | all |
| Overview | Activity | `/platform/dashboard/activity` | Every metered request (endpoint, target, domain, complexity, tokens, outcome - never the prompt); open the trace drawer, report an outcome, download the trace as JSON, share `?request_id=` | all |
| Overview | Human queue | `/platform/dashboard/handoffs` | Requests the router deferred to a person; a teammate answers or withdraws them | all |
| Insights | Usage | `/platform/dashboard/usage` | Per day, target, endpoint, app and tenant; CSV export | all |
| Insights | Savings | `/platform/dashboard/savings` | Routed cost against a baseline model; CSV export for finance | all |
| Insights | AI spend | `/platform/dashboard/spend` | What the browser extension observed the team spend outside the router against the router's quote | all |
| Insights | Impact | `/platform/dashboard/impact` | The one-page value report: cost saved, requests and success, data protected, governance score; per key, tenant or client app | all |
| Build | API keys | `/platform/dashboard/keys` | Create, name, scope, limit, rotate and revoke keys; pin a key to a tenant | all |
| Build | Apps | `/platform/dashboard/apps` | Install the browser extension and the desktop app, see whether each is connected and what it sent, the CLI / SDK sign-in | all |
| Build | Model providers | `/platform/dashboard/providers` | Connect your own Azure OpenAI, Bedrock, Vertex / Gemini, OpenAI, Anthropic or any OpenAI-compatible endpoint; check, map, rotate, remove | all (`byok`) |
| Build | MCP servers | `/platform/dashboard/mcp-servers` | Connect your own MCP servers; their tools join the routing pool as `server:tool` targets under an approval mode (auto / read-only / manual); sync, disable, remove | all (`byok`) |
| Build | Integrations | `/platform/dashboard/integrations` | SDK snippets with your key (TypeScript, OpenAI Python client, curl) and MCP configuration for VS Code, Cursor, Windsurf, Claude Code and Claude Desktop | all |
| Build | Marketplace | `/platform/dashboard/listings` | What you published and everything you installed | all |
| Operate | Governance | `/platform/dashboard/governance` | Controls in force, compliance posture and score, guard activity, key hygiene, the workspace policy editor (constraints, budgets, privacy, aliases, cache, stickiness, objective, judge), budgets, the response cache and the request-log export (*Data*), *Compliance report* download | all |
| Operate | Events | `/platform/dashboard/events` | The router's event history (1 hour to 7 days), filters, search by request or trace id, **Live** follow | all |
| Operate | Learning | `/platform/dashboard/learning` | How the router learns from your outcomes: strategy weights, leaderboards, the routing SLM and the autopilot | all (leaderboards: `stats`) |
| Operate | Health | `/platform/dashboard/health` | Deployment readiness, traffic and latency percentiles, circuit breakers, decision cache | `stats` |
| Operate | Notifications | `/platform/dashboard/notifications` | The alert inbox, delivery channels (e-mail, webhook, Slack, Teams) with *Send test*, the delivery log | all |
| Organization | Tenants | `/platform/dashboard/tenants` | Per-tenant constraints, budgets and spend | `tenants` |
| Organization | Audit trail | `/platform/dashboard/audit` | The hash-chained decision trail with chain verification; CSV / JSONL export | `audit` |
| Organization | Members | `/platform/dashboard/members` | Members, roles and pending invitations of an organization; on a personal workspace, the *become an organization* step | all |
| Organization | Workspace | `/platform/dashboard/workspace` | Name, slug and plan; create or switch to an organization; organization SSO; *Delete organization* | all (SSO: `sso`) |
| Account | Plan and billing | `/platform/dashboard/billing` | Quota, subscription, trial, plan cards, pay as you go, coupons, credits, invoices, usage and charges, service level | all |
| Account | Support | `/platform/dashboard/support` | Open a ticket, follow the conversation live, close it | all |
| Account | Account | `/platform/dashboard/account` | Sign-in identities, password, browser sessions, e-mail preferences, agreements, *Download my data*, *Delete my account* | all |
Every table with *Export CSV* downloads exactly the rows shown. A failed read says what it means (session
expired, feature not on the plan, budget exhausted with the amounts, rate limit with the retry delay) with a
*Retry* button.
### The operator console
Operators sign in at `/platform/admin/login`. The console sidebar is grouped into **Deployment** (Overview,
Analytics with the activation funnel, Health, Request log), **Directory** (Users, Workspaces, Tenants, Leads,
Sales, Support, Plans, Billing, Finance, Coupons), **Learning** (Routing SLM), **Platform** (Providers, Offers,
Marketplace, Blog and newsletter, Growth, Model rankings, Services, Notifications, Reports, Automation,
Newsletter, Configuration, Terms and privacy) and **Access** (Operators, Audit log, Mail, Search). Every page and
its endpoints are listed under *Operator console* in the [platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md#operator-console).
## 4. The new user, step by step
```mermaid
flowchart LR
A["1 Try it` (valid fourteen days), which shows the workspace, the role offered
and the address it was sent to (`GET /api/v1/auth/invites/{token}`). Sign in - or create the account - with
that address and accept (`POST /api/v1/auth/invites/{token}/accept`): you join with the offered role, the
e-mail address counts as confirmed (the invitation reached that inbox) and your session moves to the team's
workspace, so steps 1-3 below are already done. You still get a personal free workspace of your own.
### Step 1 - Try it without an account
`/estimate` quotes a request before it is sent (`POST /api/v1/estimate` is public, rate-limited per IP),
`/compare` puts alternatives side by side, `/roi` estimates the saving against always calling one model, and
`/models` and `/rankings` show every target with its cost, latency and observed quality. None of this needs a
key. The playground itself (`/platform/playground`) asks for the free sign-in first - decisions are metered on
your workspace - which is step 2.
### Step 2 - Sign up
`/platform/signup` with a provider button or an e-mail address (add a password if you want to sign in by
e-mail later; at least 8 characters). Behind it is `POST /api/v1/signup` (five sign-ups per IP address and day).
It creates you, a **personal workspace on the Free plan** and its **first API key**, signs you in and sends a
welcome message and the confirmation link.
:::warning[The API key is shown once]
Copy it into a secret manager before leaving the page. It cannot be shown again; you can create, rotate and
revoke keys at any time on `/platform/dashboard/keys`.
:::
### Step 3 - Confirm your e-mail address
The link in the mail goes to `/platform/verify-email?token=...` (valid three days; `POST /api/v1/auth/verify`).
Signing in through a provider that vouches for the address, accepting an invitation sent to it or completing a
password reset confirms it too. Until then the Account page shows a banner with *Send again*
(`POST /api/v1/auth/verify/send`, once a minute). Checklist step: **Confirm your e-mail address**.
### Step 4 - Route a first request
In the playground - *See the decision* - the request is routed on your workspace, shows the target chosen, the
alternatives and the strategies' scores, and appears on `/platform/dashboard/activity` with its trace. From a
terminal:
```bash
curl -s "$OSR_URL/api/v1/route" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"text": "Summarise this contract clause for a non-lawyer"}'
```
Checklist steps: **Create an API key** (already done by signup; `POST /api/v1/keys` for more) and **Route a first
request** (`POST /api/v1/route`).
### Step 5 - Connect a model provider
A routing decision is free; an *answer* needs a target that can execute. **Nothing has to be connected for
that**: the deployment's own providers execute the catalogue's targets for every workspace (on the hosted
platform the Azure OpenAI tiers, the self-hosted Ollama models - Qwen, Gemma, GLM, DeepSeek, gpt-oss - and the
speech server), and `/platform/dashboard/providers` lists them under *Included with the platform* with each
provider's health and the target -> model pairs (`platform_providers` in `GET /api/v1/workspace/providers`;
`GET /api/v1/targets` marks them `executable`). Bringing your own is optional: on the same page pick a preset -
Azure OpenAI, Amazon Bedrock, Google Vertex AI / Gemini, OpenAI, Anthropic, Mistral, Groq, OpenRouter, an
in-house Ollama, vLLM or LiteLLM gateway, any OpenAI-compatible endpoint - paste your own key, *Check* it and map
catalogue targets to models on it. Your key never leaves the workspace; requests to that provider are billed by
the provider to you; targets you did not map keep the platform's providers. Checklist step: **Models ready**
(ticked by the platform's providers or by your own).
### Step 6 - A target answers
Point any OpenAI client at `/v1` with `model: "auto"` (every plan), or send `execute: true` on `/api/v1/route`
(plans with the `plan` feature - Pro and up):
```bash
curl -s "$OSR_URL/v1/chat/completions" -H "Authorization: Bearer $OSR_API_KEY" -H "Content-Type: application/json" \
-d '{"model": "auto", "messages": [{"role": "user", "content": "Summarise this contract clause for a non-lawyer"}]}'
```
The answer carries `X-OSR-Target` and the token usage; the Activity row now has tokens, provider cost and, under
the target, the provider and upstream model that answered (`ollama-qwen3-5-4b / qwen3.5:4b`).
First-value milestone: **A target answered a request (not only a routing decision)**.
### Step 7 - Report an outcome
Expand the Activity row (or open its trace) and report good / poor, a quality score and, optionally, the target
that would have done better; from code, `POST /api/v1/feedback {request_id, target_id, success, quality}`. The
learners update on your own traffic and `/platform/dashboard/learning` shows what changed. Checklist step:
**Report an outcome so the router learns**; milestone: **An answer was reported as good**.
### Step 8 - Wire your application, IDE and apps
- `/platform/dashboard/integrations`: copy-ready snippets with your key - the TypeScript SDK from npm, the
OpenAI Python client against `/v1`, curl - and MCP configuration for VS Code, Cursor, Windsurf, Claude Code
and Claude Desktop.
- `/platform/dashboard/apps`: install the **browser extension** (Chrome, Edge and Firefox - Safari is not
available yet; the pre-send advisor on ChatGPT, Claude, Gemini, Copilot and Perplexity) and the **desktop
app** (Windows, macOS and Linux; the routed command bar over any application). Both connect
with a device sign-in: the app shows a code, opens `/platform/cli/authorize`, you approve, and a key labelled
for that app appears on the keys page. The **CLI** does the same with `osr login`.
- Give a key that leaves your hands the smallest `scopes` (`read`, `route`, `execute`, `feedback`, `manage`),
an `expires_in_days` and `limits` (daily / monthly budget, requests per minute, allowed targets).
### Step 9 - Come back
The second day is the milestone **Came back on another day**. What brings people back: `/platform/dashboard/usage`
and `/savings` (what routing cost and saved), `/impact` (the one-page report a team lead or auditor takes away),
the weekly **value report** e-mailed to owners (schedule under Account › E-mail: daily, weekly, monthly or off),
alerts in the inbox and by e-mail (quota at 80 %, budgets, error rate, guard activity) and *What's new* on the
overview after each release.
A workspace that has not routed anything a day after signing up receives one **getting-started reminder** with
its next step (category *getting started*, switchable off like every optional mail).
## 5. The existing user: journeys by task
### Developer - day to day
| Task | Where | Notes |
|---|---|---|
| Create a key for a service, CI or a partner | `/platform/dashboard/keys` | Scopes, expiry (1-3650 days), limits; the secret is shown once |
| Rotate a key | Keys page › *Rotate* (`POST /api/v1/keys/{key_id}/rotate`) | New secret for the same key; the old one stops at once, the expiry is kept |
| Find out why a request went where it went | `/platform/dashboard/activity` › trace drawer, or `GET /api/v1/trace/{request_id}` | Signals, admission, ranking, the winner and why, execution, what crossed the boundary |
| Follow the router live | `/platform/dashboard/events` › **Live** | `GET /api/v1/events/stream` |
| Quote before sending | `POST /api/v1/estimate`, the MCP `estimate` tool | Metered as `estimate`; no execution |
| Build a plan (persona -> skill -> model) | `plan: true` on `/api/v1/route` | Plans on Pay as you go, Pro, Team and Enterprise |
| Use the router from an IDE or agent | `/platform/dashboard/integrations` › MCP | Or `osr mcp --remote` from the CLI |
| Answer a deferred request | `/platform/dashboard/handoffs` | Resolving feeds an outcome to the learners |
### Growing into a team
1. **Become an organization.** On a personal workspace the *Members* page offers exactly this: `POST /api/v1/workspace/convert`
with a name changes only the kind - plan, subscription, keys, tenants and usage stay; you receive a fresh
free personal workspace alongside. Or create a second workspace with `POST /api/v1/workspaces` and switch
between them in the header (`POST /api/v1/workspaces/{account_id}/switch`).
2. **Invite people.** `/platform/dashboard/members` › *Invite* (`POST /api/v1/workspace/invites`): the invitee
gets a link to `/platform/invite/` valid fourteen days; the same link is returned once to you.
*Resend* issues a fresh link. Seats follow the plan (`max_members`). Checklist step for organizations:
**Invite a teammate**.
3. **Roles.** `member` routes and reads usage, savings and activity; `admin` also manages keys, tenants,
members and the workspace name; `owner` grants or removes `owner`, configures SSO and deletes the
organization. An organization always keeps at least one owner.
4. **One key per person.** In a team workspace, a key per person makes *Impact per API key* the per-person
view.
5. **Organization SSO** (Enterprise). `/platform/dashboard/workspace` › SSO (`PUT /api/v1/workspace/sso`): the
company identity provider (Microsoft, Google, GitHub, GitLab, LinkedIn or any OIDC issuer), the allowed
e-mail domains, a default role and a `role_map` from the provider's group claim, re-applied at every
sign-in. The sign-in page shows the provider as `org-`.
### Governance and compliance
| Task | Where |
|---|---|
| Set rules for every request of the workspace (data boundary, deny lists, cost and latency caps, budgets, request logging) | `/platform/dashboard/governance` › policy editor (`PUT /api/v1/policy`, admin) |
| Give a customer or team its own rules and budgets | `/platform/dashboard/tenants` (`PUT /api/v1/tenants`), then a **tenant-pinned key** on the keys page |
| Read the compliance posture and fix what it flags | Governance › posture card; each finding links to the page that fixes it |
| Hand an auditor evidence | Governance › *Compliance report* (`GET /api/v1/governance/report?format=markdown`) |
| Verify the decision trail | `/platform/dashboard/audit` (Team and Enterprise) |
| Be told when something happens | `/platform/dashboard/notifications` › channels (e-mail, webhook, Slack, Teams) |
| See who signed in, changed a role or a key | `GET /api/v1/workspace/audit` (admins); your own history `GET /api/v1/me/audit` |
### Money: usage, savings and billing
| Task | Where |
|---|---|
| What was spent, per day, target, endpoint, app or tenant | `/platform/dashboard/usage` (`GET /api/v1/usage`) |
| What routing saved against a baseline model | `/platform/dashboard/savings` (`GET /api/v1/savings`) |
| The value report for a lead or an auditor | `/platform/dashboard/impact` |
| What the team spends outside the router | `/platform/dashboard/spend` (needs the browser extension) |
| Cap spend | Budgets in the workspace policy, per tenant and per key (`429` with `X-Budget-*` when reached) |
| Upgrade, change interval, cancel | `/platform/dashboard/billing` (owners and admins) |
| Provider cost and the workspace's charge, by day, model, provider, key or tenant | Billing › *Usage and charges* (`GET /api/v1/billing/usage`) |
| Invoices | Billing › invoices (`GET /api/v1/billing/invoices`) |
| The service level this month | Billing › *Service level* (`GET /api/v1/sla`) |
### Support
`/platform/dashboard/support` opens a ticket (subject, category, priority, optionally the request id it is
about). The confirmation carries the reference `OSR-00042`; the conversation updates live, e-mailed links
(`?ticket=OSR-00042`) open it directly, a reply reopens a resolved ticket and *Close* ends it. Tickets from
Team and Enterprise workspaces open at `high` priority at least. The assistant in the header answers from the
documentation first.
## 6. Plan changes
```mermaid
stateDiagram-v2
[*] --> Free: signup
Free --> Trial: checkout of a promoted plan, or a coupon
Free --> Pro: checkout
Free --> PayAsYouGo: switch on the billing page
Trial --> Pro: subscription starts
Trial --> Free: trial ends without payment
Pro --> Team: plan change in place, prorated
Team --> Enterprise: through sales
Pro --> Free: cancel at period end
Team --> Free: cancel at period end
PayAsYouGo --> Pro: checkout
```
- **Upgrade.** *Plan and billing* › plan cards › *Start 90-day free trial* / *Subscribe* (`POST /api/v1/billing/checkout
{plan, interval}`). The first checkout of a workspace starts the plan's 90-day trial at once - no Stripe, no
card; *Subscribe now* on the trial card (or `"coupon": ""`) goes to Stripe Checkout to pay. A workspace that
already has a subscription has its prices swapped in place with prorations - never a second subscription - and
the plan applies at once.
- **Coupons and the launch offer.** The plans' own trials are the standing promotion; an operator's coupon on
top works the same way, and a coupon code entered
on the billing page (`POST /api/v1/billing/coupons/redeem`) does the same. Owners are mailed three days
before a coupon trial ends; a trial that ends without a subscription returns the workspace to Free.
- **Pay as you go.** The *Pay as you go* card switches the workspace to the prepaid plan without Stripe
Checkout; *Buy credits* tops up the wallet, and every model on offer is paid by the token from it.
`402` when the credits run out.
- **Enterprise.** *Talk to sales* (`/contact`) records an inquiry (`OSR-S-00007`); the deal is applied
by an operator on the workspace, together with custom quotas, retention and region where agreed.
- **Downgrade or cancel.** *Cancel at period end* (`POST /api/v1/billing/cancel`): the plan stays until the paid
period ends, then the workspace returns to Free; keys, history, members and providers stay. To keep a paid
plan but change it, use the plan cards. *Manage subscription* opens the Stripe Billing Portal for the
payment method, billing details and receipts.
- **Payment problems.** A failed charge sets the state to `past_due`, mails the owners a link to update the
payment method and shows a banner on the billing page; a paid invoice clears it. A subscription Stripe gives
up on returns the workspace to Free.
What a downgrade removes is exactly what the plan sold: tenants, statistics and audit pages lock again, plans
are refused with `403` and `X-Upgrade: true`, and the quota falls to the new plan's. Data is not deleted by a
plan change.
## 7. Offboarding
Three different things end at three different scopes: a **person leaves a team**, a **person leaves the
platform**, or a **team shuts a workspace down**. Each one has a checklist below. Everything on it is on the
audit trail, so an offboarding can be evidenced afterwards.
### A person leaves a team
The person can go themselves: **Leave** on the members page (`DELETE /api/v1/workspace`) removes their own
membership and moves their session to their personal workspace; an owner hands ownership over first. When the
departure is managed instead, a workspace `admin` or `owner` runs the checklist:
1. **Revoke the keys they held.** `/platform/dashboard/keys` › *Revoke* on every key they created or used
(`DELETE /api/v1/keys/{key_id}`); one key per person makes this a single row. Keys minted by their
extension, desktop app or CLI carry those names.
2. **Remove the membership.** `/platform/dashboard/members` › *Remove* (`DELETE /api/v1/workspace/members/{user_id}`).
Their browser sessions stop reaching the workspace at once; they keep their own personal workspace.
3. **Ownership first.** If they were the last `owner`, grant `owner` to someone else before removing them -
an organization always keeps one.
4. **SSO-managed teams.** Remove them from the mapped group in the identity provider; with `require_match: true`
the next sign-in refuses them, and without it they fall back to the default role until removed.
5. **Check the trail.** `GET /api/v1/workspace/audit` shows the removal (`member.removed`), the key revocations
and their last sign-ins.
### A person leaves the platform
Done by the person on `/platform/dashboard/account`:
1. **Download my data** (`GET /api/v1/me/export?days=`): one JSON document with the profile, identities,
sessions, memberships and history, and the current workspace with its plan, members, key prefixes, tenants,
policy, SSO, usage, request log and reported outcomes of up to a year.
2. **Hand over what others depend on.** Transfer `owner` on any organization with other members; an organization
with other members needs another owner before the account can go. Rotate or revoke keys that live in
shared systems if a teammate will keep using them - keys belong to the workspace, not to the person.
3. **Cancel a paid subscription on the personal workspace first** (*Plan and billing* › *Cancel at period
end*, or the Stripe portal). Deleting the account disables workspaces only you belonged to and revokes
their keys, but it does not cancel a subscription - only *Delete organization* does that immediately.
4. **Switch off the mail you no longer want** meanwhile, under Account › E-mail (or the link in any message).
5. **Delete my account** (`POST /api/v1/me/delete {confirm, password}`): type the e-mail address and, when one
is set, the password. Sessions, linked identities and memberships go; workspaces only this user belonged to
(the personal workspace) are disabled and their keys revoked; a notice is e-mailed. The lifecycle log keeps
the deletion event.
### A team shuts a workspace down
Done by an `owner` of the organization:
1. **Export what finance and compliance need.** *Usage*, *Savings*, *Activity* and *Audit trail* export CSV
(the audit trail also as the JSONL the offline verifier reads); *Compliance report* on Governance downloads
the evidence bundle; invoices stay reachable in the Stripe portal.
2. **Decide between cancelling and deleting.** *Cancel at period end* keeps the workspace on Free with its
history, keys and members - the right choice for a pause. *Delete organization* ends it.
3. **Disconnect clients.** Revoke the keys used by services, CI, the extension and the desktop app so nothing
keeps calling (`DELETE /api/v1/keys/{key_id}`); remove BYOK providers if the credentials should not remain
stored (`/platform/dashboard/providers`).
4. **Delete organization** on `/platform/dashboard/workspace` (`POST /api/v1/workspace/delete {confirm, password}`):
type the organization's name or slug and, when one is set, the password. Keys, members, invitations,
tenants, policy, the SSO connection, usage and outcomes go at once; a live subscription is cancelled
immediately - nothing more is billed, the paid remainder is not refunded automatically. Every member keeps
their personal workspace and their session moves to it. The deletion is on the audit trail (`account.deleted`).
Personal workspaces are not deleted on their own; they go with the user.
### What the platform keeps, and for how long
- **Request rows and traces** follow the deployment's retention (`OSR_PLATFORM_RETENTION_DAYS`) or the
workspace's own `retention_days` on an Enterprise agreement; `GET /api/v1/governance` reports the value in
force. Prompt and answer text is stored only where the workspace opted in with `log_payloads`, and
`DELETE /api/v1/activity/logs` erases it while the decision facts stay.
- **The account audit log** (sign-ins, invitations, roles, removals, deletions, operator actions) is kept as
evidence of what happened; it names actors and actions, never prompt text.
- **Outgoing mail** sits in the outbox under the deployment's retention; every optional category can be
switched off by the recipient at any time.
- **Region.** A workspace's keys, usage and traces never leave the regional stack it was created in.
## 8. Operators: onboarding and offboarding users from the console
The operator console mirrors every step above for the people running the deployment:
| Task | Where in the console |
|---|---|
| Watch new workspaces move from signup to a first answer, a good outcome, a return visit and a paid plan; find the ones stuck before their first request and send them the getting-started reminder | Analytics › *Activation funnel*, *Stuck before the first request*, *Send reminders now* |
| Create a user with a personal workspace, plan and first key; re-send or force the e-mail confirmation; issue a reset link | Users › *Create*, the user's page |
| Set a plan by hand (deployments without Stripe), custom quotas, retention and region after a sales agreement | Workspaces › the workspace's page › plan, *Custom quotas and retention* |
| Apply a won deal, run a coupon promotion | Sales, Coupons |
| Offboard a user: disable, sign out everywhere, delete (workspaces they alone belonged to are disabled, keys revoked) | Users › the user's page, or a bulk action on selected rows |
| Offboard a workspace: disable or delete it, remove members, revoke keys | Workspaces › the workspace's page |
| See every lifecycle event, filter by action, actor, user or workspace, export CSV | Audit log |
| Onboard or remove an operator, reset a locked-out operator's two-factor authentication, revoke console sessions | Operators (superadmins) |
Every console action is an API call recorded with `operator:` as the actor, so the same
administration works from scripts with the static admin token.
## 9. Quick reference: the endpoint behind each step
| Step | Endpoint |
|---|---|
| Sign up | `POST /api/v1/signup` |
| Confirm e-mail / send again | `POST /api/v1/auth/verify`, `POST /api/v1/auth/verify/send` |
| Sign in by e-mail; forgot / reset | `POST /api/v1/auth/password/login`, `POST /api/v1/auth/password/forgot`, `POST /api/v1/auth/password/reset` |
| Set, change or remove the password | `POST /api/v1/auth/password`, `DELETE /api/v1/auth/password` |
| Device sign-in (CLI, extension, desktop) | `POST /api/v1/auth/device/code`, `POST /api/v1/auth/device/approve`, `POST /api/v1/auth/device/token` |
| Getting-started checklist | `GET /api/v1/onboarding` |
| Keys | `POST /api/v1/keys`, `PATCH /api/v1/keys/{key_id}`, `POST /api/v1/keys/{key_id}/rotate`, `DELETE /api/v1/keys/{key_id}` |
| Route, execute, quote, feedback | `POST /api/v1/route`, `POST /v1/chat/completions`, `POST /api/v1/estimate`, `POST /api/v1/feedback` |
| Trace and events | `GET /api/v1/trace/{request_id}`, `GET /api/v1/events`, `GET /api/v1/events/stream` |
| Included models and your own providers | `GET /api/v1/workspace/providers`, `GET /api/v1/targets`, `GET /v1/models` |
| Usage, savings, quota, service level | `GET /api/v1/usage`, `GET /api/v1/savings`, `GET /api/v1/quota`, `GET /api/v1/sla` |
| Organization | `POST /api/v1/workspaces`, `POST /api/v1/workspace/convert`, `POST /api/v1/workspaces/{account_id}/switch` |
| Members and invitations | `GET /api/v1/workspace/members`, `POST /api/v1/workspace/invites`, `POST /api/v1/workspace/invites/{id}/resend`, `DELETE /api/v1/workspace/members/{user_id}` |
| Accept an invitation | `GET /api/v1/auth/invites/{token}`, `POST /api/v1/auth/invites/{token}/accept` |
| Leave an organization yourself | `DELETE /api/v1/workspace` |
| Sessions and sign-out | `GET /api/v1/sessions`, `DELETE /api/v1/sessions/{session_id}`, `POST /api/v1/auth/logout` |
| SSO | `PUT /api/v1/workspace/sso` |
| Policy, tenants, governance | `PUT /api/v1/policy`, `PUT /api/v1/tenants`, `GET /api/v1/governance`, `GET /api/v1/governance/report` |
| Billing | `POST /api/v1/billing/checkout`, `GET /api/v1/billing/subscription`, `POST /api/v1/billing/portal`, `GET /api/v1/billing/invoices`, `POST /api/v1/billing/cancel`, `POST /api/v1/billing/coupons/redeem` |
| Reports and e-mail preferences | `PUT /api/v1/workspace/reports`, `GET /api/v1/me/email-preferences`, `PUT /api/v1/me/email-preferences` |
| Support | `POST /api/v1/support/tickets`, `GET /api/v1/support/tickets/{key}`, `POST /api/v1/support/tickets/{key}/messages`, `POST /api/v1/support/tickets/{key}/close` |
| History and export | `GET /api/v1/me/audit`, `GET /api/v1/workspace/audit`, `GET /api/v1/me/export` |
| Delete | `POST /api/v1/me/delete`, `POST /api/v1/workspace/delete` |
## Related
- [Platform quickstart](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/QUICKSTART_PLATFORM.md) - the first request in five minutes.
- [Platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md) - every endpoint, header, plan rule and dashboard page in full.
- [Browser extension](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/EXTENSION.md) and [Desktop app](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/DESKTOP.md) - the two client apps and how they connect.
- [Cost estimates and the MCP server](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MCP.md) - the router inside VS Code, Cursor, Claude and agents.
- [Security](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SECURITY.md) - the threat model behind the guard, PII redaction and key scopes.
---
# Model server
The OpenSmartRoute model server (`osr-models`) hosts open-weight models on your own hardware - a laptop, a VM,
a container, a GPU node - behind the APIs clients already speak. It is what the platform's self-hosted targets
run on, and it works on its own for anyone who wants a model server with a fit check, an admission queue and
metrics instead of a cold-load lottery.
| | |
|---|---|
| **APIs** | Ollama (`/api/generate`, `/api/chat`, `/api/embed`, `/api/tags`, `/api/show`, `/api/ps`, `/api/pull`, `/api/create`, `/api/copy`, `/api/delete`, `/api/blobs`, `/api/version`), OpenAI (`/v1/models`, `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/responses`, `/v1/rerank`), Anthropic Messages (`/v1/messages`), control (`/models`, `/models/load`, `/models/unload`), operations (`/healthz`, `/readyz`, `/metrics`) |
| **Engine** | llama.cpp `llama-server`, one process per loaded model, on CPU or NVIDIA CUDA; GGUF weights |
| **Sources** | ollama.com (`qwen3.5:4b`), Hugging Face GGUF repositories (`hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M`, or a file inside a repository), local `.gguf` files |
| **Store** | content-addressed blobs and manifests in Ollama's on-disk layout: a store Ollama filled is served as-is, so switching costs no download |
| **Residency** | a model is loaded only when its footprint (weights + KV cache x context x slots + overhead) fits the memory budget after evicting *idle* models; a busy model is never evicted; `keep_alive` pins, expires or unloads |
| **Admission** | `parallel` slots per model, a bounded queue with a deadline, `503` + `Retry-After` past it (the platform re-routes to a cloud tier on that answer instead of waiting) |
| **Metrics** | Prometheus: requests, latency, tokens (prompt / cached / completion), slots, queue, resident bytes, loads, evictions, refusals - plus the engine's own series per model |
## Run it
**Docker (CPU)**
```sh
docker run -d --name models -p 11434:11434 -v osr-models:/root/.ollama \
-e OSR_OLLAMA_MODELS="qwen3.5:4b qwen3-embedding:4b" -e OSR_OLLAMA_PRELOAD=1 \
opensmartroute/models
curl http://localhost:11434/api/chat -d '{"model":"qwen3.5:4b","messages":[{"role":"user","content":"Hello"}],"stream":false}'
```
**Docker (NVIDIA GPU)** - the `cuda` image puts every layer on the GPU (`OSR_MODELS_GPU_LAYERS` to split):
```sh
docker run -d --gpus all -p 11434:11434 -v osr-models:/root/.ollama opensmartroute/models:cuda
```
**The command line** - the same binary inside the container manages the store and talks to the server:
```sh
docker exec models osr-models pull hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M
docker exec models osr-models run hf.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M "Why is the sky blue?"
docker exec models osr-models ps
```
Outside a container the package runs next to any `llama-server` binary from a llama.cpp release (point
`OSR_MODELS_LLAMA_SERVER` at it or put it on the `PATH`) and `osr-models serve` starts the server.
The server listens on Ollama's port (`11434`) and reads Ollama's variables, so an Ollama client, probe or
start-up script works unchanged; the container even answers to `ollama list`, `ollama pull`, `ollama run`.
## Models
| Command | What it does |
|---|---|
| `osr-models pull qwen3.5:4b` | from the ollama.com library (any registry speaking its protocol: `host/namespace/name:tag`) |
| `osr-models pull hf.co//:` | the GGUF of that quantisation from a Hugging Face repository (`Q4_K_M` when no quantisation is named); a multimodal projector (`mmproj-*.gguf`) in the same folder comes along; `HF_TOKEN` for gated repositories |
| `osr-models pull hf.co///.gguf` | one file of a repository |
| `osr-models pull ./model.gguf --name my-model` | import a local file |
| `osr-models pull qwen3.5:4b=hf.co/unsloth/Qwen3.5-4B-GGUF:Q4_K_M` | pull from one source, store under another name: every API, `show` and the platform's mapping see `qwen3.5:4b` while the weights come from the Hugging Face repository. Use it where the ollama.com blob is Ollama's own conversion that a standard llama.cpp cannot load (`gpt-oss:20b`, `glm-4.7-flash`, `glm-ocr`, `translategemma:4b`, `qwen3.5:4b`). `POST /api/pull` takes the same form in `model`, or the source in a `source` key |
| `osr-models list` / `ps` / `show MODEL` / `rm MODEL` / `stop MODEL` | the store, what is in memory, a model's details and metadata, delete, unload |
| `POST /api/create` | a new model from an existing one plus a system prompt, parameters (`temperature`, `stop`, `num_ctx`, ...), a template or adapters - the same body Ollama takes |
Pulls resume after an interruption, verify the digest, and are shared between replicas that mount the same
store through a lock in it. The deployment decides which sources are allowed (`OSR_MODELS_SOURCES`, default
ollama.com and Hugging Face).
A model's card is what `POST /api/show` returns: `details` (family, parameter count, quantisation),
`model_info` (the GGUF metadata - layer count, heads, trained context), `capabilities` (`completion`, `tools`,
`thinking`, `vision`, `embedding`, read off the weights), `system`, `parameters`, `template`, `license`. The system
prompt and parameters of a card apply to every request that does not set its own.
## Requests
Everything the Ollama and OpenAI dialects define for chat works on both paths: streaming, `tools` and tool calls,
images (multimodal models), `format: "json"` or a JSON schema (`response_format` on `/v1`), `think` levels
(`reasoning_effort` on `/v1` - `none` switches a thinking model's reasoning off so it answers inside a small
`max_tokens`), `keep_alive`, the Ollama `options` (`temperature`, `top_p`, `top_k`, `min_p`, `num_predict`,
`seed`, `stop`, penalties). Answers carry Ollama's statistics (`prompt_eval_count`, `prompt_eval_cached_count`,
`eval_count`, durations in nanoseconds, `load_duration` on the request that loaded the model) and the OpenAI
`usage` with `cached_tokens`. The prompt cache is on: a repeated system prompt or conversation prefix is not
evaluated again.
An empty request loads a model (`{"model": "...", "messages": []}`), `keep_alive: 0` with it unloads;
`GET /api/ps` lists what is resident with its slots, queue and `expires_at` (`0001-01-01` = pinned).
## Settings
Every setting is an environment variable; the `OSR_MODELS_*` name wins over the Ollama name it mirrors.
| Setting | Ollama name | Default | Meaning |
|---|---|---|---|
| `OSR_MODELS_HOST` | `OLLAMA_HOST` | `127.0.0.1:11434` | listen address |
| `OSR_MODELS_STORE` | `OLLAMA_MODELS` | `~/.ollama/models` if present, else `~/.osr/models` | the store directory (`/root/.ollama/models` in the image) |
| `OSR_MODELS_PRELOAD` | `OSR_OLLAMA_MODELS` | - | models pulled at start (`name` or `name=source`; a `name=source` entry whose stored weights came from anywhere else - an earlier ollama.com pull of the same tag - is pulled again from the source); with `OSR_OLLAMA_PRELOAD=1` loaded and pinned too |
| `OSR_MODELS_CONTEXT` | `OLLAMA_CONTEXT_LENGTH` | `8192` | context per slot (clamped to the model's trained window and its `num_ctx`) |
| `OSR_MODELS_PARALLEL` | `OLLAMA_NUM_PARALLEL` | `1` | generations one model decodes at once (embedding models always 1) |
| `OSR_MODELS_KV_CACHE` | `OLLAMA_KV_CACHE_TYPE` | `f16` | KV cache precision (`q8_0` halves it; `q4_0` quarters it) |
| `OSR_MODELS_MAX_LOADED` | `OLLAMA_MAX_LOADED_MODELS` | `0` = as many as fit | cap on resident models |
| `OSR_MODELS_KEEP_ALIVE` | `OLLAMA_KEEP_ALIVE` | `5m` | idle time before a model is unloaded; `-1` = never |
| `OSR_MODELS_QUEUE_MAX` | `OLLAMA_MAX_QUEUE` | `4 x parallel` | requests that may wait for a slot before `503` |
| `OSR_MODELS_QUEUE_TIMEOUT` | - | `20s` | longest wait for a slot |
| `OSR_MODELS_LOAD_TIMEOUT` | `OLLAMA_LOAD_TIMEOUT` | `20m` | longest load (a network share is slow) |
| `OSR_MODELS_REQUEST_TIMEOUT` | - | `15m` | longest silence from an engine during one request (a thinking model on a CPU can be quiet for minutes) |
| `OSR_MODELS_DEVICE` | - | `auto` | `cpu`, `cuda`, or detect |
| `OSR_MODELS_GPU_LAYERS` | - | all | layers offloaded to the GPU |
| `OSR_MODELS_THREADS` | - | engine default | CPU threads per engine |
| `OSR_MODELS_FLASH_ATTENTION` | `OLLAMA_FLASH_ATTENTION` | on | flash attention (forced on for a quantised KV cache) |
| `OSR_MODELS_MEMORY` | - | detected (cgroup limit or physical memory) | the memory budget before headroom, e.g. `28Gi` |
| `OSR_MODELS_HEADROOM` | - | `0.12` | share of memory kept for the OS and page cache |
| `OSR_MODELS_SOURCES` | - | `registry.ollama.ai huggingface.co hf.co` | registries pulls may use |
| `OSR_MODELS_HF_TOKEN` | `HF_TOKEN` | - | Hugging Face token for gated repositories |
| `OSR_MODELS_TOKEN` | - | - | bearer token required off the loopback (`/healthz`, `/readyz`, `/api/version` stay open) |
| `OSR_MODELS_SLEEP_IDLE` | - | off | seconds of idleness after which an engine releases its memory but keeps its process |
| `OSR_MODELS_LLAMA_SERVER` | - | `PATH` | the engine binary |
Sizing is arithmetic the SDK does for you: `osr capacity show --memory 28Gi --cores 4` prints a
model's footprint, whether it fits and the expected tokens/s; `osr capacity plan --memory 28Gi`
packs a set onto hosts; `osr capacity ps --url http://:11434` lists what the server holds right now.
## With the platform
The platform treats a model server as an `ollama` provider: its base URL (`http://:11434/v1`) in
`OSR_PLATFORM_PROVIDERS` or the operator console, targets mapped to model names, `concurrency` equal to the
slots the server offers. The provider check reads `/api/ps` for residency, the keep-warm pass loads mapped models
that are not resident, and the calibrator turns the server's measured latency into what the router ranks on.
The Docker Compose stacks run the model server as the `llm` profile; the Azure template's self-hosted model
apps run it when `OSR_OLLAMA_IMAGE` names the model server image (the Ollama image stays the default) -
same volume, same start-up, same probes.
## Operations
- `GET /healthz` - engine version, device, instances, memory budget and use, store usage, preload progress and
errors; `GET /readyz` - `200` once the preload list is pulled (and loaded when asked), `503` before.
- `GET /metrics` - `osr_models_requests_total`, `osr_models_request_seconds`, `osr_models_tokens_total`,
`osr_models_slots_busy`, `osr_models_queue`, `osr_models_resident_bytes`, `osr_models_loads_total`,
`osr_models_evictions_total`, `osr_models_refused_total`, `osr_models_load_seconds`, and the engine's
`llamacpp:*` series labelled by model.
- `GET /models` - every model in the store with its status (`unloaded`, `loading`, `loaded`), footprint and the
engine arguments; `GET /models//logs` - the engine's last log lines when a load fails.
- A load that does not fit answers `507` with the budget and what to change (shorter context, `q8_0` KV cache,
a smaller quantisation). A saturated model answers `503` with `Retry-After`.
- Prompts and answers are never written to disk; the engine processes bind to the loopback only.
---
# @opensmartroute/sdk
TypeScript client for [OpenSmartRoute](https://opensmartroute.ai): the hosted platform
(`https://opensmartroute.ai`) and a self-hosted `osr serve`. Zero runtime dependencies - it uses
the global `fetch`, so it runs on Node 20+, Deno, Bun, Cloudflare Workers, Vercel Edge and in browsers.
```bash
npm install @opensmartroute/sdk # pnpm add / yarn add / bun add work the same
```
The package version follows the OpenSmartRoute release (`2.1.0` today): every release publishes it to the
npm registry, and the types under `src/generated` are generated from that release's OpenAPI document.
ESM and CommonJS entry points with bundled `.d.ts`; `import { OpenSmartRoute } from "@opensmartroute/sdk"`
or `const { OpenSmartRoute } = require("@opensmartroute/sdk")`.
## Quick start: route, then report
Create an API key at [opensmartroute.ai/dashboard/keys](https://opensmartroute.ai/dashboard/keys) (free plan,
no card) or point `baseUrl` at your own `osr serve`. Only `estimate()` and `info()` work without a key.
```ts
import { OpenSmartRoute } from "@opensmartroute/sdk";
const osr = new OpenSmartRoute({
baseUrl: "https://opensmartroute.ai",
apiKey: process.env.OSR_API_KEY!,
});
const decision = await osr.route({
text: "Refund order #123, the customer is upset",
objective: { cost: 0.4, quality: 0.6 },
constraints: { data_boundary: "private" },
top_k: 3,
});
console.log(decision.target.id, decision.confidence, decision.explanation);
// ...call the chosen target, then close the loop so the learners improve on your traffic
await osr.feedback({
request_id: decision.request_id,
target_id: decision.target.id,
success: true,
quality: 0.9,
latency_ms: 850,
});
```
`route({ ..., execute: true })` also runs the target on the platform and returns the answer in
`decision.result` (`text`, `provider`, `cost_usd`, `fallback_from`). `plan: true` composes a
persona -> skill -> model plan, plus the tools the model may call (`decision.plan`). `estimate(req)` quotes a request before sending it;
`targets()` lists the catalogue; `me()`, `quota()` and `info()` describe the account, where it stands against its
monthly request quota, and the deployment.
The short forms take the same loop down to a line each - the text first, the other fields second, per-request
options third:
```ts
const d = await osr.route("Refund order #123, the customer is upset", { objective: { cost: 0.4 } });
await osr.feedback(d, { success: true, quality: 0.9 }); // request_id and target_id read from the decision
const answer = (await osr.run("Summarise this ticket: ...")).result?.text; // route + execute in one call
const text = await osr.ask("What is the capital of France?", { system: "One word." }); // chat with model "auto"
```
## OpenAI-compatible chat, streaming and embeddings
The same client speaks the platform's `/v1` surface. `model: "auto"` (the default) lets the router
choose; a target id pins one; `models: [...]` gives the router a candidate list with fallbacks.
Every answer carries the routing decision in `opensmartroute` (typed as `OsrMetadata`).
```ts
const reply = await osr.chat.completions.create({
model: "auto",
messages: [{ role: "user", content: "Write a haiku about routing." }],
osr: { objective: { cost: 0.7 }, fallbacks: true },
});
console.log(reply.model, reply.choices[0].message.content);
console.log(reply.opensmartroute.target, reply.opensmartroute.confidence, reply.opensmartroute.cost_usd);
// streaming: an AsyncIterable of chat.completion.chunk objects, parsed from the SSE stream
const stream = await osr.chat.completions.create({
messages: [{ role: "user", content: "Explain LinUCB in two sentences." }],
stream: true,
});
console.log("answering:", stream.target); // known from the X-OSR-Target header before the first token
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta.content ?? "");
if (chunk.opensmartroute) console.log("\n", chunk.usage, chunk.opensmartroute); // the last chunk
}
// or: const completion = await stream.finalCompletion();
const vectors = await osr.embeddings.create({ input: ["first text", "second text"] });
console.log(vectors.data[0].embedding.length, vectors.opensmartroute.target);
const models = await osr.models.list(); // "auto", "osr/auto:cheap", ... and every executable target
```
## Responses API, Messages API, speech, transcription, images
The OpenAI **Responses API** (what the OpenAI Agents SDK speaks) and the Anthropic **Messages API** (what the
Anthropic SDKs and Claude Code speak) run over the same routed path; media endpoints route among the targets
that can do the job:
```ts
const r = await osr.responses.create({ input: "Plan a 3-day trip to Oslo", instructions: "Be concise." });
console.log(responseText(r), r.opensmartroute?.target);
const s = await osr.responses.create({ input: "Explain LinUCB", stream: true });
for await (const ev of s) if (ev.type === "response.output_text.delta") process.stdout.write(ev.delta!);
const final = await s.finalResponse();
const m = await osr.messages.create({ max_tokens: 256, messages: [{ role: "user", content: "Explain LinUCB" }] });
console.log(messageText(m), m.stop_reason);
const ms = await osr.messages.create({ max_tokens: 256, messages: [{ role: "user", content: "hi" }], stream: true });
const done = await ms.finalMessage(); // text and tool_use blocks assembled from the events
const { input_tokens } = await osr.messages.countTokens({ messages: [{ role: "user", content: "hi" }] });
const speech = await osr.audio.speech.create({ input: "Hello there", response_format: "mp3" }); // bytes + headers
const text = await osr.audio.transcriptions.create({ file: audioBlob, filename: "memo.wav" });
const pic = await osr.images.generate({ prompt: "a lighthouse at dawn", size: "1024x1024" });
```
## Workspace management (platform)
Every route of the platform's OpenAPI document is reachable, typed from the generated schema, through
`osr.rest` - paths, path / query parameters, bodies and responses are checked by `tsc`:
```ts
const keys = await osr.rest.get("/api/v1/keys");
await osr.rest.post("/api/v1/keys", { body: { name: "ci", scopes: ["route", "feedback"], expires_in_days: 90 } });
await osr.rest.del("/api/v1/keys/{key_id}", { path: { key_id: keys[0].id } });
```
Convenience groups sit on top of it: `osr.keys` (list / create / rename / rotate / revoke), `osr.tenants`,
`osr.governance` (policy), `osr.providers` (bring your own model providers), `osr.notifications`,
`osr.support`, `osr.handoffs`, `osr.billing`, `osr.marketplace`, `osr.insights` (usage, activity, savings,
audit trail, statistics, rankings, SLA), `osr.memory` (what the routed chat remembers, the conversations) and
`osr.knowledge` (the workspace's knowledge graph: facts, "forget that ...", the context preview, suggestions,
export, training).
## Against `osr serve`
`osr serve` mounts the same routes at the root (no `/api/v1` prefix) and takes a bearer token:
```bash
osr -t targets.yaml -r rules.yaml serve --port 8000 --token my-secret-token
```
```ts
const local = new OpenSmartRoute({
baseUrl: "http://localhost:8000",
apiKey: "my-secret-token",
apiPrefix: "", // osr serve: /route, /feedback, /targets; /v1/... is the same
});
const who = await local.whoami(); // { edition: "self-hosted", authenticated: true, ... }
const d = await local.route({ text: "Summarise this ticket" });
```
## Options and headers
| Option | Header / effect |
|---|---|
| `apiKey` | `Authorization: Bearer ` (platform API key, or an `osr serve` token) |
| `tenant` / `opts.tenant` | `X-OSR-Tenant` - the tenant's constraints and budget apply |
| `app` | `X-OSR-App` - attribution echoed in the decision metadata |
| `opts.requestId` | `X-Request-Id` - your own id, echoed back and usable for `feedback` |
| `headers`, `opts.headers` | extra headers for every / this request |
| `timeoutMs`, `opts.timeoutMs` | abort when no headers arrive in time (streams keep flowing) |
| `maxRetries` | retries after a connection failure, a 408, a 429 or a 502-504 (default 2; `Retry-After` honoured up to a minute, otherwise backoff from 0.5 s to 8 s with jitter); a final 4xx is never retried, a stream only before its first byte |
| `allowInsecureHttp` | send the key over plain `http://` to a public host; off by default - a key only travels in clear text to loopback and private addresses and to names that resolve inside a network (`router`, `osr.internal`, `*.svc`) |
| `opts.signal` | your `AbortSignal` (also cancels a stream and a pending retry) |
| `fetch` | a custom `fetch` (tests, proxies, instrumentation) |
`requestWithResponse(method, path, body)` returns the decoded body **and** the `Response`, for
headers such as `X-OSR-Target`, `X-OSR-Trace-Id`, `ETag` or the quota headers.
## Errors
Every non-2xx answer raises `OpenSmartRouteError` with `status`, `detail` (the server's message),
`requestId` (`X-Request-Id` / `X-OSR-Request-Id`), `retryAfter` (seconds, from `Retry-After`),
`headers` and a stable `code`:
| Class | Status | `code` |
|---|---|---|
| `AuthenticationError` | 401 | `authentication` |
| `PermissionDeniedError` (`upgradeRequired` when `X-Upgrade: true`) | 403 | `permission` |
| `NotFoundError` | 404 | `not_found` |
| `RateLimitError` (`budget` from the `X-Budget-*` headers) | 429 | `rate_limit` |
| `OpenSmartRouteError` | 400 / 413 / 422 | `validation` |
| `OpenSmartRouteError` | 501 / 502-504 / other 5xx | `not_implemented` / `upstream` / `server` |
| `ConnectionError` | - (no response) | `network` |
```ts
import { RateLimitError } from "@opensmartroute/sdk";
try {
await osr.route({ text: "..." }); // already retried twice with backoff when the failure was transient
} catch (e) {
if (e instanceof RateLimitError) await sleep((e.retryAfter ?? 1) * 1000);
else throw e;
}
```
No error message, `detail` or `body` ever contains the key you sent.
## Types
`RouteIn`, `FeedbackIn`, `EstimateIn`, `ChatCompletionIn`, `EmbeddingsIn` and `RoutingOptions` are
generated from the platform's OpenAPI document (`src/generated/openapi.ts`, also exported as
`OpenAPIPaths` / `OpenAPIComponents` for `openapi-fetch` users). `RouteDecision`, `OsrMetadata`,
`ChatCompletion`, `ChatCompletionChunk`, `EmbeddingsResponse`, `ModelList`, `TargetInfo`, `Estimate`
and `Me` describe the answers.
## Development
```bash
npm ci
npm run generate # regenerate src/generated/openapi.ts from ../api/openapi.json (commit the result)
npm run check # eslint + tsc + vitest
npm run build # dist/ (ESM + CJS + .d.ts) with tsup
```
The live test runs only when `OSR_SDK_BASE_URL` and `OSR_SDK_API_KEY` are set
(`OSR_SDK_API_PREFIX` forces `/api/v1` or an empty prefix; without it the test probes `GET /whoami` to
recognise `osr serve`). Against a local server (`pip install "opensmartroute[server,yaml]"`):
```powershell
osr -t targets.yaml -r rules.yaml serve --port 8765 --token sdk-test-token
# in another shell
$env:OSR_SDK_BASE_URL = "http://127.0.0.1:8765"; $env:OSR_SDK_API_KEY = "sdk-test-token"
npm test --prefix platform/sdk-ts
```
Without a model provider configured the chat and embeddings calls answer with a well-formed error
(the test accepts either outcome); routing, feedback, targets and models are asserted fully.
## License
Apache-2.0 - see LICENSE (`platform/sdk-ts/LICENSE`).
---
# Marketplace
The marketplace is the part of the [hosted platform](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md) where routing building blocks are
shared: **agents, skills, personas, prompts, tools, model profiles and stack templates**, with
versions, ratings and a short review lifecycle. Every listing is a plain YAML or JSON document that
the OpenSmartRoute SDK, CLI and hosted API understand, so what you install is what you run; there
is no marketplace-only format.
Browsing, installing, rating and publishing prompts and personas work from the website without
writing code. Templates, skills and agents come with copy-paste snippets for the CLI and the SDK.
## What is on it
| Kind | What it is | Manifest |
|---|---|---|
| **Template** (stack) | A complete routing setup - models, rules, weights - in one file. Apply it and you have a working router. | `kind: stack` document ([`opensmartroute.stack`](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/REFERENCE.md#opensmartroutestack)) |
| **Agent** | A multi-step worker the router can hand a whole task to. | Open Capability Manifest ([spec](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ocm.md)) |
| **Skill** | An Agent-Skills package (`SKILL.md`) that is also a routable target. | OCM, generated from the `SKILL.md` |
| **Persona** | A system prompt with a voice, a domain and a complexity band. | OCM with `kind: persona` |
| **Tool** | A function or MCP tool with typed parameters. | OCM with `kind: tool` |
| **Model profile** (llm) | A model with cost, latency, context window and quality prior. | OCM with `kind: llm` |
| **Prompt** | A reusable prompt with `{{variables}}`. | `{"prompt": "...", "variables": [...]}` |
Every listing has a **slug** (its address, `/marketplace/`), a **version** (`1.2.3`), a
publisher, a licence, tags, a Markdown description, an install count and a star rating.
## Finding and installing
1. Open **Marketplace** in the top navigation. Filter by kind, tag, price or search; sort by
popular, top rated, newest or name.
2. Open a listing. The **About** tab is the publisher's description; **Use it** shows ready-made
snippets (CLI, Python, HTTP); **Manifest** is the raw document; **Reviews** lists ratings.
3. Click **Install** (free) or **Buy** (paid). You need an account - signing up is free. Free
installs complete at once; paid ones go through the checkout and are recorded on your account
when the payment completes.
4. Everything you installed is listed under **Dashboard -> Marketplace -> Installed by you**, together with
the version you installed and whether a newer one exists.
Installing records the listing on your account and returns the manifest and snippets. Nothing is
executed on your behalf.
### Using an installed template
```bash
osr stack apply registry://support-desk@1.0.0 # build and summarise (hosted marketplace)
osr stack apply registry://support-desk --route "Refund for order 42?"
osr stack apply registry://support-desk --out stack.yaml # flatten & keep
osr stack apply registry://support-desk --registry https:// # a self-hosted marketplace
```
`registry://` references resolve against the marketplace the CLI is configured for (`OSR_MARKETPLACE_URL`,
by default the hosted platform); `--registry` points one command at another deployment.
Or reference it from your own stack and let imports do the work:
```yaml
osr: "1"
kind: stack
name: my-desk
imports:
- registry://support-desk@1.0.0
targets:
- id: llm-onprem
kind: llm
```
`osr stack validate my-desk.yaml`, `osr stack plan my-desk.yaml --against deployed.yaml` and
`osr stack apply my-desk.yaml` are the Terraform-style verbs: check, diff, apply.
### Using an installed agent, skill, persona, tool or model
**On the hosted platform nothing more is needed.** Installing an agent, skill, persona, tool, subagent or
command puts it into your workspace's routing pool: from the next request on, `POST /api/v1/route` with
`plan: true` (and the OpenAI-compatible endpoints with plan composition) picks the installed skill or persona
for the matching slot when it fits the request better than the catalogue's, and an installed primary competes
for the answer itself. `GET /api/v1/targets` lists your installs with `source: marketplace`; `plan_mode`
(`sequential` / `parallel`) decides whether several installed skills run one after another or at the same time
([platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md#2-route-a-request)). Other workspaces never see your installs. Try it in the
playground: install a skill, ask a matching question with *Build a plan* on, and the plan's *prepare* stage
names it. An install is seen by every API replica on the very next request - no waiting for a cache.
**Skills, personas and prompts you never installed can help too.** Every plan also offers the router the
published skills, personas and prompts whose listing matches the request - the same vocabulary the router
extracts (its domains and salient words) is searched in the listing's name, summary, tags and domains, and
only a listing that works in one of the request's domains qualifies - so the marketplace shapes an answer
without anyone browsing it first. A prompt joins as an instructions-only skill when it takes its input through
at most one `{{variable}}` (the placeholder stands for the user's message); one with several variables is a
template for a person to fill. The router's own strategies and quality floor still decide whether such a
companion is used; a discovered one never answers on its own, and the decision names it under `marketplace`
with `installed: false`. What may be discovered is a matter of trust: by default only listings an operator has
marked **verified** (the deployment's own seeded skills, personas and prompts are), because a discovered
skill's instructions reach the prompt of a workspace that did not choose it. A workspace widens or narrows that
with the `marketplace_discovery` policy key - `verified` (default), `all` (every published listing) or `off`
(installs only) - and the operator sets the deployment default with `OSR_PLATFORM_MARKETPLACE_DISCOVERY`.
Agents and tools are never discovered: they *execute* (a call to the publisher's endpoint with the request in
it), so they join the pool through an install only.
Installed agents and tools do not only get picked, they **run**: when a listing's manifest declares a public
`https` endpoint with protocol `a2a` (an agent card), `http` (a JSON agent or workflow endpoint) or `mcp` (a
Streamable-HTTP MCP server), the platform binds it - an executed request that lands on the listing calls the
agent, or lists the MCP server's tools once and runs the one the request names (`context.tool`), the only one
there is, or the closest match, with `context.tool_arguments` as its arguments. `GET /api/v1/targets` shows
`executable: true` for those; a listing whose endpoint the platform may not call (a private address, a
protocol that needs credentials such as `openai` or `queue`) stays routable but not executable and carries
the reason in `metadata.endpoint_unbound`, which the dashboard's *Installed* tab shows as a badge. Skills and
personas need no endpoint: their instructions run through the model that answers. An installed **model card**
(an `llm` listing) needs your own key: map it to one of your providers on the *Providers* page - it is offered
there next to the catalogue with the listing's upstream model id prefilled - and it runs on your account like
any other target; `/v1/models` lists it and a chat may name it (`model: `).
In your own code an OCM manifest becomes a target with one call:
```python
from opensmartroute import Router
from opensmartroute.ocm import bind_endpoint, target_from_capability
manifest = {...} # GET /api/v1/registry//manifest
router = Router([bind_endpoint(target_from_capability(manifest))]) # http / a2a / mcp endpoints get a handler
```
Prompts are text: read `manifest["prompt"]` and fill in `manifest["variables"]`.
The package has a client for the whole loop - search, load, keep in sync - so a router follows the
marketplace (which itself refreshes from its public sources every day) without manual downloads:
```python
from opensmartroute import Router, TargetRegistry
from opensmartroute.adapters import MarketplaceClient
mk = MarketplaceClient() # https://api.opensmartroute.ai
skills = mk.targets("sql reporting", kind="skill", limit=5) # RouteTargets from the listings' manifests
agents = mk.targets(kind="agent", tag="a2a", limit=10) # A2A agent cards, endpoints attached
router = Router(TargetRegistry([*my_models, *skills, *agents]))
report = mk.sync(router.registry, kind="skill", tag="skill-md", limit=50) # run daily: adds new, updates changed
print(report.added, report.updated)
```
`MarketplaceClient.search` returns `Listing` rows (slug, kind, version, installs, rating, source);
`target(slug)` loads one manifest; `sync` upserts by id and replaces a target when its version, text,
instructions or endpoints changed upstream. Only routable kinds (agent, skill, persona, tool, llm) become
targets; templates are stacks - import them with `registry://slug@version`.
`sync` is safe to run forever: the `SyncReport` lists `added`, `updated`, `skipped` (no loadable manifest),
`removed`, `kept` and `seen`. A followed listing the marketplace answers 404 / 410 for is **removed** from the
registry; a synced target you edited locally (the sync stamps a fingerprint in `metadata["marketplace"]`) is
**kept** as it is until you pass `force=True`; `prune=True` also removes the targets this marketplace delivered
earlier that no longer match the search (`OSR_MARKETPLACE_PRUNE=1` does the same for the server-side
`MarketplaceSync` below). The same from the CLI:
```bash
osr marketplace search "sql reporting" --kind skill # list matching listings
osr marketplace pull acme-skills-sql-reporting --out skills.json # manifests as an OCM catalogue file
osr marketplace sync --kind skill --tag skill-md --limit 50 --targets-file marketplace.json # daily cron job
osr --targets marketplace.json route "Write the monthly revenue report" --plan
```
`sync --targets-file` creates or updates a catalogue file every routing command loads with `--targets`,
so a scheduled `osr marketplace sync` keeps a self-hosted router on the marketplace's daily state. The file
keeps each manifest's endpoints, and `load_targets` binds them again on the next start (`http`, `a2a`, `mcp`),
so an agent or MCP tool the router picks from that catalogue *runs* - the process restart loses nothing.
### Following the marketplace from `osr serve`
A running server can follow the marketplace by itself - no cron, no restart: `MarketplaceSync` upserts the
followed listings into the served catalogue every `OSR_MARKETPLACE_REFRESH_S` seconds, so a skill published
(or re-versioned) upstream is routable on the next tick.
```bash
osr -t targets.yaml serve --marketplace-refresh 86400 # once a day
OSR_MARKETPLACE_REFRESH_S=86400 OSR_MARKETPLACE_KINDS=skill,tool osr -t targets.yaml serve
OSR_MARKETPLACE_QUERY="sql reporting" OSR_MARKETPLACE_SLUGS=acme-skills-sql-reporting ...
```
`OSR_MARKETPLACE_URL`, `OSR_MARKETPLACE_KINDS` (default `skill,tool`), `OSR_MARKETPLACE_QUERY`,
`OSR_MARKETPLACE_TAG`, `OSR_MARKETPLACE_LIMIT` (per kind) and `OSR_MARKETPLACE_SLUGS` choose what is
followed (`MarketplaceSettings`). `GET /stats` reports the sync (`marketplace`: last run, added / updated,
last error) and `POST /marketplace/sync` runs one now (a write path; read-only tokens are refused). In
Python, `MarketplaceSync(router.registry).start()` does the same in any process.
### From the IDE (MCP)
The local MCP server (`osr -t targets.yaml mcp`) carries `marketplace_search` and `marketplace_get` backed by
the marketplace in `OSR_MARKETPLACE_URL` (`--marketplace-url` to point elsewhere, `--no-marketplace` to leave
them out). `marketplace_get` with `install: true` also loads the listing into the running router's catalogue,
so an assistant can find a skill and route to it in the same conversation. To try a listing without installing
it, `route` takes `marketplace: [slug, ...]`: those listings join the candidates for that one request (and fill
the plan's slots) and the result names them under `marketplace_candidates`. When something in the catalogue can
run - a model behind `OSR_LLM_BASE_URL`, an agent or MCP server with a bound endpoint - the local server also
offers `ask`, which routes and runs in one call.
The HTTP endpoints are public for published listings:
```
GET /api/v1/registry?kind=template&q=support&sort=popular
GET /api/v1/registry/{slug}
GET /api/v1/registry/{slug}/manifest?version=1.0.0
GET /api/v1/registry/{slug}/reviews
```
`GET /api/v1/registry` filters with `q` (free text, 80 characters), `kind`, `tag` (one exact tag, 64
characters), `price=free|paid`, `featured=true|false` and orders with `sort=popular|newest|name|rating`;
`q` and `tag` are matched literally. Pages are `limit` (1-100) and `offset` (at most 10 000); the whole
catalogue is paged through `GET /api/v1/registry/sitemap` instead. Each page carries the kind counts and
top tags of the catalogue, refreshed every few minutes.
## Ratings and reviews
Anyone with an account can rate a listing from 1 to 5 stars and leave a short review. One review per
account per listing (writing again replaces the previous one). Reviews from accounts that installed
the listing are marked as such. A listing shows its average rating, the number of ratings and a star
histogram.
### Leaderboard
`/rankings/marketplace` ranks the published listings per kind - most installed, best rated or newest -
from `GET /api/v1/rankings/listings?kind=&sort=installs|rating|newest`. *Best rated* is a
Bayesian average: the mean rating shrunk towards 3.5 by the weight of five ratings, so a listing with
one five-star review sits below one with forty four-star reviews until more ratings arrive.
`GET /api/v1/rankings/listings/kinds` gives the published listings, installs and reviews per kind.
Publishers appear by display name only.
## Publishing
**Marketplace -> Publish** walks through four steps: choose the kind, describe it, paste or write
the manifest, review and submit. Prompts and personas need no code at all; for templates the wizard
offers a starter you can edit. You can also post the same payload to `POST /api/v1/registry`.
### The lifecycle
```
draft -> review -> published -> archived
^ |
+--- rejected
```
- **Draft** - only you can see it. Edit anything, including the manifest.
- **Submit** - the manifest is validated (OCM schema, stack schema or prompt shape). Free listings
are published immediately; paid listings go to **review**. (Operators can require review for
everything with `OSR_PLATFORM_REGISTRY_AUTO_PUBLISH=false`.)
- **Review** - a platform reviewer approves or rejects with a note. Rejected listings go back to
you with the note; fix and submit again.
- **Published** - live. Name, summary, description, tags, price and licence stay editable. The
manifest is frozen: ship a **new version** instead, with a changelog. Old versions remain
downloadable (`?version=`).
- **Archived** - hidden from the marketplace; existing installs keep working. You can restore it as
a draft.
Your listings, their status and any reviewer note are under **Dashboard -> Marketplace -> Published
by you**; each one has an edit page with the version form.
### Paid listings
Set a price in USD when publishing (paid plans, Pro and above). Buyers pay through the platform's
checkout; the purchase is confirmed by the payment webhook and the listing is then recorded on their
account. Free plans can publish a limited number of free listings.
### Writing a good listing
- A one-line summary that says what it does and for whom.
- A description that covers what it needs (models, keys, data) and how to use it - Markdown, no
raw HTML or images.
- Tags people would search for (`support`, `pii`, `sql`, `on-prem`).
- Test templates locally first: `osr stack validate stack.yaml`.
- Skills: keep `SKILL.md` front matter complete (`name`, `description`); `osr skills validate`
checks it.
## Built-in content
Every platform ships seeded with the repository's own Agent-Skills packages, a starter template, the
`examples/stack.yaml` support-desk template, a persona and a couple of prompts, all published by
`OpenSmartRoute` and marked verified. Set `OSR_PLATFORM_REGISTRY_SEED=false` to run an empty
marketplace.
## Imported catalogue
The hosted marketplace also carries the public agentic ecosystem - Agent Skills, agent instructions, prompt
files, slash commands and workflows, chat modes, act-as roles and Fabric patterns, custom agents and subagents,
A2A agent cards, stack files, MCP servers, the reference LLM catalogue and routing stacks generated from it -
harvested and refreshed every day by
the platform itself, so **every** marketplace kind fills from public sources:
| Source | What is imported | Listing kind |
|---|---|---|
| [skills.sh](https://skills.sh) | every `SKILL.md` in each repository on the leaderboard (about 2 400 repositories) | `skill` |
| GitHub code search | the long tail of `SKILL.md` files (needs a GitHub token; optional) | `skill` |
| GitHub code search (`github-agents`) | every public `AGENTS.md` file - the cross-tool agent-instruction standard (needs a GitHub token; optional) | `skill` |
| GitHub code search (`github-instructions`, `github-cursor-rules`) | `.github/instructions/*.instructions.md` (Copilot custom instructions, `applyTo` kept as `metadata.applies_to`) and `.cursor/rules/*.mdc` (Cursor project rules, `globs`) (needs a GitHub token; optional) | `skill` |
| GitHub code search (`github-prompts`) | `.github/prompts/*.prompt.md` reusable prompt files: the frontmatter's `description`, `mode`, `model` and `tools`, the body as the prompt with `${input:name}` / `${selection}` turned into `{{name}}` placeholders declared as `variables` (needs a GitHub token; optional) | `prompt` |
| GitHub code search (`github-claude-commands`) | `.claude/commands/**/*.md` Claude Code slash commands: `description`, `argument-hint`, `allowed-tools`, `model`; `$ARGUMENTS` becomes `{{arguments}}` and `$1` ... `$9` become `{{arg1}}` ... (needs a GitHub token; optional) | `prompt` |
| GitHub code search (`github-cursor-commands`, `github-windsurf-workflows`, `github-codex-prompts`) | `.cursor/commands/*.md` (Cursor custom commands), `.windsurf/workflows/*.md` (Windsurf workflows) and `.codex/prompts/*.md` (Codex custom prompts), same conversion; slugs end in `-cursor-command`, `-workflow`, `-codex-prompt` (needs a GitHub token; optional) | `prompt` |
| GitHub code search (`github-chatmodes`) | `.github/chatmodes/*.chatmode.md` - a voice, rules and tool set layered on a model (needs a GitHub token; optional) | `persona` |
| Awesome ChatGPT Prompts (`awesome-prompts`) | every "Act as ..." role of the CC0 `prompts.csv` (the prompt is the persona's instructions, the first sentence its description); slugs `persona-` | `persona` |
| Fabric (`fabric-patterns`) | every pattern's `system.md` (identity and purpose, steps, output format; MIT) from one repository tarball; slugs `fabric-` | `persona` |
| GitHub code search (`github-custom-agents`) | `.github/agents/*.agent.md` custom agent definitions: instructions, `tools`, `model`, `handoffs` (needs a GitHub token; optional) | `agent` |
| GitHub code search (`github-claude-agents`) | `.claude/agents/*.md` Claude Code subagents: `name`, `description`, comma-separated `tools`, `model`; slugs `owner-repo--subagent` (needs a GitHub token; optional) | `agent` |
| GitHub code search (`github-a2a`) | public A2A agent cards (`.well-known/agent.json`): name, skills, capabilities and the `a2a` endpoint the router can call (needs a GitHub token; optional) | `agent` |
| GitHub code search (`github-stacks`) | OpenSmartRoute stack files anywhere on GitHub (`osr: "1"`, `kind: stack`, at least one target or import; Docker / Pulumi stacks and invalid documents are dropped by the stack validator); the manifest *is* the stack, so `registry://@` imports it; slugs `stack-owner-repo-` (needs a GitHub token; optional) | `template` |
| Generated from the model catalogue (`catalogue-stacks`) | ready-made routing stacks the platform derives from the reference LLM catalogue: one *tiers* stack per vendor with at least three priced models (cheapest / middle / strongest by intelligence index, complexity rules, cost-aware objective) and themed stacks - free models, reasoning, vision, open weights (up to four models each). The targets are the catalogue's `llm` capabilities with their OpenRouter endpoint, so a stack installs with no mapping; regenerated as the line-up and prices change; slugs `stack-` / `stack-free-models` ... | `template` |
| [MCP registry](https://registry.modelcontextprotocol.io) | the latest active version of every server, with its remote endpoints and packages | `tool` |
| Reference model catalogue (`models`) | every hosted LLM of the platform's model catalogue (the OpenRouter listing snapshot, `/models.json` when the rankings loop has refreshed it): vendor, prices per million tokens (`cost.usd_per_1k_tokens` is the input/output average), context window, modalities, tool calling, open weights, the OpenAI-compatible endpoint `https://openrouter.ai/api/v1` and a quality prior from the Artificial Analysis intelligence index when published | `llm` |
Imported listings are published under the upstream owner's name, are not marked verified, and carry
`metadata.source` in their manifest (`provider`, `repository`, `path`, `ref`, `url`) so the listing page
links back to the original file; the readme starts with the upstream install command
(`npx skills add owner/repo --skill name`). Copyright and licence stay with the author (`license` is
taken from the frontmatter when present). Domains are inferred from the description with the router's
own lexicon; the `skill-md` / `mcp` tags and the source name are added to the author's tags.
The harvest runs inside the platform on its own schedule (`OSR_PLATFORM_MARKETPLACE_REFRESH_S`; operators can
run it now from the console): it only fetches what the catalogue does not have yet, re-checks stale rows, and
publishes each source's new listings as soon as that source finishes. Slugs are `owner-repo-skill`
(`mcp-` for tools,
`model--` for models, `owner-repo--` such as `acme-app-deploy-prompt` for
prompt / chat-mode / agent / instruction files, so two conventions sharing a file name stay distinct),
truncated with a short hash when longer than 63 characters.
Imported text is screened before it is stored: control characters are dropped and credential-shaped
strings an author left in a file (AWS access keys, GitHub / OpenAI / Slack / Stripe / Google API keys,
platform keys, JWTs, `PRIVATE KEY` blocks) are replaced with `[REDACTED ]`, including documentation
placeholders - the original stays one click away upstream. Repositories that publish attack corpora
(poisoned, malicious or prompt-injection skill datasets) are never mirrored: the harvester skips them, the
cache replay drops rows cached earlier, and the admin import endpoint rejects them
(`source: repository publishes attack corpora and is not mirrored`).
For a platform whose SQLite file lives on a network share (the Azure Container Apps deployment mounts
Azure Files), publish tens of thousands of rows **offline**: build the database locally with
`publish --data-dir`, stop the API, replace `platform.sqlite3` on the share, start it again. Sustained
bulk writes through the HTTP endpoint over SMB have corrupted the listings table before; the endpoint is
meant for incremental updates of a few hundred rows.
The hosted marketplace stays current from inside the platform process - no CI job or cron is involved.
`OSR_PLATFORM_MARKETPLACE_REFRESH_S` is the interval (default `86400` = **daily**; `0` = only from the
console) and the process that owns the marketplace runs every source in
`OSR_PLATFORM_MARKETPLACE_REFRESH_SOURCES` (default: every source -
`skills.sh,mcp-registry,models,catalogue-stacks,awesome-prompts,fabric-patterns,github,github-agents,github-instructions,github-cursor-rules,github-prompts,github-chatmodes,github-custom-agents,github-claude-agents,github-claude-commands,github-cursor-commands,github-windsurf-workflows,github-codex-prompts,github-a2a,github-stacks`;
`OSR_PLATFORM_MARKETPLACE_REFRESH_LIMIT` new rows per source per run, default 2000) into
`/harvest/`, publishes the listings the catalogue lacks in batches of 100, and re-fetches the
oldest cached rows older than `OSR_PLATFORM_MARKETPLACE_STALE_DAYS` (default 7; at most
`OSR_PLATFORM_MARKETPLACE_STALE_LIMIT` = 500 per source per run) so listings whose upstream file changed are
republished with the new text, version and endpoints. After harvesting, each cache file is compacted to the
newest line per key (re-fetched rows are appended, so the files would otherwise grow without bound), and the
publish step reads bodies back from the cache one batch at a time - a catalogue of a hundred thousand rows
costs the process tens of megabytes, not the size of the caches. Every run publishes what the caches already
hold *before* it harvests and again after each source, so a run interrupted by a deploy or an eviction loses
nothing: the next run puts out
the rows it had fetched and carries on. The first run after a (re)start waits five minutes, or until the
interval has passed since the last run finished; a run that started and never finished is resumed after five
minutes, then three times longer for every further interruption in a row, capped at **one hour** (a crash loop
backs off to hourly attempts; a deploy costs five minutes, and a service deployed several times a day still
completes its harvest). When the marketplace runs on several replicas, one of them takes a Redis lease
(`osr:marketplace:leader`) per run and the others skip it, so the shared cache files are written by one process
at a time. The lease lives ten minutes and the leader renews it while it works and releases it when done, so a
leader killed mid-run (a deploy, an eviction) frees the slot within ten minutes; a replica that found the lease
taken records the skip (`refresh.last_skipped` in the status) and looks again after fifteen minutes instead of
waiting a whole interval. The console's *Refresh now* bypasses the lease. Sources that need GitHub code search are skipped -
and reported as such - until `OSR_PLATFORM_GITHUB_TOKEN` is set (on Azure: the `OSR_PLATFORM_GITHUB_TOKEN`
deployment input becomes the `github-token` secret of every API app; a token with the `public_repo` scope is
enough). `GET /api/v1/admin/registry/import`
reports the schedule, the sources, the last run (new, updated and invalid rows per source) and the next run
time under `refresh`; `POST /api/v1/admin/registry/import/refresh` runs it now.
From a machine, `publish --url ... --only-new` does the same: it asks
`POST /api/v1/admin/registry/import/check {"slugs": [...]}` which slugs already exist and sends only the rest.
## For operators
| Setting | Default | Effect |
|---|---|---|
| `OSR_PLATFORM_REGISTRY_AUTO_PUBLISH` | `true` | Free listings go live on submit; `false` sends everything to review. |
| `OSR_PLATFORM_REGISTRY_SEED` | `true` | Seed built-in skills, templates, personas and prompts at startup. |
Moderation uses the admin token (`X-Admin-Token`):
```
GET /api/v1/admin/registry?status=review
POST /api/v1/admin/registry/{slug}/approve {"note": "..."}
POST /api/v1/admin/registry/{slug}/reject {"note": "why"}
POST /api/v1/admin/registry/{slug}/flags {"featured": true, "verified": true}
POST /api/v1/admin/registry/{slug}/unpublish {"note": "..."}
GET /api/v1/admin/registry/import published listings per source
POST /api/v1/admin/registry/import {"items": [...], "refresh": false} (up to 500 harvested rows)
POST /api/v1/admin/registry/import/check {"slugs": [...]} -> {"existing": [...]} (up to 2000 slugs)
POST /api/v1/admin/registry/import/refresh run the scheduled harvest now (202; 409 while running)
```
Paid purchases need billing to be configured (Stripe keys); without it `POST .../install` on a paid
listing answers `402`.
## Security notes
- Manifests are validated against the OCM / stack schemas and size-limited; descriptions are
rendered without raw HTML, scripts or images and external links carry `rel="nofollow"`.
- Installing never runs anything: it returns a document. Review what a template or agent points at
(endpoints, tools) before applying it, the same way you would review a Terraform module.
- Slugs are stable; a new version never changes the slug, so pin `@version` in production stacks.
## Related
* [Platform guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md) - authentication, routing, plans, organizations and the dashboard.
* [REST API reference](/docs/api/marketplace) - the *Marketplace* group lists every endpoint.
* [User guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md) - stack files, `osr stack` and the Open Capability Manifest.
---
# Cost estimates, recommended models and the MCP server
Three ways to ask the router "what will this cost and what should I use?" before spending tokens:
a quote API and page, a recommended-models digest, and a [Model Context Protocol](https://modelcontextprotocol.io)
server that puts all of it inside VS Code, Cursor, Claude, Windsurf or any agent framework. The
same pieces exist in the SDK (`opensmartroute.estimate`, `opensmartroute.mcp_server`, `osr estimate`,
`osr mcp`) and on the hosted platform (`POST /api/v1/estimate`, `GET /api/v1/models/recommended`,
`/mcp`, the `/estimate` page and *Dashboard -> Integrations*).
## 1. Quotes
A quote routes the request exactly as `route()` would - signals, policy, strategies, learned
quality - and then prices every ranked candidate instead of executing one:
- **Tokens.** Input tokens come from the text (word / digit / punctuation / CJK / code heuristic,
`estimate_tokens`, calibrated against billed gpt-4.1 prompt tokens on twelve text types - mean ratio 1.02, every
type within about 20 %) plus the history (`estimate_messages_tokens`, 4 tokens of framing per message).
Output tokens are the caller's `output_tokens`, else the signals' predicted answer length, else
`DEFAULT_OUTPUT_TOKENS` (256); `output_tokens_source` says which.
- **Prices.** `usd_per_1k_input` / `usd_per_1k_output` on the target when declared, else the blended
`usd_per_1k_tokens`; a fixed `usd_per_call` is added for agents, tools and humans. On the platform a
target that resolves to a reference model is priced from the live catalogue (`priced_from:
"catalogue"`), so quotes follow vendor list prices.
- **Picks.** `recommended` is the router's decision; `best_quality` the highest quality estimate;
`cheapest` and `fastest` are chosen among candidates within `quality_tolerance` (0.1) of the best
that fit their context window, so a free-but-useless target never wins. `savings_usd` is the gap
between the dearest candidate and the recommended one.
- **Nothing is executed and the text is never stored.**
```python
from opensmartroute.estimate import estimate
q = estimate(router, "Summarise this contract in five bullets", output_tokens=300)
print(q.input_tokens, q.output_tokens, q.output_tokens_source) # 9 300 caller
print(q.recommended.name, q.recommended.cost_usd) # Small model 0.000062
for t in q.targets: # every candidate, ranked
print(t.rank, t.target_id, f"${t.cost_usd:.6f}", t.quality_estimate, t.latency_ms, t.fits_context)
print(q.cheapest.target_id, q.best_quality.target_id, q.fastest.target_id, q.savings_usd)
q.to_dict() # the JSON the API returns
```
`estimate(router, request, *, output_tokens=None, kinds=None, prices=None, quality_tolerance=0.1,
decision=None)` accepts a `RouteRequest` or a string, a `kinds` filter and a `prices(target) ->
(usd_per_1k_in, usd_per_1k_out) | None` hook for live price feeds; pass `decision=` to quote what an
earlier `route()` already ranked.
```bash
osr -t targets.yaml estimate "Translate this contract into German" --monthly 100000
osr -t targets.yaml estimate "..." --output-tokens 500 --cost-weight 1.0 --json
```
### `POST /api/v1/estimate`
Same body as `POST /api/v1/route` (`text`, `history`, `objective`, `constraints`, `kinds`) plus
`output_tokens` and `monthly_requests`. Without a key the endpoint is rate limited per client (20 per
minute, 8 000 characters) and ignores tenant rules; with a key it is metered as `estimate`, applies
your plan and tenants and returns `X-Request-Id`.
```bash
curl -s https:///api/v1/estimate -H "Authorization: Bearer $OSR_KEY" \
-H "Content-Type: application/json" \
-d '{"text": "Summarise this contract...", "monthly_requests": 50000}'
```
```json
{
"request_id": "…", "input_tokens": 9, "output_tokens": 256, "output_tokens_source": "signals",
"total_tokens": 265, "confidence": 0.71, "savings_usd": 0.00209,
"recommended": {"target_id": "llm-small", "name": "Small model", "vendor": "openai", "model": "openai/gpt-4.1-nano",
"input_cost_usd": 0.0000009, "output_cost_usd": 0.0001, "cost_usd": 0.000103,
"latency_ms": 400, "quality_estimate": 0.62, "fits_context": true, "priced_from": "catalogue",
"rationale": {"semantic": "…", "cost": "…"}},
"cheapest": {"…": "…"}, "best_quality": {"…": "…"}, "fastest": {"…": "…"},
"targets": [{"…": "…"}],
"signals": {"complexity": 0.31, "domains": ["legal"], "task_type": "summarise", "contains_pii": false, "…": "…"},
"policy_rejections": {},
"monthly": {"requests": 50000, "recommended_usd": 5.15, "cheapest_usd": 5.15, "best_quality_usd": 109.6,
"dearest_usd": 109.6, "savings_usd": 104.45, "tokens": 13250000},
"anonymous": false
}
```
The `/estimate` page is the same call with a form: paste a prompt, choose what to optimise for, set the
expected answer length and monthly volume, and read the four picks, the signals the router saw and the
full candidate table with input / output cost bars.
## 2. Recommended models
`GET /api/v1/models/recommended` (no key) quotes a representative prompt per use case - `chat`, `code`,
`reasoning`, `summarise`, `extraction`, `pii` - against the live catalogue and returns, for each, the
`recommended`, `cheapest`, `best_quality` and `fastest` pick with cost per request, quality estimate,
latency, 30-day traffic, the reference model's intelligence index and the reasons (`why`). A
`leaderboard` counts how often each target was the router's pick. The digest is recomputed at most
once a minute and follows the deployment's targets, prices and learned quality rather than a
hand-maintained list. It renders on `/models` ("Recommended per use case") and on `/estimate`.
## 3. MCP server
An MCP client - VS Code, Cursor, Claude, Windsurf or an agent framework - calls the router's tools
over JSON-RPC and gets a decision (or a quote, or the executed answer) back:
```mermaid
sequenceDiagram
participant C as MCP client (Cursor / Claude / agent)
participant S as MCPServer(router)
participant R as Router
C->>S: tools/list
S-->>C: route, estimate, recommend, explain, list_targets, feedback, ask
C->>S: tools/call route "translate this contract"
S->>R: route(request)
R-->>S: decision + trace
S-->>C: target, confidence, alternatives, reasons
Note over C,R: ask also runs the chosen target and returns the answer, tokens and cost
```
### What it exposes
| Tool | Purpose | Side effects |
|---|---|---|
| `route` | Pick the best target: decision, confidence, alternatives, signals, ranked candidates with reasons. Takes everything `Router.route` does - `objective` weights (quality, cost, latency, energy, carbon, `quality_floor`), hard `constraints` (region, data boundary, tenant, PII, cost and latency caps, allow / deny lists), `kinds`, `plan`, `top_k`, `exclude` (an agent handing the request back), `context`, `history` and `session_id`; with a marketplace hook also `marketplace: [slug, ...]` - listings considered as candidates for this request only, nothing installed | none |
| `estimate` | The quote above for a request | none |
| `recommend` | One-paragraph recommendation for a task; `priority` = balanced, cost, quality or speed | none |
| `explain` | Human-readable trace: signals, policy filtering, per-strategy breakdown | none |
| `list_targets` | The catalogue (optionally one `kind`) with prices, latency, quality prior, boundary | none |
| `feedback` | Report an outcome (`request_id`, `target_id`, `success`, `quality`, `cost_usd`, `latency_ms`) | learners update |
| `ask` | Route **and** run the chosen target / plan (same `objective`, `constraints`, `history`, `context`, `session_id` as `route`); returns the answer, tokens, cost and the plan's steps | executes |
| `marketplace_search`, `marketplace_get` | Search listings; fetch one with its manifest and snippets (`install: true` also loads it into a local router's catalogue) | installs on request |
Resources: `osr://targets` (the catalogue as JSON) and `osr://stats` (feedback statistics). The
server speaks MCP `2025-06-18` (also accepts `2025-03-26` and `2024-11-05`) over JSON-RPC 2.0:
`initialize`, `ping`, `tools/list`, `tools/call`, `resources/list`, `resources/read`,
`prompts/list`, `logging/setLevel` (validated, applied to the package logger); notifications get no reply; unknown
methods return `-32601`, unknown tools `-32602`, and every tool failure - invalid arguments, *no admissible target*, a
provider that is down, a marketplace listing that does not exist - comes back as an `isError` result the model can
read and act on, never as a protocol error.
`ask` and `marketplace_*` appear only when the server can honour them: `osr serve` and the platform provide both;
`osr mcp` (local, stdio) offers `ask` when something in the catalogue can run - an LLM with a chat handler
(`OSR_LLM_BASE_URL`) or a bound manifest endpoint such as an A2A agent or MCP server synced from the marketplace -
and the marketplace tools through `MarketplaceClient.mcp_hook` (`--no-marketplace` leaves them out); a bare
`MCPServer(router)` exposes the read-only tools and `feedback`.
### Hosted platform
`GET /mcp` describes the server (tools, resources, protocol version, connection snippets) without a
key. `POST /mcp` takes one JSON-RPC message or a batch with the same `Authorization: Bearer `
(or `X-API-Key`) header as the REST API; notifications return `202`. Every routed call carries the
workspace tenant and is metered as `mcp` against the plan's quotas; `feedback` needs the feedback
feature and `ask` the plan feature (Pro and above). Quota, plan and routing errors surface as tool
errors such as `429: rate limit exceeded`. *Dashboard -> Integrations* renders ready-to-paste
configuration for each client with the workspace's key.
HTTP transport (the client talks to `https:///mcp` directly):
```jsonc
// VS Code: .vscode/mcp.json
{"servers": {"opensmartroute": {"type": "http", "url": "https:///mcp",
"headers": {"Authorization": "Bearer osr_..."}}}}
// Cursor: .cursor/mcp.json
{"mcpServers": {"opensmartroute": {"url": "https:///mcp",
"headers": {"Authorization": "Bearer osr_..."}}}}
// Windsurf: ~/.codeium/windsurf/mcp_config.json
{"mcpServers": {"opensmartroute": {"serverUrl": "https:///mcp",
"headers": {"Authorization": "Bearer osr_..."}}}}
```
```bash
claude mcp add --transport http opensmartroute https:///mcp --header "Authorization: Bearer osr_..."
```
Stdio bridge for clients that only launch local processes (Claude Desktop and older clients): the
CLI forwards each line to the platform with your key.
```json
{"mcpServers": {"opensmartroute": {"command": "osr",
"args": ["mcp", "--url", "https:///mcp", "--api-key", "osr_..."]}}}
```
#### Streamable HTTP: sessions, streams and notifications
The endpoint implements the Streamable HTTP transport of the MCP specification, not only request /
reply:
* **Sessions.** The reply to `initialize` carries an `Mcp-Session-Id`; send it back on later calls.
The id is signed with the deployment secret and bound to the workspace, so every replica accepts it
without shared state; an id of another workspace, an expired one (24 hours) or one that was ended
answers `404` - re-initialize. `DELETE /mcp` with the header ends the session (`204`).
* **Streamed replies.** Send `Accept: text/event-stream` on `POST /mcp` and the answer is a
server-sent-events stream: `notifications/progress` for a `_meta.progressToken` (`ask` reports
*routing*, *running on ``*, *recording*), `notifications/message` log lines at the level set
with `logging/setLevel`, then the JSON-RPC result as the last `message` event. Without that header the
reply is plain JSON as before.
* **Notification stream.** `GET /mcp` with `Accept: text/event-stream` and your credential opens the
server's own stream: `notifications/tools/list_changed` and `notifications/resources/list_changed`
when the catalogue, your installed listings or the tools change, a `: ping` comment every 15 seconds,
closed after `max_s` seconds (default and ceiling `OSR_PLATFORM_MCP_STREAM_MAX_S`, one hour). Every
frame has an `id`; reconnect with `Last-Event-ID` and the stream starts with a `list_changed` so a
change during the gap is not missed. `initialize` advertises `listChanged: true` for tools and
resources because of it.
* **Guards.** A browser `Origin` must be one of the deployment's (the web app, the API's own origin,
`OSR_PLATFORM_CORS_ORIGINS`, loopback) - the DNS-rebinding protection the specification asks for;
non-browser clients send none. An `MCP-Protocol-Version` header must name a revision the server speaks
(`2025-06-18`, `2025-03-26`, `2024-11-05`), else `400`. Bodies are capped at 256 KB.
#### Signing in with your identity provider (OAuth 2.1, on-behalf-of)
Besides workspace API keys, `/mcp` accepts a bearer token your organization's identity provider
issued **for this resource** - the OAuth 2.1 resource-server flow MCP clients implement (VS Code,
Claude, Cursor ...): the client is refused with `401` and
`WWW-Authenticate: Bearer resource_metadata="https:///.well-known/oauth-protected-resource/mcp"`,
reads that document (RFC 9728: `resource`, `authorization_servers`, `scopes_supported`), signs the
person in at the authorization server and comes back with a JWT. The platform verifies the signature
against the issuer's JWKS (RS256 / PS256 / ES256 families, keys refreshed on rotation), the issuer,
the audience (`https:///mcp`, or the app's client id / `api://`), expiry and
not-before, then maps the token to the platform user: a linked single sign-on identity whose subject is
the token's `sub` or `oid` first, else the user whose address the token names (`preferred_username`,
`email`, `upn`; `OSR_PLATFORM_MCP_OAUTH_EMAIL_MATCH=false` turns that off). Someone the platform does
not know gets `403` with a pointer to sign in on the site once. The call then runs in the person's
workspace, metered and audited like a keyed one, with their role. The endpoint is reachable both on the
API host and on the web origin (`https:///mcp`, proxied with the session, protocol-version and
stream headers intact); the metadata, the challenge and the accepted audience name whichever URL the
client used, and *Dashboard -> Integrations* shows the key-less configuration when the deployment has an
identity provider.
Configuration defaults from the `microsoft` single sign-on provider (`OSR_PLATFORM_SSO_MICROSOFT_*`): its
tenant names the issuer (`https://login.microsoftonline.com//v2.0`; a multi-tenant app keeps
Entra's `{tenantid}` placeholder and matches any tenant), its client id the audience, its credentials the
on-behalf-of client. Any OIDC issuer works explicitly:
| Variable | Meaning |
|---|---|
| `OSR_PLATFORM_MCP_OAUTH_ISSUERS` | Comma-separated issuers whose tokens are accepted (discovery at `/.well-known/openid-configuration`) |
| `OSR_PLATFORM_MCP_OAUTH_AUDIENCES` | Audiences accepted besides `/mcp` |
| `OSR_PLATFORM_MCP_OBO_CLIENT_ID`, `OSR_PLATFORM_MCP_OBO_CLIENT_SECRET` | The confidential client the on-behalf-of exchange authenticates as |
| `OSR_PLATFORM_MCP_OBO_TOKEN_URL` | Token endpoint for the exchange (default: the token's issuer's) |
| `OSR_PLATFORM_MCP_OAUTH_EMAIL_MATCH` | Whether a verified address claim may identify the user (default `true`) |
| `OSR_PLATFORM_MCP_STREAM_MAX_S` | Longest `GET /mcp` notification stream (default 3600) |
**On-behalf-of.** A model behind the organization's own identity - Azure OpenAI or Azure AI Foundry
with Entra authentication, a corporate gateway - can be a provider whose credential is `obo:`
instead of a key (`obo:https://cognitiveservices.azure.com/.default`), in the deployment file, the
operator console or a workspace's own providers. When a call that arrived with an identity provider's
token reaches such a provider, the platform exchanges that token for one with the scope - Entra's
`jwt-bearer` grant with `requested_token_use=on_behalf_of` for a Microsoft issuer, the RFC 8693 token
exchange for any other - and calls the model with `Authorization: Bearer `: the person's
own permissions apply, no shared key exists, and the exchange is cached per user and scope until shortly
before it expires and recorded in the audit log (`mcp.obo_exchange`, no token material). A request that
came with an API key has nobody to act for: an `obo:` provider is unavailable to it and the fallback
loop moves on to the next executable target. The health probe of such a provider checks reachability
without a credential.
```jsonc
// VS Code: no key in the file - the editor signs you in at the authorization server
{"servers": {"opensmartroute": {"type": "http", "url": "https:///mcp"}}}
```
### Self-hosted and local
`osr serve` mounts the same server at `POST /mcp` (and `GET /mcp` for the description) next to
`/route` and `/v1`, with `ask` backed by the configured providers - and the same Streamable HTTP
transport as the hosted platform: a signed `Mcp-Session-Id` on `initialize` (bound to the access token
it was issued to; `DELETE /mcp` ends it), event-stream replies with `notifications/progress` and log
messages when the client accepts `text/event-stream`, a `GET /mcp` notification stream that announces
`tools/list_changed` / `resources/list_changed` when the catalogue moves (a marketplace sync, a
`/targets` write; `max_s` bounds it, ceiling `OSR_SERVER_MCP_STREAM_MAX_S`), the protocol-version check
and the browser-origin guard (loopback always passes; `OSR_SERVER_MCP_ORIGINS` names other origins).
Without a service, the CLI runs the
server over stdio straight from a catalogue - useful for a laptop IDE setup; set `OSR_LLM_BASE_URL` (and
`OSR_LLM_API_KEY`) and `ask` runs the routed model for real, exactly as `osr serve` would:
```bash
osr -t targets.yaml -r rules.yaml mcp # newline-delimited JSON-RPC on stdin/stdout
osr -t targets.yaml mcp --list-tools # what the IDE will see
osr -t targets.yaml mcp --marketplace-url https://api.example.test # marketplace_* against another marketplace
osr -t targets.yaml mcp --no-marketplace # local tools only
osr mcp --url https:///mcp --api-key osr_... # bridge to a hosted platform
osr mcp --remote --list-tools # tools of the platform you signed in to (osr login)
```
```json
{"servers": {"opensmartroute": {"type": "stdio", "command": "osr",
"args": ["-t", "targets.yaml", "-r", "rules.yaml", "mcp"]}}}
```
When the IDE's environment does not have `osr` on `PATH` (a project virtualenv, `uv run`), use the interpreter
instead: `"command": "python", "args": ["-m", "opensmartroute", "-t", "targets.yaml", "mcp"]`. Both transports are
exercised in the test suite with the official `mcp` Python client: initialize,
`tools/list`, `tools/call`, `resources/list` over stdio against `python -m opensmartroute ... mcp`, and over
Streamable HTTP against `osr serve`'s `POST /mcp` with a bearer token.
### In Python
```python
from opensmartroute.mcp_server import MCPServer, serve_stdio
server = MCPServer(router, execute=my_execute, marketplace=my_marketplace, prices=my_prices)
server.tools() # tool specs (tools/list)
server.call("estimate", {"text": "…"}) # {"content": [...], "structuredContent": {...}}
reply = server.handle(
{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {"name": "recommend", "arguments": {"task": "…", "priority": "cost"}},
}
)
serve_stdio(server) # block on stdin/stdout
```
`execute(request) -> dict` enables `ask`; `marketplace(action, args)` with `action` in `search` /
`get` / `target` (the last returns a `RouteTarget` for a slug and powers the `route` tool's `marketplace`
candidates) enables the marketplace tools; `prices` is the quote hook above. `RemoteMCP(url, api_key)` and
`bridge_stdio(url, api_key)` are the client side used by `osr mcp --url`. Everything is standard
library only; `osr serve` needs the `server` extra (FastAPI).
### Importing other MCP servers as targets
The router is also an MCP *client*: a server's tools, resources and prompts become routable targets.
```python
from opensmartroute.adapters import connect_mcp, connect_mcp_http
client, targets = connect_mcp(["npx", "-y", "@modelcontextprotocol/server-filesystem", "."], resources=True)
http, remote = connect_mcp_http("https://tools.example.com/mcp", api_key="...", resources=True, prompts=True)
for t in [*targets, *remote]:
registry.upsert(t)
```
`connect_mcp` spawns a stdio server; `connect_mcp_http` (`adapters.HttpMCPClient`) speaks the Streamable HTTP
transport (JSON or SSE replies, `Mcp-Session-Id`) to remote servers over https. Tools become `tool` targets
whose handler runs `tools/call`; with `resources=True` every resource becomes a read-only `tool` target that
reads it (`resources_from_mcp`), with `prompts=True` every prompt becomes an instructions-only `skill` slot
whose handler renders the template (`prompts_from_mcp`). Tool descriptions are still scored for injection
risk at import time and signed manifests still verify as before.
### Things to ask your assistant
- "Estimate what this prompt will cost and which model to use."
- "Recommend the cheapest model that can refactor this file."
- "Explain how OpenSmartRoute would route this request."
- "Search the marketplace for a SQL reporting skill and show me its manifest."
- "Route and answer this with the best model, then record that the answer was good."
---
# Content publishing for your website
The platform writes and publishes content for your own website as a service of your workspace: follow the RSS or
Atom feeds you write about, or send a brief, and the posts are written through the router as **your workspace's
own metered requests**, reviewed in the dashboard (or not, if you say so), then **delivered to your website** - a
signed webhook any site or CMS can receive, a WordPress site, a Ghost publication - **and to your social
accounts** (LinkedIn, X, Facebook, Threads, Discord, Telegram, Mastodon, Bluesky, Reddit, or re-posted on Dev.to,
Hashnode, Medium, Blogger, Tumblr with a canonical link), each on its own posting schedule - or **pulled** from
the API and an RSS feed of your own. The same operations are tools on your MCP server, so an agent in your IDE or
your automation can list, write and publish posts. Everything is scoped to the workspace: another workspace never
sees your feeds, posts or destinations.
The platform's own blog and newsletter run on the same engine; this page is about *yours*.
## Where it lives
| Surface | What you get |
|---|---|
| `/platform/dashboard/content` | Write a post from a brief, review and publish posts, follow feeds (check a URL first, rename, recategorise, pause, fetch now), add destinations and test them, copy or replace the RSS pull link, set the editorial profile, read the delivery log. |
| `GET /api/v1/content` | The service at a glance: profile, post and delivery counts, feeds, destinations, the writer in use, limits, the pull URL. |
| `/api/v1/content/*` | Every operation below, with your API key (`Authorization: Bearer `). |
| `POST /mcp` | The `content_*` tools on your MCP server (see [MCP](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MCP.md)). |
Members write, edit and publish; adding, changing or removing a **destination** (it holds a credential of your site)
takes the `admin` role. API keys act as the workspace and pass every role check.
## Posts: three ways in
1. **From a brief** - `POST /api/v1/content/posts/write` with `topic`, optional `notes` (facts, figures, names the
writer stays within) and optionally `url`, the page the article is about - its readable text is fetched and
given to the writer next to your notes. The writer routes the request through
the platform with a plan - persona, skill and model slots, the `editorial-writer` skill always in a skill slot -
and the post's `routing` says which model wrote it, at what cost, with which companions. `status: published`
publishes at once; the default is a draft. The answer carries `writer_decision`: when no executable model is
available, or the request was refused (quota, policy), the post is your brief itself and the decision says why.
2. **From your feeds** - `POST /api/v1/content/feeds` with `url` (RSS 2.0, RSS 1.0 or Atom - or the page of a blog:
the feed it advertises in its ``, or at `/feed`, `/feed.xml`, `/rss.xml`, `/rss`, `/atom.xml` or `/index.xml`,
is the one followed), an optional `name` (the feed's own title when left out) and `category` (a tag on every post
from it). The URL is looked at once before it is followed: the answer carries the feed with what was found under
`probe` (`ok`, `title`, `site_url`, `entries`, the newest `recent` entries, `discovered_from` when a page led to
the feed); a page that answers but has no feed is refused with the reason (400), a URL that does not answer is
followed with the error on its row and tried again on every pass. `POST /api/v1/content/feeds/preview` is the
same look without following - the dashboard's *Check* button - and also says whether you follow it already.
The scheduler polls every enabled feed on the deployment's cadence (`OSR_PLATFORM_PUBLISHING_REFRESH_S`,
an hour by default; a newly added feed replays the last seven days), stores each new entry once (deduplicated by
the entry's id and by the article URL across your feeds) and writes a post for it from the article page's readable
text, or the feed's excerpt when the page cannot be fetched. `POST /api/v1/content/feeds/{feed_id}/fetch` polls
one feed now, `POST /api/v1/content/refresh` all of them; `PATCH /api/v1/content/feeds/{feed_id}` renames,
recategorises or pauses a feed (`enabled`), `DELETE` stops following it (its posts stay). Each feed carries its
last fetch, its last HTTP status and error, how many entries it has yielded and the site it belongs to. Posts
from feeds wait as **drafts** unless the profile's `auto_publish` is on.
3. **Written elsewhere** - `POST /api/v1/content/posts` with `title`, a Markdown `body`, optional `summary`, `tags`,
`source_url`, `image_url` and `status` - your own tool, a form on your intranet, an agent.
`GET /api/v1/content/posts` lists them newest first (`status`, `q`, `limit`, `offset`; bodies left out),
`GET /api/v1/content/posts/{post_id}` has the Markdown body, an HTML rendering and the post's deliveries,
`PATCH /api/v1/content/posts/{post_id}` edits (title, summary, body, tags, image, status) and
`DELETE /api/v1/content/posts/{post_id}` removes a post with its delivery log.
`POST /api/v1/content/posts/{post_id}/regenerate` rewrites a feed post with the current writer, keeping its slug
and status.
Every post has a `slug` (unique within the workspace), `summary`, `why` (one sentence on why it matters to your
readers), `highlights`, `tags`, `keywords` and an `seo` block with a title, description and keywords for your page.
## The editorial profile
`GET|PUT /api/v1/content/profile` - what the writer is told about you: the `publication` name, its `audience`,
the `tone`, the `language` it writes in, the `site_url` where the posts live (the RSS links to it),
`auto_publish` and `article_words` - the length a post aims at (300 to 4,000 words; `0` or empty = the deployment's
default, 2,000). Empty fields fall back to plain defaults (the workspace name, "readers of the publication's website
who want the facts first", "plain, factual and friendly", English). The writer never invents facts: it only uses
what the source text or your notes say.
## How a post is written
Every post from a feed or a brief is written in two metered passes, both routed through the platform with the
`editorial-writer` skill in a skill slot: first the **brief** - a small JSON object with the title, the summary, the
key points, *why it matters*, an outline of sections (three to nine, scaled to the length asked), tags and keywords -
then the **article** itself, plain Markdown from that outline, written for *your* publication, its readers and tone,
in your language, from the source page's readable text (a feed entry) or from your notes and the linked page (a
brief). A draft that stops short of three quarters of the length asked is continued once from where it ended. The
post's `routing` carries both passes (`passes`), the measured length (`length`: words, target, minimum, whether it
came in short) and, when the article pass could not produce at least 400 words, a `reason` - the post then keeps
the brief's own text or your notes. `GET /api/v1/content` reports the lengths in `writer.article`.
## Destinations: getting posts onto your website - and your social accounts
`GET|POST /api/v1/content/destinations`, `PATCH|DELETE /api/v1/content/destinations/{destination_id}`. Each has a
`kind`, a `name`, a `url` (the site, for the three CMS kinds), `auto_publish` (every post that becomes published
is delivered there), a `schedule` (below) and, for the CMSs and blog platforms, `post_status` - `publish`
(default) or `draft`, when you want a last look inside the CMS. Credentials are stored encrypted (AES-GCM under
the deployment secret, bound to your workspace) and never returned; the URL must be a public endpoint (a
self-hosted deployment that runs its CMS on a private network sets `OSR_PLATFORM_BYOK_ALLOW_PRIVATE=true`, the
same switch as for private model endpoints).
| Kind | What the platform sends | What you provide |
|---|---|---|
| `webhook` | `POST` to your URL: JSON with `event: "post.published"`, the `post` (fields above plus `body` in Markdown and `html`), the `workspace` and the `platform`; headers `X-OSR-Event`, `X-OSR-Delivery` and, with a secret, `X-OSR-Signature: t=,v1=.")>` - verify it like a Stripe signature. Answer `2xx`; a JSON body with `id` and `url` is recorded as the post's address on your site. | Any URL that accepts JSON: a route of your site, a serverless function, a CMS plugin, an automation platform. Optional `secret`. |
| `wordpress` | `POST /wp-json/wp/v2/posts` with the title, the HTML content, the excerpt, the slug and the status, authenticated as your user. | The site URL, `username` and an **application password** (Users → Profile → Application passwords) as `secret`. |
| `ghost` | `POST /ghost/api/admin/posts/?source=html` with the title, the HTML, the excerpt, the slug, the tags, the lead image and the source as canonical URL, authenticated with a short-lived token made from your key. | The site URL and an **Admin API key** (`:`, Settings → Integrations → custom integration) as `secret`. |
| `linkedin`, `x`, `facebook`, `threads`, `mastodon`, `bluesky`, `reddit` | A message on your account or page - title, summary, the link to the post on your site (the profile's **site URL** plus the slug; the source article when you have none) and hashtags from the tags, or your own `template` with `{title}`, `{summary}`, `{url}`, `{hashtags}`, `{brand}`, `{why}` - fitted to the network's limit without cutting the link, with the post attached as a link card where the network has one. | `credentials`: the fields `GET /api/v1/content/destinations` lists under `networks` for the kind (a LinkedIn access token and author URN; X's four OAuth 1.0a values or an OAuth 2 user token; a Facebook page id and page token; a Threads user id and token; a Mastodon instance and token; a Bluesky handle and app password; a Reddit script app, account and subreddit). No `url`. |
| `discord`, `telegram` | A channel message with an embed or link preview: title, summary, link, lead image. | A Discord webhook URL; a Telegram bot token and chat id. |
| `devto`, `hashnode`, `medium`, `blogger`, `tumblr` | The article itself (Markdown or HTML as the platform wants it, the tags, the lead image) with your post's page as the canonical URL, published or as a draft (`post_status`). | A Dev.to API key; a Hashnode token and publication id; a Medium integration token; a Blogger blog id and OAuth client with refresh token; Tumblr's blog and OAuth 1.0a values. |
**Schedules.** `schedule` decides when a queued delivery goes out: `{"mode": "immediate"}` (default) sends the
moment a post is published; `{"mode": "window", "days": [0,1,2,3,4], "start": "09:00", "end": "17:00",
"timezone": "Europe/Berlin"}` waits for the next weekday (0 = Monday) and time of day in that zone;
`gap_minutes` keeps two deliveries to the same destination at least that far apart and `max_per_day` caps them
per local day - so a feed that drops five stories at once posts them spread out instead of flooding your
followers. The slot is computed after the last delivery sent or planned on that destination. An explicit
`POST /api/v1/content/posts/{post_id}/publish` always goes now.
`POST /api/v1/content/destinations/{destination_id}/test` sends a clearly marked test post now and answers with
the site's verdict (`ok`, HTTP `status`, `detail`, the `external_url` the site gave it); nothing is stored.
The raw HTML a feed, a model or an editor may have put into a Markdown body is dropped before rendering, so only
what Markdown produces reaches your site.
## Publishing and the delivery log
A post becomes public when it is created or edited with `status: published`, or with
`POST /api/v1/content/posts/{post_id}/publish` - which also delivers it *now* to every enabled destination (or the
`destination_ids` you name), waits for the sites' answers and returns each delivery. A status change alone (PATCH,
auto-publish from a feed, a post created as published) queues one delivery per destination set to `auto_publish`
at the time its schedule allows and sends what is due in the background at once; the scheduler's minute pass
picks up anything left over and the deliveries whose slot has come.
`GET /api/v1/content/deliveries` (`post_id`, `limit`) is the log: every attempt with the destination, the site's
HTTP status, a short `detail`, the `external_id` and `external_url` the site assigned, and `next_at` while a retry
is pending. A delivery is `queued` until its time comes, `sending` during an attempt (an attempt a crashed process
left behind is retried after ten minutes), then `sent` or `failed`. A site that did not answer, failed on its side
(5xx) or asked for a pause (429) is retried after 1, 5,
30 and 120 minutes - five attempts in all; a rejected configuration (401, 403, 404, 400) fails at once with the
site's message. `POST /api/v1/content/deliveries/{delivery_id}/retry` tries again on demand. Each destination
carries its last status, last error and how many posts it accepted.
## Pulling instead of pushing
Your site can fetch the posts itself: `GET /api/v1/content/posts?status=published` and
`GET /api/v1/content/posts/{post_id}` with your API key, or `GET /api/v1/content/feed.xml` - RSS 2.0 of your
published posts, each linked under your `site_url` (or its source when none is set), with the summary, the tags,
the lead image and the source. A feed reader cannot send an API key, so the feed also answers to the workspace's
**feed token** in the query string: the overview's `feed_url` is the complete pull URL
(`.../api/v1/content/feed.xml?token=...`), shown with a copy button on the Content page under *Destinations*. The
token reads published posts and nothing else; `POST /api/v1/content/feed/token` (admins; *Replace link* on the
page) replaces it, and the old link stops working at once. A static-site build step, a feed reader or an
automation platform can consume it.
## From an agent: the MCP tools
Your MCP server (`POST /mcp`, [MCP](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MCP.md)) carries, next to `route`, `estimate` and the others:
| Tool | Does |
|---|---|
| `content_list_posts` | List posts (filter by `status`, search with `q`). |
| `content_get_post` | One post by id or slug with its body and deliveries. |
| `content_write_post` | Write a post from a brief (`topic`, `notes`, `url`, `status`) through the router - a metered request of the workspace. |
| `content_create_post` | Store a post the agent wrote itself (Markdown body). |
| `content_publish_post` | Publish a post and deliver it to the destinations now; returns each site's answer. |
| `content_list_feeds` | The feeds the workspace follows, each with its last fetch and error. |
| `content_add_feed` | Follow a feed, or the page of a blog (the feed it advertises is found); the answer says what was found under `probe`. |
| `content_fetch_feed` | Poll one feed now and write posts for its new entries (a metered request per post). |
| `content_list_destinations` | The websites posts are delivered to, with their last delivery. |
So "write a post about our new opening hours from this page and publish it to the site" is one instruction to an
agent connected to your workspace - and one entry on your usage, activity and savings pages.
## Limits, metering and privacy
- A workspace follows up to `OSR_PLATFORM_PUBLISHING_MAX_FEEDS` feeds (20) and has up to
`OSR_PLATFORM_PUBLISHING_MAX_DESTINATIONS` destinations (10); one scheduler pass writes at most ten posts per
workspace, the rest follow on the next passes.
- Every written post is one metered request (`endpoint: content`, app `publishing`) against your plan's quota,
budgets and wallet, visible on `/platform/dashboard/usage` and `/platform/dashboard/activity` with its routing
decision; the models learn from the outcome like from any other request. Listing, editing, publishing and
delivering are free.
- Feed entries, posts, destinations and deliveries are deleted with the workspace.
- The operator switches the service off with `OSR_PLATFORM_PUBLISHING=false` (no routes, no tools, no scheduler) and
sets the polling cadence with `OSR_PLATFORM_PUBLISHING_REFRESH_S` (0 = only on request).
---
# Browser extension
The OpenSmartRoute browser extension is a **pre-send advisor** for the chat sites people already use -
ChatGPT, Claude, Gemini, Microsoft Copilot and Perplexity. While you type, a small card next to the
composer shows what the request is, how many tokens it carries, whether it contains personal details or
pasted secrets, which kind of model it deserves and what it would cost across the OpenSmartRoute
catalogue. Everything runs on-device by default; the platform is only involved when you ask for it.
Get it from the [product page](https://opensmartroute.ai/extension) (store listing or the packaged download for
your browser). Chrome and Edge share one package (Manifest V3); Firefox has a build of its own.
## What the card shows
**The card is quiet by default.** It opens by itself only when there is a reason to interrupt you: personal
details or a key in the draft, a jailbreak pattern, a held send, a model that will not do what the draft needs, a
draft near the model's context limit, or a cheaper model that fits and saves at least a cent on this request.
Everything else - small slips, tips, prices for their own sake - waits in the pill by the field (see *Anywhere you
write*) until you click it. Close the card and it stays closed for that draft.
The card leads with **one verdict line** that is always visible - *Nothing flagged*, *2 personal details in
your draft*, *An API key is in your draft*, *GPT-5 Thinking is more than this needs*, *3 small slips to fix* - with
an icon for its tone (shield, alert, tip, check) and a live region so screen readers hear the change. The
details underneath open by themselves only when there is something to act on; otherwise they stay folded until you
click the chevron, and your choice is remembered for that site (Settings › Checks › *Show the card's details by
default* opens them everywhere). Every detail is one row: an icon, one sentence in plain words, at most one button.
| Row | Source | What you can do |
|---|---|---|
| What the router sees (task, tokens, the kind of model it needs - *a quick model is enough*, *a standard model fits*, *needs a thinking model*) | On-device signals (a TypeScript port of the router's heuristics) | Read it |
| Personal details (email, phone, card, IBAN, SSN, Aadhaar, IP, API keys and tokens) | On-device patterns, the same list the platform's guard uses | **Redact** swaps them in the composer for ``-style placeholders; **Undo** puts the original back for 30 seconds. While the send guard holds Enter the row also offers **Redact and send** and **Send anyway** |
| The model the site has selected | The picker label matched against the catalogue and priced for this draft | *X fits this request, about $0.002*; *X is more than this needs - Y does it for about $0.0003 (-90%)* when a lighter model of the same vendor fits; *X may not be enough* when the draft needs reasoning. Switch in the site's picker |
| Small slips, filler | On-device writing checks | **Fix** repairs typos, doubled words and capitals; **Trim** removes filler and repeats. Tips that are not mistakes - a missing audience or format, a vague ask - sit behind one line, *N tips for a better answer*; open it and **Add** appends the missing line |
| Your router's pick and price | Connected only: an on-device estimate over the catalogue, or the workspace's quote (`POST /api/v1/estimate`) when *Send my draft to get prices* is on | The pick with its price and, when the site's own model is known and dearer by at least half a cent, *saves $0.012 vs GPT-5 Thinking* - a real comparison, never a percentage against a model you were not going to use. One line for the cheapest and the best, **Compare all** for the full list, *Quoted by* followed by the workspace's name in the footer. An anonymous quote that hit the rate limit says when to retry and offers **Connect** |
| Improve with your router | Connected only | **Proofread**, **Make clearer**, **Shorten**, **Continue** - the result is shown first; **Use this** replaces the draft |
| Connect your workspace | Signed out | One button to Settings; the public catalogue is never shown as if it were your models |
The extension **reads only the draft in the composer** and the model label. It never reads or changes
what the site sends, never blocks a request and never automates the site's UI beyond replacing your own
text when you click Redact, Fix, Trim or Use this - and sending it when you click *Send anyway* or *Redact and
send* on a held draft. On sites that are not one of the five chat sites the card says *draft only*: the
conversation on the page is never read. When a chat site changes its layout and the built-in selectors no longer
find the box, the card carries on through the general-purpose selectors and the popup says so.
## Anywhere you write, anything you select
The extension behaves like a writing assistant and an AI sidebar at the same time:
- **Writing widget** - a small pill sits in the bottom-right corner of the field you are typing in, in any text
field once *Help in every text field* is on and on the chat sites out of the box. It has three controls:
- the **sparkle** opens the *commands popover*: one box (*What can OpenSmartRoute help you with?*) with the send
arrow inside it (grey until there is something to send, `Enter` sends, `Shift`+`Enter` a new line) whose
instruction runs on the selection inside the field, or the whole field, through your router. The answer
**streams**: the result panel opens the moment you send, shimmering lines stand in until the first words arrive,
the text grows with a caret, and **Stop** keeps what has come so far. When it is done: **Insert** / **Replace
selection**, **Copy** and the target and cost that answered. *Starter ideas* -
**Rewrite this text** (three variants in a *Rephrase / Simplify / More formal / More casual / Shorten / Expand*
menu, the changed words marked, **Show more** for another three, one click puts the pick into the field),
**Fix spelling and grammar**, **Make it shorter**, **Summarise**, **Translate**, **Draft a reply**, **Continue
writing**. Signed out it offers **Connect** instead;
- the **score** - on the chat sites a **count**: how many things are worth a look before you send (personal
details, slips Fix can repair, a model worth changing, a jailbreak pattern); the pill stays empty when there
is nothing. Everywhere else a 0-100 *writing score* computed on the device from the same checks the card runs
(spelling, clarity, delivery, style, terms, privacy; personal details and secrets cost the most). Its
colour is the risk: green, amber for personal details, red for secrets or financial identifiers - and its
label says the same in words. Click it to open or close the card; the card also opens by itself for the reasons
listed under *What the card shows*;
- the **power** button (shown on hover) asks *Disable OpenSmartRoute suggestions on this site?* - **Disable
until next visit** (this browser session), **Disable forever** (the domain joins *Turned off on these sites*
in Settings, removable there) or **Keep it on**. The popup's switch for the active tab does the same. Every
switch - the power button, the popup, the chat-site switches and the *Help in every text field*, *Where the
widget appears* and *Turned off on these sites* controls in Settings - reaches the open tabs at once: the widget
goes or comes back without a reload.
- **Selection actions** - select at least a dozen characters of text on any enabled page (outside a field you
are editing) and, once the mouse settles, a rounded toolbar appears above it: *Explain*,
*Summarise*, *Rewrite*, *Translate*, *Fix grammar* and *Ask*. Each runs through the sidebar with the selection
attached, so the router picks the model and you see the decision under the answer. The same actions are in the
right-click menu, together with *Summarise this page* and *Open routed chat*.
- **Your own commands** (Settings › Commands) - prompts you save once and run with one click: they join the
selection toolbar and the right-click menu next to the built-in actions, the writing widget's commands popover
under *Your commands*, and the [desktop app](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/DESKTOP.md)'s command bar (the same list, the same shape). Write
`{{selection}}` where the text you selected (or typed in the field) goes; `{{page}}` (the page's text), `{{title}}`,
`{{url}}`, `{{app}}` (the site) and `{{language}}` are filled in the same way, and a prompt that names no text gets
it quoted under it, exactly as *Summarise* does. `{{page}}` and `{{url}}` exist only in the browser: a command that
uses them is not offered on the desktop bar, and Settings says so when you tick it. Each command says where it
appears (selected text, writing widget, desktop bar - or
everywhere); **Add three examples** seeds *Reply in my tone*, *Turn into a post* and *Explain for a newcomer* to
edit. Up to 50 commands, kept in this browser; an organisation can pin the list (`commands` below).
- **Answer in** (Settings › Commands) - the language every routed answer is asked for, in the sidebar and in the
commands; empty means the language of the question. It travels as a system line in front of each request.
- **Export** (the download icon in the sidebar's header) - the open conversation as a Markdown file through the
browser's downloads: who said what, the target and the price under each answer, the attached text quoted. Nothing
is uploaded; the file is written from what the sidebar shows.
- **Edge tab and shortcut** - on pages that are not one of the chat sites a tab on the right edge slides out to
*Sidebar* and opens the sidebar; click it again (or press `Alt+Shift+O` again) and the sidebar closes. Right-click
the tab to hide it on that site; `Alt+Shift+O` and the popup work anywhere, including the chat sites, which get no
tab of their own. The empty sidebar offers idea tiles - *Brainstorm ideas*,
*Plan a project*, *Create content* (each starts a prompt in the box) and *Prompt library* (the marketplace) - and
quick-action tiles - *Summarise this page* and, for whatever is attached, *Explain*, *Summarise*, *Rewrite*,
*Translate*; a context chip shows what will travel with the next message and can be detached.
- **Help in every text field** (Settings › More options, on by default) - the writing widget follows the field you
type in on any site: a `textarea`, a text input, a rich editor (`contenteditable`, including the ones that live in
a frame or in a web component), a classic WYSIWYG frame in design mode. Mail, docs, tickets, forms and CMS editors
get the widget without anything to set up; the extension reads only the field you type in, never the page around
it. Turn the switch off to keep the advisor to the chat sites and Google Docs; a **Revoke** row takes the
browser's site access back as well.
**Where the widget appears** narrows it to *All sites* or *Only these domains* (one per line, `*.example.com`
covers the subdomains); **Turned off on these sites** lists the domains disabled from the widget (the power
button on the widget adds the site you are on, for this visit or for good).
- **Google Docs** - a document is a site of its own, on out of the box (Settings › Sites), and works without
*Help in every text field*. Docs paints the page on a canvas, so there is no text field for the widget to sit in
and the browser's own selection is always empty; the extension reads the selection from the hidden editor frame
Docs types into, places the selection toolbar over the highlighted lines, and finds the pill at the edge of the
page. Select a passage and the toolbar's actions (Summarise, Explain, Rewrite, Translate, your own commands) hand
it to the sidebar; the card on the pill reads the selection - personal details, slips, the model the request
deserves - and **Redact**, **Fix** and **Use this** write the result back over the selection the way a paste
does. When Docs will not take the paste the text is put on the clipboard and the card says to press `Ctrl+V`. With
the document's words visible to the extension (Docs offers them to an extension that announces itself before the
editor loads; Chrome, Edge and Firefox) the sidebar's *Summarise this page* reads the document; in Safari it
reads the selection alone. Nothing is read from pixels, and nothing is sent until you ask.
## Surfaces
| Surface | What it is for |
|---|---|
| **Card** (on the page) | The verdict on your draft and the actions on it. Nothing to configure there |
| **Popup** (toolbar icon) | Status: the connected workspace (a selector when the profile remembers more than one), one switch for the site in the active tab (a chat site's adapter, or the writing widget on any other site), today's counters (drafts checked, details hidden, tokens saved, model tips) or a one-line hint while there are none, tracked API calls when tracking is on, *Open the sidebar*, the extension's version, an **Update to x.y.z** card when a newer build is published and the colour-scheme button (system / light / dark). No settings |
| **Sidebar** (edge tab, `Alt+Shift+O`, popup) | Two tabs: **Ask** - chat through your router, with the decision under every answer - and **Suggestions** - the writing score of the field you are typing in, its categories and findings with one-click *Fix* / *Add* / *Redact*, and the stats (words, characters, words per sentence, reading time, grade level). **Sessions** keeps earlier chats of this browser session |
| **Settings** (gear in the popup, or the browser's extension options) | Everything, in one place, saved as you change it. Four essentials are visible - Workspace (Connect; the platform address, tenant and key live in a *Self-hosted or using a key* fold), Privacy, Chat sites, Checks - and the rest sits behind **More options** (help in every text field, where the widget appears, the sites turned off, selection actions, the edge tab, API tracking, one-click connect, model prices). Folds open by themselves when a setting inside them is already on |
| **Welcome** (opens once after install) | Two steps: *1 Explore features* - four slides (the widget, the commands, the score, the sidebar) - and *2 Activate* - **Connect a workspace** (the code sign-in) and **Help in every text field** (the browser's permission prompt). Both optional, both changeable later; *Skip* at any point |
All surfaces share one design system - the brand palette, a stroke icon set, pills,
chips and tiles - and follow the *page's* colour scheme on the chat sites (a dark ChatGPT under a light OS gets a
dark card and the reverse mark). Copy is written for a general reader: no tiers, catalogues or quotes on screen -
*a quick model is enough*, *public prices*, *from your workspace*.
## Analysis policy
Settings › Privacy › **Send my draft to get prices**. Connecting a workspace turns it on (personal details hidden
first); you can turn it back off.
| Level | What leaves the browser |
|---|---|
| **Off** (default until you connect) | Only the public catalogue download (`GET /api/v1/models`, no user data). Signed out, the card shows the on-device checks and a *Connect* row - it does not list the public catalogue as if it were your models; connected, an estimate computed on this device from public prices |
| **On** | The draft is sent to `POST /api/v1/estimate` - anonymously (rate limited) or with your key - **after** on-device redaction unless you turn that off. The platform stores no request text (activity rows carry request id and metadata only). The card labels the result *From your workspace* or *Anonymous quote · public prices* |
Keys are kept in `chrome.storage.local` (not synced) while **Stay signed in** (Settings › Privacy, on by default) is
on; turn it off and the key lives in `chrome.storage.session` - memory only - and you connect again after the
browser restarts. Sign out from the popup or Settings, or revoke the key from the dashboard's API keys page.
### Managed by an organisation
An administrator installs the extension in every browser of every account and settles its settings before anyone
opens it - no store visit, no developer mode, no first-run choice. Two policies do it, and one command writes both
in the shape each browser reads. Download the **policy kit** from the extension page
(`/extension/download/policy`, the `opensmartroute-extension-policy-.zip` of the current build), unzip it
anywhere with Node.js 18 or newer and run the script it carries:
```bash
node policy.mjs --out policy/ --release https://opensmartroute.ai/extension/release.json --settings settings.json
```
The kit is self-contained - `policy.mjs`, the `managed_schema.json` it validates against, a `settings.example.json`
to start from and a README with this command. `--release` takes the store ids and the signed Firefox package from
the website's manifest (explicit `--chrome-id`, `--edge-id`, `--firefox-xpi` win over it); `--settings` is a JSON
object of the keys below. It writes:
| File | Deliver with |
|---|---|
| `opensmartroute-extension-windows.reg` | Group Policy, Intune (as a script or PowerShell `reg import`), a login script; imports as administrator. HKLM `ExtensionInstallForcelist` items 900 and 901 for Chrome and Edge (the rest of your list is untouched), `3rdparty\extensions\\policy` for the settings, Firefox's `ExtensionSettings` and `3rdparty\Extensions\` |
| `opensmartroute-extension-macos.mobileconfig` | Jamf, Intune, Kandji, Mosyle or any MDM: one profile with a payload per browser (`com.google.Chrome`, `com.microsoft.Edge`, their `.extensions.` settings domains, `org.mozilla.firefox`) |
| `opensmartroute-extension-firefox-policies.json` | Firefox's `distribution/policies.json` (Windows, macOS) or `/etc/firefox/policies/policies.json` (Linux) |
| `opensmartroute-extension-chrome-linux.json`, `-edge-linux.json` | `/etc/opt/chrome/policies/managed/`, `/etc/opt/edge/policies/managed/` |
| `opensmartroute-extension-ExtensionSettings.json` | Google Admin's or Intune's policy editor, when you keep the force-install in your own `ExtensionSettings` |
The settings an administrator may pin (the extension declares them in `managed_schema.json`; anything else is
refused by the script and ignored by the browsers): `platformUrl`, `apiPrefix`, `tenant`, `policy`, `sites`,
`everywhere`, `siteMode`, `allowedDomains`, `blockedDomains`, `consoles`, `shareObserved`, `logActivity`,
`staySignedIn`, `askBeforeSend`, `commands` (a list of `{ "id", "label", "prompt", "where" }` objects that replaces
the person's own commands) and `answerLanguage`. Pinned keys are shown locked in Settings with *Set by your organisation*. Chrome
and Edge read them from the `3rdparty` policy (`Software\Policies\Google\Chrome\3rdparty\extensions\\policy` or the
Edge equivalent; on macOS the `com.google.Chrome.extensions.` domain), Firefox from `policies.json` ->
`3rdparty.Extensions.`:
```json
{ "platformUrl": "https://router.example.com", "apiPrefix": "/api/v1", "tenant": "acme", "policy": "quote", "staySignedIn": false }
```
Deploying the [desktop app](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/DESKTOP.md#installing-for-an-organisation) for every account? Its Windows installer takes
`/EXTENSIONS=all` and registers the extension for every account itself (an *Enable OpenSmartRoute?* prompt per
person, no policy); the files above are the way to a silent, forced install, and the only way on macOS.
### Diagnostics
Settings › More options › Advanced › **Copy diagnostics** puts a support bundle on the clipboard: version and
channel, browser, the platform address, every setting with secrets redacted, which keys are policy-pinned, today's
adapter health per site and the last events the worker recorded (quote failures, message errors, install and
update). Nothing is uploaded on its own. The popup also tells you when the extension could not find the message
box on a chat site today ("the site may have changed", with a *Report it* link) or when it fell back to its
general-purpose mode there.
## Connect
Popup or Settings -> **Connect**. The extension starts a device sign-in
(`POST /api/v1/auth/device/code`), opens the approval page in a tab and polls
`POST /api/v1/auth/device/token` until you approve; the minted key is labelled *Browser extension* in
the dashboard. For a self-hosted `osr serve` or an existing key, open Settings › Workspace › *Self-hosted or
using a key*, paste the token and set the API style to *Self-hosted osr serve*.
Once connected, quotes are metered as `estimate` on your workspace, tenant rules apply
(set the tenant in Settings) and the traffic is attributed to the app `browser-extension`: every call carries
`X-OSR-App: browser-extension`, and the dashboard's *Usage* page breaks metered calls down **by app**
(`GET /api/v1/usage` -> `by_app`), so a workspace can see how much of its routed traffic came from the
extension, the CLI or its own integrations.
**Several workspaces.** A person with more than one workspace (their own and a team's, a customer's) connects
each one in turn - the popup's **Add** and Settings' **Add another** run the same sign-in - and the browser profile
remembers all of them. A *Workspace* selector appears in the popup once there are two; Settings › Workspace lists
them with **Use** and a forget button. The card's quote row says which one priced the draft (*From Acme*), the
sidebar's header chip names it, and **Sign out** leaves only that key: another remembered workspace takes over,
so a revoked or expired key never leaves you signed out of everything. The list follows *Stay signed in* the way
the key does - on disk while it is on, in memory only while it is off. The workspace record (name, plan) is
re-read from the platform when it is over an hour old, so a rename or a plan change reaches the popup.
## Routed chat in the sidebar
Click the toolbar icon on Chrome or Edge (click it again to hide the sidebar), or the overview's **Open the
sidebar** on Firefox and Safari (or **Continue in the sidebar** on the card when you are connected). The
sidebar sends your messages to `POST /v1/chat/completions` with `model: "auto"` and streams the answer as
rendered Markdown (headings, lists, tables, code blocks with their own *Copy*; never raw HTML). While an answer
streams a **Stop** button takes Send's place; what arrived stays and is marked *stopped*. Under each answer it
shows the decision as chips - *answered by* the target (on the hosted platform a link that opens the request's
trace on the dashboard: the decision, the spans, the outcome), the cost
and latency the platform measured, the steps when there were several and *backup model* when a fallback ran -
**Copy** for the whole answer and thumbs that post `POST /api/v1/feedback`, so the workspace's learners hear how
the answer went. The preference chips in the composer (*Balanced*, *Cheapest*, *Best*, *Fastest*) set the objective
(`osr.objective`: quality / cost / latency) for every request. Every call the extension makes - quotes, chat,
the catalogue refresh, sign-in, the activity uploads - carries a request id and a W3C `traceparent`, so each one
is a traceable row on the workspace with `app: browser-extension`. **Attach selection** and **Attach the page** in the
header pull text in from the active tab without leaving the panel; from a chat site, *Continue in the sidebar*
carries the visible turns and your draft across as history. **The conversation lives in your workspace**, not in
the browser: the first question starts a conversation the platform keeps (`osr.conversation`), every later one
sends only the new message and the platform prepends the thread - the last twelve turns verbatim, everything older
as a rolling summary - so it survives closing the browser, continues on the desktop app or in another browser,
and never re-sends itself. **Sessions** lists those conversations (marked *from the desktop* when they started
there), reopens or deletes any of them, and still keeps up to twenty browser-only threads from before you
connected. A question that reads like a change of subject gets a line under its answer offering **Move to a new
chat**; the platform moves that exchange into a conversation of its own. When you say something about yourself -
*I work at Northwind*, *we use Rust*, *answer briefly*, *remember that …* - a line under the answer confirms what
was **remembered**, and later questions on every client get that context without repeating it; *forget …* removes
it. The settings page's **Memory** section lists every remembered fact with a bin, an *Add* field and the switch
that turns it off for good. On Chrome and
Edge a one-time bubble points at the browser's pin so the panel is one click away. When something fails the status
line says what happened in plain words and offers the one action that
helps - *Try again* (the question goes back into the box), *Connect* (the key was revoked), *See plans* (the plan's
allowance is used up), *Open settings* (the platform could not be reached).
The **Suggestions** tab needs no workspace: the content script scores the field on the device and hands the panel
the score, the findings and the stats (never the text beyond a 200-character preview). *Fix*, *Add* and *Redact*
on a finding change the field in the page; the score follows.
## Tracking provider API calls
Two channels, both off until you switch them on in Settings › More options › API tracking.
- **Provider consoles and your own apps** - an observe-only `webRequest` listener on the LLM API hosts
(OpenAI, Azure OpenAI, Anthropic, Gemini, OpenRouter, Groq, Mistral, Together, DeepSeek, xAI, Cohere), on
`localhost` / `127.0.0.1` and on any origin you add. It reads the request body (model, messages) and the
status, estimates the completion from `max_tokens`, prices the call from the catalogue and never blocks or
changes anything. The browser asks for access to those hosts once. The popup shows today's calls and
estimated cost per vendor.
- **DevTools panel** (F12 -> *OpenSmartRoute*) - every LLM call the inspected tab makes with the exact
`usage` from the response body (OpenAI chat and Responses, Anthropic Messages, Gemini, Cohere; JSON and
SSE), latency, status, the detected task and the catalogue price; vendor and model chips that narrow the
table (the totals follow what is shown - the tab's model mix at a glance); a detail view with the messages and
the answer; JSONL / CSV export; and **Replay through OpenSmartRoute**, which sends the same messages through
the router and shows both answers with both costs side by side.
**Share what was tracked with my workspace** (Settings › More options › API tracking, connected) uploads call *shapes* - vendor, model,
token counts, site, task type, complexity - to `POST /api/v1/usage/observed` every few minutes. No text
field is ever sent; the platform rejects unknown fields (`422`). The workspace's opportunity report
(`GET /api/v1/usage/opportunity`) compares what those calls cost at list price with what the router's
recommended target would have cost for the same token shape.
### Activity on the workspace
**Log my activity to the workspace** (Settings › Advanced, on by default) records what the extension did as
events on the connected workspace - `POST /api/v1/usage/activity`, posted every few minutes with the observed
usage. An event is a kind, the site's hostname, a count and a figure; there is no field for text and the
platform refuses one. The kinds: `card.analysed` (a draft checked, with the saving between the card's pick and the
dearest candidate), `quote.fetched`, `model.hinted`, `pii.redacted` (with the number of details), `draft.compacted`
(with the tokens saved), `writing.fixed`, `writing.suggested`, `writing.answered` (with the target and request id
of the answer written into a text field), `send.held`, `session.signed_in` / `session.signed_out`,
`extension.installed` / `extension.updated`. The workspace's Apps page turns them into counts per kind and
site; the plan decides how long they are kept and how many a day are recorded (see PLATFORM.md). Administrators
can pin the switch with the `logActivity` managed-storage key.
The first event the extension records for a workspace also sends *Your browser extension is connected* to the
person who connected it: what the extension does, that the workspace sees counts only, the Apps page and this
guide, and what to do if they did not install it. A workspace that routes through the API for three days
without the extension or the desktop app ever reporting in gets one note about the two (*Getting started* mail,
switchable on the account page).
### The AI spend page
Signed-in users read the report on `/platform/dashboard/spend`: observed calls and spend, the
router's quote for the same requests, the gap, the models with the widest gap and the target the router
would have picked, breakdowns by model, site and task, the per-call history (`GET /api/v1/usage/observed`,
newest first, never any text) and a CSV export. Only calls the catalogue can price count towards the gap,
so both sides describe the same requests; the page says so and never calls the number a saving. Rows are
kept for 90 days and deleted with the workspace. While nothing has been observed yet the page explains
what to install and how to switch sharing on.
## Install
The product page `/extension` on the website is the front door. It detects the visitor's browser (Chrome
and the other Chromium browsers, Edge, Firefox, Safari) and offers, in this order:
1. **the store listing** for that browser - one click, no developer mode - as soon as the release carries it
(`stores` in `latest.json`, see below): Chrome Web Store,
Microsoft Edge Add-ons, Firefox Add-ons, Mac App Store;
2. **a vendor-signed package that installs from the click**: the Firefox build is signed by Mozilla (AMO,
unlisted channel) and served as an `.xpi`, so **Add to Firefox** installs without `about:debugging`;
3. **the packaged build** for that browser - **Download for Chrome / Edge** - followed by a guided
developer-mode install whose steps follow the selected browser (extensions page address with a copy
button, Developer mode, Load unpacked); Safari's package is the Mac app the store will carry;
4. when no build is published yet, the page says so and shows the same steps for the day it does.
Chrome and Edge cannot install an extension from a website without developer mode - Google and Microsoft
removed every non-store path in 2018 - so for those two the store listing is the only one-click install and
every stable release is submitted to both stores. The desktop app shortens the wait: **Get the
extension** on its welcome screen downloads the Chrome or Edge package, unpacks it into a folder of its own
under the app's data directory and names that folder, so the two clicks left are Developer mode and **Load
unpacked**. The folder is removed again with the extension (*Remove* in Settings, or uninstalling the app).
The page recognises an installed extension - the platform-origin content script sets
`data-osr-extension=""` on `` - and turns every call to action into "Installed" with a link to
the AI spend page. The **Product** menu carries the extension as its featured card with a browser-aware
"Add to Chrome" button, visitors on a supported browser without the extension see a floating install prompt
once per month on the public pages, and signed-in users see a one-time, dismissible suggestion on the
dashboard overview and the Usage page.
**Where the packages come from.** Every release publishes one package per store - Chrome, Edge, Firefox and
Safari (the Safari package is a Mac app) - stamped with the release version, or with a four-part preview
number for developer previews so no two builds share a file name (see [RELEASES.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/RELEASES.md)). The
Firefox build is signed by Mozilla: previews as a one-click unlisted `.xpi`, stable releases through the
Firefox Add-ons review queue once the listing is live, delivered by the store after approval. Mozilla's
review never holds the other browsers back: while signing is pending the manifest lists the Firefox zip in
place of the `.xpi`, and the signed package takes its place once Mozilla has issued it. Stable Chrome and
Edge packages are submitted to their stores' review; the product page switches to a store button the moment
a listing is approved. Each channel publishes a manifest (`latest.json` for stable, `preview.json` for
previews) naming every package with its size, SHA-256, kind (`zip`, `xpi`, `app`), signature state and the
live store URLs.
The website serves:
- `GET /extension/download/{chrome|edge|firefox|safari}` - 302 to the current package for that browser (a
Mozilla-signed `.xpi` for Firefox installs on arrival; Edge takes its own package or Chrome's); the stable
release by default, `?channel=preview` for the developer preview; 404 while no build is published;
- `GET /extension/download/policy` - 302 to the administrator's policy kit of the same build (`policy` in the
manifest), see *Managed by an organisation*;
- `GET /extension/release.json` - the manifest with same-origin download links, for scripts and the
extension's own update check (`?channel=preview` likewise; `channel` in the answer says which build it is).
### Stores and signing without a paid developer programme
No browser needs a paid licence to *run* the extension; what a paid or verified account buys is the one-click
install from the vendor's store. What each browser needs:
| Browser | One-click path | What it needs | Without it |
|---|---|---|---|
| Firefox | Mozilla signs the package (AMO, unlisted channel) and **Add to Firefox** installs from the click. | A free Firefox account and its API credentials (`OSR_EXTENSION_AMO_JWT_ISSUER` / `_SECRET`). | Load Temporary Add-on from `about:debugging`. |
| Edge | Microsoft Edge Add-ons listing. | A free Partner Center registration for Edge Add-ons; `OSR_EXTENSION_EDGE_PRODUCT_ID`, `_CLIENT_ID` and `_API_KEY`. With them set, every stable version is submitted. | Developer mode: download the package, `edge://extensions`, Load unpacked. Managed devices: the `ExtensionSettings` force-install policy on the product page installs it without developer mode. |
| Chrome and other Chromium browsers | Chrome Web Store listing. | A Chrome Web Store developer account (a one-time registration fee, no yearly programme). Register the extension once by hand in the developer dashboard (the store creates no items through the API), note the item id and the publisher id, enable the *Chrome Web Store API* on a Google Cloud project and give it a service account (added under Developer Dashboard > Account) or a Desktop-app OAuth client. With `OSR_EXTENSION_CWS_*` set, every stable version is submitted through the Web Store API (V2; the V1 API is switched off on 2026-10-15). | Developer mode as for Edge; the `ExtensionInstallForcelist` policy for managed devices. |
| Safari | Mac App Store listing. | An Apple Developer Program membership (yearly fee) to sign and notarise the Mac app. | The Xcode wrapper is still published; anyone with a free Apple ID opens it in Xcode and runs it on their own Mac (Safari > Settings > Developer > *Allow unsigned extensions*). |
In short: Firefox costs nothing and is signed today, Edge costs nothing but a registration, Chrome a one-time
fee, Safari a yearly membership. The product page and the dashboard's Apps page never claim a store that is not
configured: they show the signed package, the packaged build with its steps, or the store button - whichever
the release manifest can back.
**Updates.** A store install updates through its store. The signed Firefox `.xpi` from the downloads page updates
itself too: its manifest names `/extension/updates.json` (the preview build `?channel=preview`), Firefox reads it
daily, checks the SHA-256 and installs the newer signed package - the same mechanism the store uses. A package
installed unpacked on Chrome or Edge and the Safari app have no updater, so the extension asks
`/extension/release.json` for its own channel twice a day (and when you open the overview or the sidebar after that
long): a small dot appears on the toolbar icon, the sidebar shows an **Update to x.y.z** line under its header and
the overview an **Update to x.y.z** card, both linking to the package for
your browser and saying what to do (download, then *Reload* the unpacked folder). Nothing is downloaded or installed
by itself on those browsers.
**Self-hosted platform.** The packages ask permission for `https://opensmartroute.ai` only. When you enter another
platform URL in the options, the browser asks you to allow that site (an optional host permission); decline and
the hosted platform stays selected.
While no stable release of the extension has been published yet, the developer preview stands in on both
routes and the product page says so ("developer preview" next to the version). Versioned packages are
immutable (cached for a year); the two manifests are the moving pointers (cached 5 minutes, mirrored by the
website). A stable release publishes the stable packages; every change to the platform publishes a new
developer preview. The source repository is private: the downloads and the store
listings are the only distribution, and `/extension/release.json` carries the SHA-256 to verify a package against.
**Managed browsers.** A web page cannot install an extension by itself (browsers removed inline install in
2018); what installs it automatically for everyone in a company is the `ExtensionSettings` policy - Group
Policy or Intune for Edge and Chrome on Windows, Google Admin for managed Chrome profiles. The product page
renders the snippet with the store id (`NEXT_PUBLIC_EXTENSION_CHROME_ID` / `_EDGE_ID`):
```json
{ "": { "installation_mode": "force_installed", "update_url": "https://clients2.google.com/service/update2/crx" } }
```
Edge uses the same shape with `https://edge.microsoft.com/extensionwebstorebase/v1/crx`.
Loading a downloaded package by hand (the guided installer on `/extension` shows the same steps for your browser):
- Chrome: `chrome://extensions` -> Developer mode -> **Load unpacked** -> the unzipped folder (the one holding
`manifest.json`).
- Edge: `edge://extensions` -> Developer mode -> **Load unpacked** -> the same folder.
- Firefox: a signed `.xpi` installs when you open it; an unsigned zip loads through
`about:debugging#/runtime/this-firefox` -> **Load Temporary Add-on** -> `manifest.json`.
- Safari: unzip the app, move `OpenSmartRoute.app` to Applications, open it once, then Safari > Settings >
Extensions; an unsigned build needs Develop > **Allow unsigned extensions** first.
The same packages can be built from a checkout with `npm run build:all` (or one of `build:chrome`,
`build:edge`, `build:firefox`, `build:safari`), `npm run safari:xcode` (macOS: the Xcode project and app) and
`npm run pack` (zips + the channel manifest in `dist/`).
Permissions: `storage`, `alarms`, `sidePanel`, `webRequest` (observe-only; no install warning), `contextMenus`,
`scripting` and `activeTab` (page summaries and waking the tabs already open when a setting changes), and every
`http` / `https` site for the content script - the writing widget works in any text field out of the box, so the
browser names that at install (Firefox asks for it the first time instead; until it is granted the advisor runs
on the chat sites and Google Docs alone). The hosted platform origin (`https://opensmartroute.ai`) is where the
worker talks to. A published package carries no local address (`localhost`, `127.0.0.1`), so the one-click
connect bar never runs on a site you are developing. A self-hosted
platform URL on another origin - a local `osr serve` included - is requested when you save it.
## Test
Every package is installed into a real Chrome, Edge and Firefox and driven through a sign-in, a redaction and a
routed request before it is published. Nothing reaches a store or the downloads page until those checks pass.
## Design
The card and the pages follow the brand guide: the real mark (positive artwork on light surfaces, reverse
on dark), the Quicksand wordmark with *Smart* in the cyan text tone, the website's editorial palette in both
colour schemes - warm cream and charcoal neutrals (paper, snow, mist, line, ink, slate, muted) and one warm accent,
*clay*, for everything you can press or follow (primary buttons, the send control, switches, links, focus rings,
the streaming caret, the toolbar badge) - hairline rules instead of shadows for emphasis, mono numerals in
the quote ledger and the three brand hues reserved for the mark, the tier badge, the pick labels and
informational chips. Icons are the
website's set (lucide, one stroke weight) and every one names the thing it stands for - a pen for *rewrite*, a spell-check mark for *fix*, a power
button for *turn off here*; the generic sparkles and wands are not used anywhere. The extension, the desktop app
and the site share one look.
The extension's own pages (popup, sidebar, Settings, welcome, DevTools) follow the system colour scheme or the one
pinned under **Settings › Appearance** (also the sun / moon / monitor button in the popup and sidebar header); the
card and the widget follow the *page* they sit on, so a dark ChatGPT under a light OS gets a dark card. The card sits
just above the composer, right-aligned with it, collapses to a single line, folds long candidate lists
behind *Show all*, shows a loading row while a draft is first analysed and respects `prefers-reduced-motion`.
## Privacy summary
- Default policy is local only; nothing about your prompt leaves the browser.
- With the quote policy, personal details are redacted on-device before the draft is sent.
- The verdict says *Nothing flagged* - no personal-detail **patterns** were found (emails, phones, cards, IBAN,
SSN, Aadhaar, IP addresses, API keys). Names, addresses and free-text facts are not detected; read before you send.
- No third-party analytics. What reaches *your* workspace is the activity described under *Activity on the
workspace* - counts, token estimates, the site category, the target that answered, never text - and, only if you
switch it on, the model and token counts observed on the chat sites. No remote code: the heuristics and adapters
are bundled; the catalogue is data. Diagnostics are copied by you, never uploaded.
- Keys are stored locally (or in memory only with *Stay signed in* off), never synced, and can be revoked from the
dashboard at any time.
---
# Desktop app
OpenSmartRoute Desktop puts your router one keystroke away in every application on your computer. Press
**Ctrl+Shift+Space** (**⌘⇧Space** on a Mac) and a small bar opens over whatever you are doing - Word, Outlook,
Slack, Teams, VS Code, Notes, a terminal, a browser. Type a request, press Enter, and the router picks the model,
answers, and tells you which model it was, what it cost and how long it took. The app also
installs the [browser extension](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/EXTENSION.md) into every browser on the machine and keeps itself up to date.
Windows, macOS and Linux. Download from the [product page](https://opensmartroute.ai/desktop) or the
[downloads page](https://opensmartroute.ai/downloads); the current build, its size and SHA-256 are listed there.
## Install
| System | File | Notes |
|---|---|---|
| Windows 10/11 (x64, ARM) | `OpenSmartRoute--win-x64.exe` / `-win-arm64.exe` | Installs for you, no administrator rights, and starts when it is done. Run as administrator and choose *Anyone who uses this computer* (or `/S /ALLUSERS`) to install for every account - see [Installing for an organisation](#installing-for-an-organisation). `-win-x64-portable.exe` runs without installing (and does not self-update) |
| Windows 10 (1809+) / 11 (x64, ARM) | `OpenSmartRoute--win-x64-native.exe` / `-win-arm64-native.exe` | The [native Windows app](#the-native-windows-app): the same bar and workspace in WPF, Mica and Acrylic on Windows 11. Installs for you, no administrator rights; `/S` silently, `/S /ALLUSERS` from an elevated prompt for every account. `-native-portable.exe` is the bare self-contained exe: put it where you like and run it |
| macOS 12+ (Apple silicon, Intel) | `OpenSmartRoute--mac-arm64.dmg` / `-mac-x64.dmg` (or `.zip`) | Drag to Applications. The app lives in the menu bar; a Dock icon appears only while the main window is open. Administrators: the `.pkg` installs to Applications for everyone, and the `.mobileconfig` grants the two privacy permissions through your MDM |
| macOS 15+ (Apple silicon, Intel) | `OpenSmartRoute--mac-universal-native.dmg` (or `.zip`) | The [native Mac app](#the-native-mac-app): the same bar and workspace as SwiftUI, one universal binary of about a megabyte, Liquid Glass on macOS 26. Drag to Applications |
| Linux | `.AppImage` (any distribution), `.deb` (Debian, Ubuntu), `.rpm` (Fedora, openSUSE) | The AppImage self-updates; `.deb` and `.rpm` update through your package manager |
The first run opens a short welcome: what the app does, **Add** the browser extension, **Sign in** to a
workspace, **Check my writing as I type** where this system can read a focused field, and **Start when I sign in**.
Every step can be done later from Settings. After that the app sits in the
system tray (Windows, Linux) or menu bar (macOS) and waits for the shortcut.
Until code signing is in place the installers are unsigned, and the first launch needs one confirmation:
- **Windows:** SmartScreen shows *Windows protected your PC*. Choose *More info*, then *Run anyway*.
- **macOS:** the app is refused with *Apple could not verify "OpenSmartRoute" is free of malware*. Open **System
Settings > Privacy & Security**, scroll to the notice about OpenSmartRoute and choose **Open Anyway**; on macOS 14
and earlier, Control-click the app in Applications and choose **Open**. It opens normally from then on.
Check the SHA-256 on the downloads page if you want to be sure of the file.
## The main window
Two surfaces, for two ways of working. The **main window** is the app itself - a normal, resizable window with a
taskbar entry, an application menu, a title that names the conversation in view, and a size, place and
maximised state it remembers between runs. Open it from the tray (*Open OpenSmartRoute*), by double-clicking the
tray icon, from the Dock on macOS (the icon appears while the window is open; right-click it for *Ask from Anywhere*,
*New Conversation* and *Settings…*), from the taskbar's Jump List on Windows (right-click the pinned icon: the same
three tasks), or with an `osr://open` link.
The **sidebar** on the left holds **New chat**, the five sections, and every past conversation, grouped by day
(Today, Yesterday, This week ...) with the time and how many turns each has. A search field narrows the list by
title or content. Click a conversation to carry on where it stopped; double-click it (or the pencil) to rename it
in place; the download icon exports it as a Markdown file wherever you choose (who said what, the target and the
price under each answer - written from what the window shows, nothing uploaded); the bin asks once, in place,
before removing it. Conversations live in your workspace: the same list
appears in the browser extension's side panel and on the dashboard's Memory page, a thread started there opens
here (marked *from the browser extension*), a rename or a deletion reaches every client, and a reinstall loses
nothing. A copy of the recent ones is kept on this computer too (Settings › *Keep past conversations*), so the
list is there before the workspace answers. The footer shows the workspace you are connected to and opens
Settings. **Ctrl+B** (⌘B) folds the sidebar away.
**Chat** is where the window opens.
- **The conversation** shows each answer as Markdown - code blocks carry a copy button - followed by one line with
what the router decided: which target answered, the provider behind it, whether it fell back from another target,
the cost, the time taken and how many prompt tokens were trimmed. **Copy**, **Regenerate** and the two rating
buttons sit on the same row; a rating goes back to the router as feedback, which is how it learns what works for
you.
- **The thread is the platform's.** Only your new question travels; the platform prepends the conversation so far
(the last twelve turns verbatim, everything older as a rolling summary), so a long conversation never outgrows
the model and never re-sends itself. When a question reads like a change of subject, a line under the answer
says so and offers **Move to a new chat**, which lifts that exchange into a conversation of its own; when you
return to a thread after hours, a line above it says how long ago it was and offers to start fresh instead.
- **It remembers what you tell it.** Say *my name is Priya*, *I work at Northwind*, *we use Rust*, *answer
briefly*, *remember that the sprint ends Friday* and a small line under the answer confirms what was kept;
later questions - in this app, the extension or the playground - get that context without repeating it. *Forget
that I work at Northwind* removes it. Settings › *Memory* lists every remembered fact with a bin, an *Add* field
and the switch that turns the whole thing off; nothing is ever inferred, only what you said.
- **What matters most** - *Prefer* Balanced, Cheapest, Best quality or Fastest - is in the header and applies to the
next question.
- **Enter** (or the arrow button) sends, **Shift+Enter** adds a line. **Stop** (the same button, or **Esc**)
interrupts an answer that is still streaming.
**Models** is the catalogue the router chooses from, so you can see the choice before you ask. Every target your
workspace can reach, with the list price per million tokens in and out, the context window, the typical reply
time, its benchmark score or quality prior, and - once you have used it - how many requests it took, how many
succeeded and what they cost. Search by name or vendor, and order by most capable, cheapest, fastest or most
used. A thinking model is marked, and so is one turned off for the workspace.
**Usage** is what your workspace counted - every client together: this app, the browser extension, the API, the
playground. Requests over the last 7, 30 or 90 days (how many answered without an error, what they cost the
providers, how many came from this app), one bar per day, the split by app and by target, and the newest requests
one by one with when, from where, what answered, the cost, the time taken, whether it went through and how you
rated it. The same figures as the workspace dashboard, read live from `GET /api/v1/usage` and `GET /api/v1/activity`;
the trace of each request stays on the dashboard.
**Writing** is the writing assistant as a page: on or off, how many checks and fixes it has done, autopilot, what
it checks, which kinds of app it watches, and every application it has met with its own switch - the same
controls as Settings › *Writing assistant* (see below), arranged so "is it watching Word?" is one glance.
**Overview** is the workspace and plan you are connected to, the requests, tokens and spend your workspace counted
over the last 30 days across every client (the same figures as *Usage*, so they do not change when this computer's
history is trimmed or cleared), how many conversations are kept here, the most recent conversations, which targets
actually answered your questions as a share of the total, and what the catalogue currently holds - the cheapest and
the fastest target your workspace can reach. Without a workspace it falls back to the answers kept on this computer
and says so.
### Menu and shortcuts
The application menu (Alt shows it on Windows and Linux; it is always in the menu bar on macOS) carries the
shortcuts a desktop app is expected to have:
| Menu | Item | Shortcut |
|---|---|---|
| File | New Conversation | Ctrl+N / ⌘N |
| File | Ask from Anywhere (the command bar's global shortcut) | Ctrl+Shift+Space / ⌘⇧Space by default |
| File | Settings… | Ctrl+, / ⌘, |
| File | Quit | Ctrl+Q / ⌘Q |
| Edit | Undo, Redo, Cut, Copy, Paste, Select All | the usual |
| Edit | Find Conversation… (focuses the sidebar search) | Ctrl+K / ⌘K |
| Edit | Focus Composer | Ctrl+L / ⌘L |
| View | Chat, Models, Usage, Writing, Overview | Ctrl+1 … 5 / ⌘1 … 5 |
| View | Toggle Sidebar | Ctrl+B / ⌘B |
| View | Appearance (Follow System, Light, Dark), zoom, full screen | |
| Help | Documentation, Open Dashboard in Browser, Copy Diagnostics, Check for Updates…, About | F1 for the documentation |
**Right-click** works the way it does in every other app. On a field it offers Undo, Redo, Cut, Copy, Paste, Paste
and Match Style, Delete and Select All, greyed when the field does not allow them; over a word the system spell
checker underlined (Windows and macOS - the app uses the checker your system already has, so nothing is downloaded
and no text leaves the computer for it) the suggestions come first, with *Add to dictionary*; on a selection in an
answer it offers Copy, *Ask about "…"* (opens the command bar with the selection as the question) and *Search the web
for "…"*; on a link, *Open link in browser* and *Copy link address*. Links never open inside the app.
The **command bar** below is the other surface: it opens over whatever you are doing, answers, and gets out of
the way.
## The command bar
- **Open** it with the shortcut from any app, from the tray menu (*Ask*), or by opening an `osr://` link. If
another program already holds Ctrl+Shift+Space the app falls back to Ctrl+Alt+Space and says so in Settings.
The first open shows a one-line coach mark - *two ways to ask*: type and press Enter, or select text in the app
you are in (it is named) and press the shortcut (shown as the real keys) - and the empty bar offers five
starters that each do a whole task: **Fix the text I copied**, **Summarise what I copied** and **Explain what I
copied** attach whatever you copied last and send it with the right instruction (if nothing is copied the bar
says so and how to copy), **Write an email** and **Translate something** put an opening in the field to finish.
A signed-out bar says so up front with a **Sign in** button; a signed-in one names the workspace in the footer.
- **Type** a request. While you type, the same checks the browser extension runs happen on this computer: the
verdict line says *Nothing flagged*, *2 personal details in your text* or *An API key is in your text*, with the
token count and whether a quick, standard or thinking model is enough. **Redact** swaps personal details for
placeholders such as `` before anything is sent. *Nothing flagged* means no personal-detail patterns
were found; names, addresses and free-text facts are not detected.
- **Choose what matters** with the four chips after *Prefer*: **Balanced** (default), **Cheapest**, **Best quality**,
**Fastest**. They set the router's objective for the next request.
- **Your own commands** (Settings › Commands) sit next to the starters on the empty bar and next to the quick actions
under attached text: prompts you saved once - *Reply in my tone*, *Turn into a post* - with `{{selection}}` where
the text goes (`{{title}}` for the window's title, `{{app}}` for the app in front and `{{language}}` work too; a
prompt that names no text gets it quoted under it). Chosen with nothing attached, a command that works on text
takes what you copied last, like the starters do. The list is the same one the [browser extension](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/EXTENSION.md)
offers on selected text; a command that uses the browser-only `{{page}}` or `{{url}}` stays there.
- **Enter** or the arrow button sends, **Shift+Enter** adds a line, **Esc** steps back one level at a time (stops a streaming answer,
clears the draft, clears the thread, closes the bar). **New chat** starts a fresh thread; the bar keeps the
thread while it is open and says *Draft kept from last time* when it comes back after you switched windows.
- **Answers render as Markdown** - headings, lists, tables and code blocks with their own *Copy* - never raw HTML.
The status line says *Picking a model…* and then *Answering on llm-small* as soon as the router has decided -
before the first word - while a caret follows the text as it streams; a thinking model shows *is thinking* and a
glimpse of its reasoning until the answer proper begins. A **Stop** button sits in the status line throughout;
what arrived stays, marked *stopped*.
- **Every answer** carries one line with the decision under it - *answered by llm-small · azure/gpt-4.1-mini ·
$0.0004 · 780 ms* (the target, the provider and upstream model that ran it, the cost, the time; *after a
fallback* when the first target failed, *N tokens trimmed* when the prompt was compacted) - with **Copy**,
**Regenerate** on the last answer, **Paste into …** and thumbs up / down. Your thumbs are outcome feedback: the
router learns from them. On the hosted platform the target is a link that opens the request's trace on the
dashboard; every call the app makes carries a request id and a W3C `traceparent` and the workspace tenant, so
each one is a traceable row with `app: desktop`. Answers from a self-hosted router on this machine show *$0 · on
this machine* instead of a price.
- **History** (the clock icon, or Ctrl+H) lists past conversations saved on this computer and brings one back;
Settings › Command bar decides how many are kept (0 keeps none) and clears them.
- **Pin** (the pin icon) keeps the bar open when another window takes focus; otherwise it steps aside on blur and
your draft and thread are waiting when you open it again.
- The bar follows the light or dark appearance of your system, or the look you choose in Settings › Appearance.
Every window wears the same editorial palette as the site and the browser extension: warm cream (light) or
charcoal (dark) surfaces and one warm *clay* accent on the things you can press - the primary button, switches,
links and focus rings. The tokens are generated at build time from the extension's `brand.ts`
(`dist/tokens.css`), so the three products never drift apart.
- When something fails the status line says what happened in plain words and offers the one action that helps -
*Try again* (the question goes back into the field), *Connect* (the key expired), *See plans* (the plan's monthly
allowance is used up), *Open settings* (the platform could not be reached).
## Working inside the app in front
When the bar opens it asks the operating system which application is in front and reads the text you have
selected there (Settings › *Read what I selected when the bar opens*, on by default). The app is recognised by the
name its maker gave it - *Word*, *VS Code*, *Slack*, *Chrome* - with its own icon, and the document, page or
conversation the window title names (*Q3 plan.docx*, *Inbox - Gmail*); Store apps are named by their window, not by
the host process. The selection appears as a card - *240 chars · selected in Word · Q3 plan.docx* - with one-click
actions that fit the app: in an editor **Explain code**, **Find bugs**, **Add comments**, **Write tests**,
**Refactor**; in a mail client **Draft a reply**, **Summarise**, **More polite**; in Slack, Teams and other chat
apps **Reply** and **Recap**; in documents and notes **Rewrite**, **Fix grammar**, **Shorten**, **Summarise**,
**Expand**; everywhere else **Explain**, **Summarise**, **Rewrite**, **Fix grammar**, **Shorten**, **Translate**.
While the answer streams a greyed *Paste into Word when done* chip says
paste-back will be possible; every finished answer then offers **Paste into Word** (or whichever app it was, with
its icon), which puts the answer where your cursor was. Pasting is always an explicit click; nothing is typed into
another app on its own. If you select new text while something attached is still unused, the bar asks before
swapping it.
| Where | What happens |
|---|---|
| Word, Outlook, Notes, mail and documents | Selection read, answer pasted back at the cursor |
| Slack, Teams | A thread or message turned into a decision or action items, pasted into the composer |
| VS Code and other editors | Explain an error or a function, write the test, paste it back |
| Terminals (Windows Terminal, PowerShell, iTerm, GNOME Terminal…) | The bar never sends Ctrl+C to a terminal - it would interrupt the running program - so nothing is read there; type or paste the command output instead. Pasting back works |
| Browsers | The selection is read like any other app; on the chat sites the browser extension already sits under the composer |
*Attach the clipboard when the bar opens* (off by default) offers whatever you copied last as context, on this
computer only, until you send.
Requirements per system: Windows needs nothing. macOS asks once for the **Accessibility** permission: the welcome
window and Settings have an **Allow** button that opens System Settings › Privacy & Security › Accessibility on the
right pane and tick themselves once the permission is there. Linux on X11 needs
`xdotool`; on Wayland it needs `wl-clipboard` (to read the highlighted text - no key is sent) and `wtype` (to paste
back), and `kdotool` (KDE), `hyprctl` (Hyprland) or `swaymsg` (Sway) names the app in front - elsewhere the chip
says *selected text* and the paste button says **Paste back**. Settings names exactly what is missing; the bar
itself works everywhere.
## The writing assistant
It is off until you ask for it: the first run offers it as a step, and Settings › *Check my writing as I type, in
every app* (or the **Writing assistant** entry in the tray menu) turns
on a checker that follows you into whatever you are typing in - a mail, a chat message, a document, a form in the
browser. It watches the text control that has the keyboard focus through the operating system's accessibility
layer (UI Automation on Windows, the AX API on macOS, AT-SPI on Linux - the same way a screen reader does; no key
is ever pressed and no text is typed on its own). When you pause for a second the text is checked **on this
computer**:
- **Spelling** with the dictionary you already have - the Windows spell checker (every language installed in
Windows, your custom words included) or the macOS spell checker; on Linux `hunspell` or `aspell` when one is
installed. Nothing is bundled and no word list is guessed.
- **Grammar** with a deterministic rule set that a careful reader would apply without context: a doubled word,
two spaces, a space before a comma, a missing one after a full stop, a sentence starting in lower case, a lone
*i*, *a* / *an* (silent letters and initialisms included: *an hour*, *a user*, *an MRI*), *alot*, *could of*,
*your welcome*, *their is*, *its a*, missing apostrophes (*dont*, *im*, *youre*), day and month names.
- **Style**, as advice only: overlong sentences, repeated punctuation.
A small badge appears at the bottom-right corner of the field - a count while there is something to fix, a tick
while the text is clean - and opens into the list: each item shows the change, other spellings the dictionary
offered, **Apply** and **Ignore** (a word you ignore is remembered). **Fix all** applies every fix at once. Each fix
replaces just its own slip, in place: the rest of the document, its formatting, your caret and the app's undo
history stay as they were (Windows selects the slip through UI Automation and pastes over the selection; Linux
writes the range through the accessibility API, no key sent; macOS walks the caret to the slip with the arrow keys -
System Events can read a selection range but not set one - selects it with Shift-Right, checks that what is selected
is the slip and writes the fix through the accessibility API; an app that lets the range be selected but not
written - Office on macOS - gets the fix pasted over that selection). A document, a note or a mail is never
rewritten whole: where none of that works the badge says the fix failed rather than flatten your formatting. In a
plain control - a chat composer, a form field, a plain-text editor - the corrected text may go in whole through the
accessibility layer when the control takes a value; elsewhere, and only when *Replace text by pasting* is on, the
app receives Select all + Paste. Your clipboard is restored afterwards either way. **Improve with AI** hands the
text to the command bar with the app's quick actions
(*Fix grammar*, *Rewrite*, *Shorten*, ...) - the only path on which the text leaves this computer, and only through
your own workspace's router; the answer's button reads **Replace in** followed by the app's name (*Replace in Word*) and puts the answer in place of the text
the assistant checked (not at the caret, as *Paste into* does for a selection).
What is never read: password and PIN fields, read-only text, address and search bars, terminals, the app's own
windows, a document longer than 20,000 characters (its length is read, not its text), and any kind of app you switch
off (Settings › *Where*: browsers, code editors, mail, chat, office, notes,
other). Code editors are off by default. Every control the system reports as text is watched - Word, PowerPoint and
Outlook documents and composers, Notes, Mail, the text fields and rich editors of Chrome, Edge, Safari and Firefox,
the composers of Teams, Slack and Discord; Chromium and Electron apps build their accessibility tree only when a
client asks, so the first check in such an app can take a moment longer. Under *Apps*, Settings lists **every application the assistant has met** -
its kind, how many checks and fixes it did there, when it last saw it - each with its own switch that wins over the
kind (so code editors can stay off while one particular editor is on, or mail can stay on while one mail client is
off); *Off for this app* on the badge writes that switch from where you are, and *follow its kind* takes it back. The
language follows the text (or the one you fix in Settings); the pause is adjustable. On Linux the field is read
through AT-SPI with the `python3-pyatspi` bindings (the package your desktop's screen reader uses; `python3-atspi`
on some distributions) and accessibility switched on for the desktop - Settings names what is missing when it
cannot read; the shortcut and the bar work regardless.
**Autopilot** (Settings › *Autopilot*, off by default) applies the sure fixes on its own once you pause, so you keep
typing and the slips just disappear: *Fix spelling automatically* takes a misspelling the dictionary corrects with
one or two letters ("teh" → "the", "ocupation" → "occupation"; a word the dictionary would swap for a different word
is left for you), *Fix spelling and grammar automatically* adds the rule set (doubled words, spacing, capitals, *a*
/ *an*, contractions). Style advice is never applied on its own. The badge then reads **N fixed** and opens to
**Undo**; a fix you take back is never applied automatically again in that field, and at most two rounds run on
one pause so a fix can uncover the next without ever looping.
## Connect your workspace
The on-device checks work without an account. To ask through your router, press **Sign in** in the welcome, the
bar or Settings › Workspace: the browser opens a sign-in page for this computer, you approve it, and the app
receives a workspace key. Settings shows the code with a **Copy** button, an *Open the page again* link in case
the tab was closed, and how long the code is still valid. From then on answers come from your workspace's own
models, prices and rules, and every request appears in your [dashboard](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md#13-dashboard) with its trace.
The app talks to OpenSmartRoute (`https://opensmartroute.ai`) unless you tell it otherwise, and telling it
otherwise takes a licence. *On-premises router or a key* (Settings › Workspace) holds the platform address, the
API style and the tenant for a router you run yourself - `osr serve` on this machine (a **Local osr serve** preset
points the app at `http://127.0.0.1:8765` in one click), a self-hosted platform, a proxy in your network - and a
field to paste a key directly instead of signing in. Any address other than OpenSmartRoute's is an **on-premises
router**, and an on-premises router needs a workspace on a Pro plan or higher: sign in to OpenSmartRoute with such
a workspace and the fold's *On-premises licence* row turns to *Licensed* on this computer; until then it reads *Pro
required* with a link to the plans, the router switch, the address and the API style refuse to change, a pasted
serve token is refused, and a router set up earlier gets no answers - the bar says why and Settings offers **Use
OpenSmartRoute** to come back. The licence is kept sealed next to the key and stays through sign-outs; a later
sign-in to OpenSmartRoute with a workspace below Pro withdraws it. An address pinned by your organisation's
[managed file](#managed-settings-for-organisations) needs no licence. A platform address that uses plain `http`
to anything but this machine gets a warning: the key would travel unencrypted. **Sign out** removes the key from
this computer. The key is stored encrypted with your account's own secret store (Windows DPAPI, the macOS Keychain,
the Linux keyring) so a copied file is worthless on another machine or account; where no keyring exists it stays
in a file readable by your user only.
## Add the extension to every browser
**Add to my browsers** (welcome, Settings › Browser extension, tray menu) finds every browser that has a profile
on this machine - Chrome, Edge, Brave, Chromium, Vivaldi, Arc, Firefox, Safari - and registers the browser
extension the way each browser allows:
| Browser | What you will see |
|---|---|
| Chrome, Edge, Brave, Chromium, Vivaldi, Arc | The next time the browser starts it asks once, *Enable OpenSmartRoute?* Accept and the extension's welcome page opens |
| Firefox | Installed silently from the signed package; on Linux the app shows the system-wide path an administrator has to write instead |
| Safari | Opens the App Store listing when there is one |
Settings lists the outcome per browser. **Remove** undoes everything the app wrote, and so does uninstalling the
app on Windows (the uninstaller takes the registrations back before it removes the files; an update leaves them in
place). For a browser whose store listing is not available, the button does everything a program is allowed to do
on a personal computer: it downloads the package, unpacks it into the app's own folder, puts that folder on the
clipboard and opens the browser on `chrome://extensions` (`edge://extensions`). Two clicks remain for you - switch
on **Developer mode**, click **Load unpacked** and paste the folder - because Chrome and Edge only install an
unlisted extension silently on a managed device (domain-joined or enrolled), where an administrator's
`ExtensionInstallForcelist` policy does it.
## Settings
Open Settings from the gear in the bar, the tray menu or `osr://settings`.
| Section | What it holds |
|---|---|
| **Workspace** | Sign in / Sign out, the connected workspace, the device code with Copy and *Open the page again*; *Price my draft while I type* - off: nothing leaves this computer until you press Enter; on: your workspace prices the draft as you type (the text is not stored); *On-premises router or a key* - the *On-premises licence* row (Licensed / Pro required / Managed), the OpenSmartRoute / Local osr serve preset, platform address, API style, tenant, a pasted key |
| **Command bar** | The shortcut (press the keys you want; a combination another app holds is refused, the old one stays and the note offers the fallback in one click; **Reset** returns to the default), **Appearance** (System / Light / Dark), *Start when I sign in*, *Keep the bar open when I switch windows*, *Read what I selected when the bar opens* (with **Allow** on macOS), *Attach the clipboard when the bar opens*, *Keep past conversations* (how many, and **Clear**) |
| **Commands** | **Answer in** - the language every answer is asked for (empty: the language of the question; it travels as a system line in front of each request) - and your own commands: name, prompt with `{{selection}}` and friends, where each appears (selected text in the extension, the extension's writing widget, the desktop bar); **New command**, **Add three examples**, edit and delete. Up to 50, kept in `settings.json` |
| **Browser extension** | Add / Remove, one line per browser with what happened |
| **Writing assistant** | *Check my writing as I type, in every app*; *Autopilot* (off, spelling, spelling and grammar); what to check (spelling, grammar, style), where (browsers, code editors, mail, chat, office, notes, other), the *Apps* list with one switch per application the assistant has met, the pause before checking, the language, *Replace text by pasting when an app cannot be edited directly*, and the words you told it to accept |
| **Updates and about** | The installed version next to *Check now*, *Download*, *Restart to update* and *What's new*; *Download updates automatically*; **Copy diagnostics**; a *Self-hosted update server* fold that shows the effective feed and warns when it is not on the platform's host |
## Updates
The app checks for a new version 30 seconds after it starts and every six hours. With *Download updates
automatically* (default) the new version downloads in the background; a notification and the first line of the
tray menu then offer **Restart to update**. Nothing restarts on its own - a downloaded update is applied when you
choose to restart or when you quit. Development builds, the Windows portable exe and `.deb` / `.rpm` installs do
not self-update; the Updates section says why. The same goes for the macOS build until it is signed with an Apple
Developer ID: macOS only lets an app replace itself with one carrying the same signature, so the Updates section
reads *not signed with an Apple Developer ID* and the downloads page is the way to a new version.
Self-hosted deployments point *Self-hosted update server* at any web folder that holds the installers and the
`latest*.yml` files the build produces (see the [deployment guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/INSTALL.md)). After an update the bar says once
*Updated to x.y.z* with a *What changed* link.
### Updating without the internet, or across a fleet
You never have to wait for the in-app update: **download the installer for the new version and run it over the
one you have**. It keeps your settings, the workspace you connected, your chat history and the browser
extensions, replaces the application, and leaves a single entry in *Apps & features* - no need to uninstall
first. This is the way to update an air-gapped machine, and the way to roll a version out with your own
deployment tooling: copy the installer and run it with `/S` (Windows), install the `.deb` / `.rpm` with your
package manager, or replace the application bundle on macOS.
### Managed settings for organisations
An administrator pins settings for every account on a machine with one JSON file the app reads and never writes:
`%ProgramData%\OpenSmartRoute\managed.json` on Windows, `/Library/Application Support/OpenSmartRoute/managed.json`
on macOS, `/etc/opensmartroute/managed.json` on Linux (or the path in `OSR_DESKTOP_MANAGED_SETTINGS`). Drop it
with Intune, Jamf, Ansible or a login script; the app re-reads it on every use, so a change lands without a
restart, and removing the file gives people their own values back. The keys it may set: `platformUrl`,
`apiPrefix`, `tenant`, `policy` (`local` / `quote`), `autoUpdate`, `updateUrl`, `logActivity`, `readSelection`,
`attachClipboard`, `commands` (a list of `{ "id", "label", "prompt", "where" }` objects that replaces the person's
own commands), `answerLanguage` and any field of `assist` (`enabled`, `autopilot`, `spelling`, `grammar`, `style`, `pauseMs`,
`apps`, `appRules`, `pasteFallback`, `ignore`, `language`). Anything else in the file is ignored. Settings shows a
pinned control greyed out with *Set by your organisation*; **Copy diagnostics** names the file and the keys it pins.
```json
{
"platformUrl": "https://osr.example.com",
"tenant": "acme",
"policy": "local",
"updateUrl": "https://files.example.com/opensmartroute/desktop",
"assist": { "enabled": true, "autopilot": "spelling", "apps": { "editor": false } }
}
```
### Installing for an organisation
Everything above is for one person on one computer. An administrator installs once for every account and settles
the questions the first run would ask, so people open the bar and it works.
**Windows.** The installer installs for the person running it by default (no administrator prompt,
`%LocalAppData%\Programs`). Run it as administrator and choose **Anyone who uses this computer**, or install
silently with `/S /ALLUSERS`, and it goes to `Program Files` for every account, with one entry in *Apps & features*.
A `managed.json` beside the installer - or the file named by `/MANAGED=` - is copied to
`%ProgramData%\OpenSmartRoute\managed.json` at the same time, so the settings above are pinned before anyone signs
in; the uninstaller removes that copy again. Updates keep the mode they find: an all-users install stays one.
```powershell
# every account on this computer, silently, with the organisation's settings beside the installer
Start-Process .\OpenSmartRoute-2.1.0-win-x64.exe -ArgumentList '/S','/ALLUSERS' -Wait
# ... or naming the settings file
Start-Process .\OpenSmartRoute-2.1.0-win-x64.exe -ArgumentList '/S','/ALLUSERS','/MANAGED=\\files\it\osr\managed.json' -Wait
# uninstall
& "$env:ProgramFiles\OpenSmartRoute\Uninstall OpenSmartRoute.exe" /S /allusers
```
Windows needs no privacy permission for the bar or the writing assistant (UI Automation is open to every program
on the desktop). Add `/EXTENSIONS=all` and the installer registers the browser extension for every account as well:
the app runs once, elevated, and makes the same registrations its **Add** button makes for one person in the
machine-wide places - the HKLM external-extension key of every Chromium browser installed on the computer (Chrome,
Edge, Brave, Vivaldi, ...; each person still sees *Enable OpenSmartRoute?* once) and Firefox's HKLM
`ExtensionSettings` policy (installed silently) - and keeps the receipts in `%ProgramData%\OpenSmartRoute\extensions.json`,
which the uninstaller reads back. A browser with no store listing yet is skipped (an unpacked folder cannot be
installed for every account). For a silent, forced install with the organisation's settings pinned, roll out the
policy files the [extension guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/EXTENSION.md#managed-by-an-organisation) generates instead; they work without the
desktop app.
```powershell
Start-Process .\OpenSmartRoute-2.1.0-win-x64.exe -ArgumentList '/S','/ALLUSERS','/EXTENSIONS=all' -Wait
```
**macOS.** Two files come with every build for administrators:
- `OpenSmartRoute--mac-arm64.pkg` / `-mac-x64.pkg` installs the app into `/Applications` for everyone
(`sudo installer -pkg OpenSmartRoute-2.1.0-mac-arm64.pkg -target /`, or pushed by Jamf, Intune, Kandji, Mosyle).
A `managed.json` beside the package is copied to `/Library/Application Support/OpenSmartRoute/managed.json` by
the package's postinstall; a management server that pushes the package from a temporary folder drops the file
itself.
- `OpenSmartRoute--mac-universal.mobileconfig` is a *Privacy Preferences Policy Control* profile that
grants the two permissions macOS would otherwise ask each person for: **Accessibility** (read the selection and
the field you write in, put a fix back) and **Automation towards System Events** (which app is in front, Copy and
Paste). Nothing else is granted. macOS accepts such a profile only from a device-management server, never from a
double-click, so upload it to your MDM as a custom configuration profile scoped to the Macs that get the app. The
profile is bound to the build's code signature: use the one published with the build you deploy (an ad-hoc,
unsigned build ships one profile per architecture, valid for that build alone). Without the profile every person
allows the two prompts once, from the welcome window or Settings.
The browser extension on a fleet of Macs goes through the MDM as well: the [extension guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/EXTENSION.md#managed-by-an-organisation)
generates one `.mobileconfig` that installs it in Chrome, Edge and Firefox for every account and pins the
organisation's settings, alongside the app's privacy profile.
**Linux.** Install the `.deb` / `.rpm` with your package manager and drop `/etc/opensmartroute/managed.json`.
On a machine with no route to the update server the app simply carries on: the check fails quietly, Settings ›
*Updates and about* shows *Could not reach the update server*, and the next check happens on the usual schedule.
Nothing blocks the bar or the tray while updates are unreachable.
To pre-seed settings on a machine before anyone signs in, write `settings.json` into the app's data folder
(`%APPDATA%\OpenSmartRoute` on Windows, `~/Library/Application Support/OpenSmartRoute` on macOS,
`~/.config/OpenSmartRoute` on Linux) before the first start; the app keeps what it understands and fills in the
rest. A file it cannot read is kept next to the original as `settings.json.corrupt-` rather than
being overwritten.
## Diagnostics
Settings › Updates and about › **Copy diagnostics** puts a support bundle on the clipboard: version, install kind,
system, every setting with the key left out, what the OS bridge can do on this machine, the update state and the
last lines of `diagnostics.log` (a small rolling file in the app's folder that records warnings, blocked
navigations, crashes and what the updater did - the feed it read, the version it found, the installer it started
on *Restart to update*; it never leaves the computer). Paste it into a support ticket.
## The native Mac app
Beside the Electron build there is a native app for macOS 15 and later, written in SwiftUI: the same command bar,
main window and settings, drawn with the system's own Liquid Glass on macOS 26 (the system material on macOS 15)
and weighing about a megabyte. It talks to the same platform, keeps the same conversations and sends the same
`osr.objective`; the platform lists its traffic as the `desktop-mac` client. Download
`OpenSmartRoute--mac-universal-native.dmg` from the downloads page, drag it to Applications, and press
**⌘⇧Space**. On a Mac without a GPU the system can draw with - a virtual machine, a remote session, a build agent -
the surfaces draw flat and opaque instead of as glass (the same layout, no blur), because the material would
otherwise stop the app; `OSR_MAC_FLAT=1` forces that look anywhere.
What it does the same way:
- **The bar** over the app in front: the selection read through Accessibility when you allow it, the clipboard
offered when you ask, the on-device verdict (tokens, an e-mail address, a card number, a secret) and **Redact**
before anything leaves the Mac, the streamed answer with the model, confidence, cost and latency, **Paste into
…** straight back into the field you came from, thumbs up and down, **Prefer** (Balanced, Cheapest, Best
quality, Fastest), history, pin, Esc that steps back one level (stop, clear, close). Return sends, ⌘Return too.
- **The main window** (⌘⇧Space then the menu bar icon, or *Open OpenSmartRoute*): Overview with today's
requests, spend and the recent request log, the workspace's conversations (open, rename, delete; the ones the
browser extension held appear too), Models with prices and reasoning flags, Usage per client and per model,
and a conversation view with the same streaming.
- **Sign-in** with the browser (the device code) or a pasted key; the key is sealed in your login Keychain.
Everyone talks to the hosted platform. The **on-premises router** option (`osr serve`, a self-hosted platform,
a company proxy) appears in Settings › Workspace only for a workspace on the Pro plan or higher - the hosted
sign-in grants the licence - or when the organisation's managed file pins the router; a router left in the
settings without either is put back to the hosted platform at the next launch. The rules under
[Connect your workspace](#connect-your-workspace) apply unchanged.
- **Managed settings**: the same `/Library/Application Support/OpenSmartRoute/managed.json` the Electron app reads,
so one MDM payload serves both apps (`platformUrl`, `apiPrefix`, `tenant`, `policy`, `logActivity`,
`readSelection`, `attachClipboard`, `assistEnabled`, `autoUpdate`). The person's own settings live in
`~/Library/Application Support/OpenSmartRoute/mac/settings.json`.
- **Updates**: at launch and once a day the app asks the downloads page for a newer native build (Settings › About
› *Check for updates automatically*, off = only the button there; nothing else is ever contacted besides your
router). A newer build shows on the island and in Settings › About with its size; **Download** fetches it and
checks the published SHA-256, then **Install and relaunch** unpacks the zip, swaps it in for the running app and
opens the new copy - your settings and key stay. When the app sits where your account cannot write (an
administrator's install) the disk image is downloaded instead and opened for you to drag over the old copy. A
preview build follows the preview channel; a release only ever moves to the next release. Nothing is installed
without a click.
- **`osr://` links**: `osr://ask?text=…` opens the bar with the text, `&send=1` sends it at once (a Shortcut or a
launcher can route a request without touching the bar), `osr://main`, `osr://settings`, `osr://welcome`.
### The writing assistant, natively
The assistant watches the text control that has the keyboard focus in whatever app is in front - Word,
PowerPoint, Excel, Keynote, Teams, Slack, Mail, Notes, a browser's page, VS Code, the ChatGPT or Claude desktop
app - through the accessibility layer, and once you pause it checks the text on this Mac: the system's dictionary
and grammar checker (every language macOS spells) plus the shared mechanics rules (the doubled word, *i* for
*I*, *should of*, *a honest*, the space before a comma, the sentence that starts in lower case). A small glass
badge sits beside the field with the count; hover or click it and the card lists each suggestion with **Fix**
(or a menu of guesses), **Fix all**, **Undo**, **Ignore**. Fixes are written straight into the control where the
app allows a ranged edit and pasted over the verified selection otherwise, so a document keeps its formatting.
*Autopilot* (Settings › Writing) applies the sure spelling fixes - or spelling and grammar - on its own while you
pause; style stays advice. Terminals, password fields, address and search boxes, URLs and the app's own windows
are never read; any app can be excluded from the badge's menu.
Beyond spelling, the card carries what OpenSmartRoute itself notices, each with the one button that resolves it:
- **A key or personal data about to leave.** An API key anywhere, a card number, an IBAN, an SSN or an Aadhaar
number in any text, and - when the field is the composer of an AI assistant - e-mail addresses, phone numbers
and IP addresses too. **Redact** swaps each for the bar's placeholder (``) in the field itself; Undo
puts the data back. Nothing is redacted without a click. An address in a mail or a chat is left alone: it
belongs there.
- **Tone and length.** Capitals and stacked exclamation marks read as shouting; a chat message past ~900
characters, a mail past ~2,500. **Ask** opens the bar with a rewrite prompt and the text, and the answer's
*Paste into* replaces the field.
- **A question.** A short question in a chat or a mail offers **Ask** to draft the answer.
- **In an AI assistant's composer** (ChatGPT, Claude, Perplexity, Copilot, Gemini desktop apps): the token
estimate and the tier the request usually lands on (*a small model will do*, *a mid-tier model*, *a reasoning
model*), with **Route it** - the prompt becomes the bar's request, OpenSmartRoute picks the model and says
which one answered, what it cost and how long it took.
### The island
A live activity in the spirit of the iPhone's Dynamic Island continues the notch on a MacBook (a black shape at
the top edge on any other display) and folds away when nothing is happening. It shows one line while the router
thinks (*Routing…*, *Answering with claude-mid*), the decision when the answer lands (*claude-mid · 87% · 640 ms
· $0.0021*), what the assistant found in the field you are typing in (*Contains a card number · Teams*, *3 writing
suggestions · Mail*), fixes and redactions as they are written, the sign-in code to approve, an error as a
sentence. Move the pointer up to it and it opens into a card with the actions - Copy answer, Paste into the app,
Stop, Redact, Fix all, Improve, Undo - and **Ask** (the bar) and **Workspace** (the main window); with nothing
going on the card is the workspace at a glance: who is signed in, today's requests and spend, whether the
assistant is on. Settings › Command bar › *Island* turns it off. It is non-activating: the app you were in keeps
the keyboard.
What differs from the Electron app: no `.pkg` or privacy profile - the Electron app's `.mobileconfig`
does not cover this bundle. Both apps can be installed side by side; they hold different bundle identifiers
(`ai.opensmartroute.mac` here) and settings folders, and the shortcut goes to whichever registered it first, so
pick one shortcut per app in Settings.
Colours follow the system: the accent colour you chose in System Settings on every primary action, the semantic
system colours (red, orange, green, blue, purple) for spelling, grammar, success, style and privacy, `Label`
colours for text, so the app reads like the rest of the Mac in light, dark and increased-contrast appearances. The
OpenSmartRoute mark - the open ring with three routes - is the menu bar icon (a template image, so it follows the
menu bar), the bar's corner, the welcome and the island; the Dock and Finder show the brand's app icon.
The first launch is *Open Anyway* as described under [Install](#install) until a Developer ID signs the build;
the bundle is ad-hoc signed with the hardened runtime otherwise.
### Building it
Building from source needs Xcode (the Command Line Tools alone cannot build SwiftUI); Xcode 26 compiles the
Liquid Glass path, an older Xcode builds an app that draws the system material everywhere, and `OSR_MAC_XCODE=26`
picks that Xcode when several are installed. `npm run mac:bundle -- --arch universal` produces the app bundle, the
`.dmg`, the `.zip` and the SHA-256, stamped with the release version. The bundle is signed with the Developer ID
in `OSR_DESKTOP_MAC_CSC_LINK` / `OSR_DESKTOP_MAC_CSC_KEY_PASSWORD` and notarised and stapled when
`OSR_DESKTOP_APPLE_ID`, `OSR_DESKTOP_APPLE_APP_PASSWORD` and `OSR_DESKTOP_APPLE_TEAM_ID` are set (`--no-notarize`
skips that), or with `OSR_DESKTOP_MAC_SIGN_IDENTITY`, an identity already in the keychain, or ad-hoc otherwise.
The published native builds carry a `-native` marker and the downloads page labels them *native app (macOS 15+)*.
## The native Windows app
Beside the Electron build there is a native app for Windows 10 (1809) and later, written in C# on WPF and .NET 9
with the Fluent theme: the same command bar, main window, settings, welcome, writing assistant and island as the
Mac app, drawn with Windows 11's own materials - Mica behind the main window and Settings, Acrylic behind the
bar - and the system accent colour on every primary action; on Windows 10 the surfaces are the theme's solid
colours. It is one self-contained exe of about 70 MB (the runtime travels inside), talks to the same platform,
keeps the same conversations and sends the same `osr.objective`; the platform lists its traffic as the
`desktop-win` client. Download `OpenSmartRoute--win-x64-native.exe` (or `-win-arm64-native.exe` on an
ARM PC) from the downloads page and run it: the installer puts the app in `%LocalAppData%\Programs\OpenSmartRoute`
for you - no administrator prompt - with a Start-menu entry, a *Programs and Features* entry and the `osr://`
link, and offers to start it. Then press **Ctrl+Shift+Space**. Run a newer installer over it and it upgrades in
place: the running app is asked to quit, the exe is replaced, your settings and the sealed key stay. The
`-native-portable.exe` beside it is the bare exe for a USB stick or a locked-down PC. In a remote session,
without desktop composition, or with `OSR_WIN_FLAT=1`, the surfaces draw flat and opaque (the same layout, no
backdrop).
What it does the same way as the [native Mac app](#the-native-mac-app):
- **The bar** over the app in front: the selection read through UI Automation (the interface screen readers use,
open to every program - nothing has to be granted), the clipboard offered when you ask, the on-device verdict
and **Redact** before anything leaves the PC, the streamed answer with the model, confidence, cost and latency,
**Paste into …** straight back into the field you came from (the app in front is brought forward and Ctrl+V
pressed for it; the clipboard is restored a moment later), thumbs up and down, **Prefer**, history (Ctrl+H),
pin, Esc that steps back one level. Enter sends, Shift+Enter breaks the line, Ctrl+N starts over, Ctrl+. stops.
The bar hides from Alt+Tab and the taskbar and closes when another window takes focus unless pinned or
*Keep the bar open* is set.
- **The main window** (double-click the notification-area icon, or *Open OpenSmartRoute*): Overview, the
workspace's conversations (open, rename, delete), Models, Usage with a 7 / 30 / 90-day window, and the
conversation view. Ctrl+1 / 2 / 3 switch the pages, Ctrl+R refreshes, Ctrl+, opens Settings.
- **Sign-in** with the browser or a pasted key; the key is sealed with DPAPI under your Windows account in
`%APPDATA%\OpenSmartRoute\win` (a copied file is worthless on another machine or account). The on-premises
router option and its licence follow the same rules; the organisation's `managed.json` lives at
`%ProgramData%\OpenSmartRoute\managed.json`, the same file the Electron app reads.
- **The writing assistant**, through UI Automation: the focused text control in Word, Outlook, Teams, Slack,
Notepad, a browser's page, VS Code, the ChatGPT or Claude desktop app; Windows' own spell checker (every
language installed under Settings › Time & language, your custom words included) plus the shared rules and the
insights - a key or personal data about to leave, tone, length, a question, the cost and tier of a prompt in an
AI assistant's composer. Fixes are written through the control's value pattern where it allows a write, pasted
over the verified selection otherwise. The badge is a small non-activating window beside the field.
- **The island** at the top edge of the primary display: folded to a sliver when nothing is happening, one line
while the router thinks or when the assistant has something to say, a card with the actions when the pointer
comes up to it.
- **`osr://` links** (`osr://ask?text=…&send=1`, `osr://main`, `osr://settings`, `osr://welcome`), registered
for your account the first time the app runs unless another live installation already owns the scheme. A second
copy started with a link hands it to the running one.
- **Start at login** through the account's Run key (visible in Task Manager › Startup), off by default.
- **Updates**: at launch and once a day the app asks the downloads page for a newer native build (Settings › About
› *Check for updates automatically*; `autoUpdate` in `managed.json` pins it). A newer build shows on the island
and in Settings › About with its size; **Download** fetches the installer and checks the published SHA-256, then
**Install and restart** runs it silently: the app is asked to quit, the exe is replaced in place, and the app
starts again with your settings and key. A copy installed for every account asks for an administrator's approval
first. The portable exe is never swapped under itself: the new file is downloaded beside it and shown for you to
replace the old one. Nothing is installed without a click.
What differs from the Electron app: no browser extension installer. Administrators install it the same way as the
Electron app - `OpenSmartRoute--win-x64-native.exe /S /ALLUSERS` from an elevated prompt puts it in
`Program Files` for every account, and a `managed.json` beside the installer (or `/MANAGED=`) becomes
`%ProgramData%\OpenSmartRoute\managed.json`; `/D=` picks the folder; the uninstaller in the install folder
(also listed under *Programs and Features*) takes `/S` and removes the Start-menu entry, the `osr://` link and the
start-at-login value while they still name that install, and leaves your settings and key alone. Both Windows
apps can be installed side by side; they hold different install and settings folders and separate *Programs and
Features* entries (*OpenSmartRoute* and *OpenSmartRoute (native)*), and the shortcut goes to whichever registered
it first - the other falls back to Ctrl+Alt+Space and says so in Settings.
### Building it
Building from source needs only the .NET 9 SDK (`winget install Microsoft.DotNet.SDK.9`) and `makensis` for the
installer. `npm run win:bundle -- -Arch both` publishes a self-contained single-file build per architecture,
stamped with the release version, as an installer (`-native.exe`) and a portable exe (`-native-portable.exe`),
each with its `.sha256`. Both files are signed with Azure Trusted Signing when `OSR_DESKTOP_AZURE_SIGN_ENDPOINT`,
`OSR_DESKTOP_AZURE_SIGN_ACCOUNT` and `OSR_DESKTOP_AZURE_SIGN_PROFILE` are set (the same variables as the Electron
app), with a certificate from the account's store when `OSR_WIN_SIGN_THUMBPRINT` names one, and left unsigned
otherwise. `OpenSmartRoute.exe --quit` asks a running copy to quit (the installer uses it). The published native
builds carry a `-native` marker and the downloads page labels them *native app (Windows 10+)*.
## Code signing without buying a certificate
The installers work unsigned; what a signature buys is the absence of the SmartScreen and Gatekeeper prompts
described under [Install](#install). None of the paths below needs a traditional code-signing certificate from a
commercial CA; each one takes effect for a deployment's builds as soon as it is configured - nothing else changes.
| System | Path | What it needs | Build configuration |
|---|---|---|---|
| Windows | **Azure Trusted Signing** (Artifact Signing): Microsoft issues short-lived Public Trust certificates from an identity-validated account and signs through the service; SmartScreen treats the publisher as known from the first download. | An Azure subscription, the Basic tier of the service (a small monthly fee), and identity validation - an organisation in the United States, Canada, the European Union, the United Kingdom, Australia, New Zealand, Japan, South Korea, Singapore, Switzerland, Norway or Israel, or an individual in the United States or Canada (government ID, a few days). The build's service identity gets the *Artifact Signing Certificate Profile Signer* role. | `OSR_DESKTOP_AZURE_SIGN_ENDPOINT` (the region endpoint, e.g. `https://weu.codesigning.azure.net`), `OSR_DESKTOP_AZURE_SIGN_ACCOUNT`, `OSR_DESKTOP_AZURE_SIGN_PROFILE` (naming the certificate profile switches signing on) and `OSR_DESKTOP_AZURE_SIGN_PUBLISHER` = the certificate's subject name so the updater accepts signed updates. |
| Windows | A conventional certificate (`.pfx`) from a CA. | The certificate and its password. | `OSR_DESKTOP_WIN_CSC_LINK` (base64 of the `.pfx`), `OSR_DESKTOP_WIN_CSC_KEY_PASSWORD`. |
| macOS | Developer ID signing and notarisation. | An Apple Developer Program membership (yearly fee); there is no free notarisation. | `OSR_DESKTOP_MAC_CSC_LINK`, `OSR_DESKTOP_MAC_CSC_KEY_PASSWORD`, `OSR_DESKTOP_APPLE_ID`, `OSR_DESKTOP_APPLE_APP_PASSWORD`, `OSR_DESKTOP_APPLE_TEAM_ID`. The same five sign, notarise and staple the [native Mac app](#the-native-mac-app) and its disk image. Until then both bundles are ad-hoc signed so Gatekeeper reports them as unverified rather than damaged, and the first launch is *Open Anyway* as described above. |
| Linux | Nothing to sign for the AppImage; `.deb` / `.rpm` repositories can carry a GPG key. | - | - |
What the unsigned Windows build cannot avoid is the one *More info › Run anyway* on first launch; the SHA-256 on
the downloads page is the check that the file is the published file. The portable exe carries the
same prompt and never self-updates.
## Tray menu
The mark in the system tray or menu bar gains a dot while an answer is streaming. Its menu mirrors the app:
**Open OpenSmartRoute**, **Ask**, the connected workspace or **Sign in to a workspace**, the extension's status, **Settings**,
**Start when I sign in**, **Writing assistant** (on or off), **Dashboard**, **Help**, an update line when one is ready, **Quit**. Double-clicking the
icon opens the main window.
## Privacy
- Nothing leaves this computer until you press Enter, unless you turn on *Price my draft while I
type*. The personal-data scan, redaction and the model-tier check run on the device.
- A request you send goes to your workspace (or, signed out, nowhere - the bar asks you to sign in). The platform
stores the decision and its trace, not the text, unless your workspace has turned request logging on.
- The selection is read by sending Copy to the app in front and reading the clipboard, which is restored
immediately afterwards. It is never read in a terminal.
- Settings, past conversations (`threads.json`, when history is on) and the workspace key live in the app's own
folder - `%APPDATA%\OpenSmartRoute` on Windows,
`~/Library/Application Support/OpenSmartRoute` on macOS, `~/.config/OpenSmartRoute` on Linux. Uninstalling
keeps the folder so a reinstall finds your settings; delete it to remove everything.
- The renderer windows never navigate or open windows themselves: a link in an answer opens in your browser,
anything else is dropped. No telemetry to us; diagnostics are copied by you, never uploaded.
- **Log my activity to the workspace** (Settings › Command bar, on by default) records what the app did as
events on *your* workspace (`POST /api/v1/usage/activity`), posted every few minutes and once more when the app
quits. The first event sends *Your desktop app is connected* to you: what the app does, that the workspace sees
counts only, and what to do if you did not install it. An event is a kind, the kind of app that was in front (`editor.app`, `browser.app` ...), a count and a
figure - never your question, the answer or a window title; the platform refuses a text field of any name. The
kinds: `app.started` / `app.updated`, `bar.opened`, `context.attached`, `pii.detected`, `ask.sent`,
`answer.received` (with the target, request id and latency), `answer.failed`, `answer.cancelled`,
`answer.pasted`, `feedback.good` / `feedback.bad`, `session.signed_in` / `session.signed_out`,
`extension.registered` / `extension.unpacked` per browser. The workspace's Apps page shows them as counts; the
plan decides how long they are kept and how many a day are recorded (PLATFORM.md). Unsent events wait in
`activity.json` and go with an uninstall.
- Every request carries an `X-OSR-App: desktop` header so the dashboard can tell the app from the extension and
the API.
## Troubleshooting
| Symptom | What to do |
|---|---|
| The shortcut does nothing | Another app holds it. Settings › Command bar shows the active shortcut; press new keys there, use the fallback the note offers, or **Reset** |
| *Selected text is not read* | macOS: press **Allow** in Settings (opens the Accessibility pane) and the row ticks itself once granted. Linux X11: install `xdotool`; Wayland: install `wl-clipboard` and `wtype` (Settings says which is missing). Terminals are never read by design |
| The bar closes when I switch to another window | That is the default; pin it (the pin icon, or Settings › *Keep the bar open when I switch windows*). Your draft and thread are kept either way |
| *Enable OpenSmartRoute?* never appears in Chrome or Edge | Restart the browser fully (check the tray for a lingering process). Settings › Browser extension shows what was written per browser |
| The bar says *connect a workspace* | Press Connect and approve the sign-in in the browser; for an on-premises router fill in *On-premises router or a key* |
| The bar says *an on-premises router needs a Pro plan* | The settings name a router other than OpenSmartRoute and this computer holds no licence. Sign in to OpenSmartRoute with a workspace on Pro or higher (the licence is granted on that sign-in), or press **Use OpenSmartRoute** in Settings › Workspace |
| macOS keeps asking to *control System Events*, or the answer is never pasted back | Allow it once (System Settings › Privacy & Security › Automation › OpenSmartRoute › System Events). On a managed Mac ask your administrator for the privacy profile published with the build; a profile from another build or another signing identity does not apply |
| The Windows installer shows *Anyone who uses this computer* greyed out | It was started without administrator rights. Right-click › *Run as administrator*, or install for yourself only - both work the same afterwards |
| No update is offered | Portable, `.deb` and `.rpm` builds, and the macOS build while it is not Developer-ID signed, do not self-update; download the new version from the downloads page. Installed Windows builds and the AppImage: Settings › Updates › Check now |
| The Windows installer closes at once, or Windows reports *OpenSmartRoute-…-win-x64.exe has stopped working* | Installers published before the electron-builder 26 upgrade could crash while choosing the install folder (an over-read in the installer runtime); download the current installer from the downloads page |
| Chrome or Edge refuses the download - *this file isn't commonly downloaded and may be dangerous* | The browser scores installers by how many people have downloaded them, and an unsigned build of a new app scores low. Open the browser's downloads list, choose **Keep**, then **Keep anyway**; compare the SHA-256 with the downloads page before you run it |
| Windows SmartScreen or macOS Gatekeeper warns about the file | The build you have is not code-signed (a deployment signs and notarises its installers once the certificates are configured; the downloads page then drops these steps, and the manifest at `/desktop/release.json` says so under `signed`). Compare the SHA-256 with the downloads page, then allow it (*Run anyway*; on macOS *Open Anyway* under Privacy & Security) |
| macOS says *"OpenSmartRoute.app" is damaged and can't be opened* | The first macOS builds were published without any signature; download the current image again. If it persists, the quarantine flag on the old copy is to blame: in Terminal, `xattr -dr com.apple.quarantine /Applications/OpenSmartRoute.app`, then open it once from Applications |
---
# VS Code extension
The OpenSmartRoute extension for Visual Studio Code puts the router inside the editor. Select code or text and the
**OpenSmartRoute** panel tells you on the spot what it is - the task, the language, how many tokens the selection and
its surroundings weigh, whether it contains personal data - and what the router would do with it: the recommended
model, what the request costs, what the cheapest and the best model would cost instead, and what a month of such
requests adds up to. Then **Ask** a question about the selection, **Explain** it, or **Rewrite** it in place, and the
answer comes back through the router with the model that wrote it, its price and its time under it.
The analysis runs on your computer; nothing leaves it until you ask for a quote or an answer. Personal data -
e-mail addresses, phone and card numbers, IBANs, national ids, IP addresses, API keys - is replaced by placeholders
before a request is sent and put back in the answer, so the model never sees the values.
Works with the hosted platform at [opensmartroute.ai](https://opensmartroute.ai) and with a self-hosted
[`osr serve`](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md). Visual Studio Code 1.90 or later, on Windows, macOS and Linux.
## Install
Download `opensmartroute.vsix` from the [downloads page](https://opensmartroute.ai/downloads) - it is published
there with each release; a deployment that has not published one yet says so on that page - and either drag it
onto the Extensions view or run:
```bash
code --install-extension opensmartroute.vsix
```
Each release is also submitted to the Visual Studio Marketplace (publisher `opensmartroute`) and to Open VSX - the
registry VSCodium, Cursor and Windsurf read - as soon as the deployment holds the store tokens; the downloads page
links the listings once they exist, and the editor then updates the extension by itself.
The compass icon appears in the activity bar and a status item on the right of the status bar. From the
repository: `npm install && npm run package` in `platform/vscode` builds the same file.
## Sign in
Run **OpenSmartRoute: Sign in** from the command palette (or press the button in the panel). A code is shown and
the browser opens the platform's device page; approve the code there while signed in to your workspace and the
editor is connected. The token is kept in VS Code's secret storage (the operating system's keychain), never in a
settings file. **Sign out** removes it.
For a self-hosted `osr serve` or a key you already have, run **OpenSmartRoute: Use a token** and paste it. Set
`opensmartroute.platformUrl` to the server's origin and `opensmartroute.apiPrefix` to an empty string for
`osr serve` (the platform keeps `/api/v1`).
Without a sign-in the panel still analyses the selection and asks for an anonymous quote (rate limited, with the
public catalogue and prices); routing an answer needs a workspace.
## The panel
Open it from the activity bar or with **OpenSmartRoute: Show the routing panel**. It follows the selection as you
make it (`opensmartroute.analyzeOnSelection`) and has four parts:
- **Connection** - the platform you talk to and who you are signed in as.
- **Selection** - the local analysis: the task the text asks for (summarisation, translation, rewriting, code,
extraction, a question ...), the document language and, for prose, the language the text is written in, the
complexity and how much reasoning it needs, the tokens in the selection and in the lines around it, the
domains, whether the text tries to steer the model, and the personal data found - by type and count - with a
note that it will be redacted.
- **Quote** - what the platform answers: the recommended model with the cost of this request and the input and
output tokens it was priced on; under it the cheapest model, the best model and the fastest one when they
differ, each with its cost, expected latency and quality estimate; and a projection for a month of such requests
(`opensmartroute.monthlyRequests`, 1 000 by default). Turn `opensmartroute.quoteOnSelection` off to keep the
analysis entirely local until you run a command; **Quote again** asks on demand.
- **Answer** - the model that wrote the last answer, its cost and time, and two buttons, **Good** and **Not good**,
which send the rating back so the router learns which model to prefer for that kind of request.
The status bar item shows the same in one line - `code · 240 tok · PII → Qwen3 8B $0.0001` - and opens the panel.
## Commands
| Command | Shortcut | What happens |
|---|---|---|
| **Ask about the selection** | `Ctrl+Alt+O` (`⌘⌥O`) | Asks for your question, routes it with the selection and its context, and opens the answer as a Markdown document beside the editor |
| **Explain the selection** | | Same, with a fixed question: what the text does, how, and what is risky about it |
| **Rewrite the selection in place** | | Asks what to change (leave it empty for the default: fix and simplify code, fix spelling and grammar and clarity for prose) and replaces the selection with the model's answer. **Undo** is one keystroke away, and the notification offers Good / Not good / Undo |
| **Analyze selection** | `Ctrl+Alt+Shift+O` (`⌘⌥⇧O`) | Runs the local analysis and asks for a quote now |
| **Sign in** / **Use a token** / **Sign out** | | The connection to your workspace |
| **Show the routing panel** | | Focuses the panel |
| **Rate the last answer** | | Good or Not good for the last answer routed from this editor |
The three writing commands are also in the editor's right-click menu under **OpenSmartRoute** when text is
selected. A request in flight can be cancelled from its progress notification.
## What is sent
A request carries the selection, the lines around it (`opensmartroute.contextLines` above and below, 20 by
default, 0 for the selection alone), the file name and language and the line numbers, and your instruction - as
one prompt in the OpenAI-compatible `/v1/chat/completions` shape, with `model: auto` so the router chooses. The
answer streams back; the longest answer is `opensmartroute.maxTokens` (1 024). The whole request is shown to the
model as text; the extension never sends files it did not select, the workspace tree or other open editors.
With `opensmartroute.redactPii` on (the default) every personal item is replaced before the quote and before the
request, with the same placeholder for the same value everywhere in the prompt, and the answer's placeholders
are turned back into the values on your machine. The panel and the input box say how many items were redacted.
## Settings
| Setting | Default | Meaning |
|---|---|---|
| `opensmartroute.platformUrl` | `https://opensmartroute.ai` | The platform or `osr serve` origin |
| `opensmartroute.apiPrefix` | `/api/v1` | Where the REST routes live; empty for `osr serve` |
| `opensmartroute.redactPii` | `true` | Redact personal data before sending, restore it in the answer |
| `opensmartroute.analyzeOnSelection` | `true` | Analyse the selection as you make it |
| `opensmartroute.quoteOnSelection` | `true` | Ask the platform for a quote while you select |
| `opensmartroute.monthlyRequests` | `1000` | Requests a month for the cost projection |
| `opensmartroute.contextLines` | `20` | Lines above and below the selection sent as context |
| `opensmartroute.maxTokens` | `1024` | Longest answer written back |
## Troubleshooting
- **"sign in to continue"** - the token was removed or expired; run **Sign in** again, or **Use a token** for
`osr serve`.
- **"the workspace is over its quota or rate limit"** - the plan's monthly requests are used up or the anonymous
quote limit was hit; the dashboard's Usage page shows the quota, signing in lifts the anonymous limit.
- **The quote names a model the answer did not use** - the quote is the router's pick before the request; at
request time a busy or unhealthy target is skipped for the next best one, and the answer's line names the model
that actually wrote it.
- **The rewrite left a code fence** - the extension strips a fence that wraps the whole answer; a model that mixes
prose and code into a rewrite is best rated *Not good* so the router stops choosing it for rewrites.
- **Nothing happens on the shortcut** - the shortcut needs focus in a text editor with a selection; another
extension may hold `Ctrl+Alt+O` (change it under *Keyboard Shortcuts*).
- The **OpenSmartRoute** output channel (*View › Output*) logs every failed request with its message.
---
# Word add-in
The OpenSmartRoute add-in for Microsoft Word puts the router in a pane beside your document. Select a passage
and the pane tells you what it is - the kind of writing task, the language, how long it is, whether it carries
personal data - and what the router would do with it: the recommended model, what the request costs, what the
cheapest and the best model would cost instead, and what a month of such requests adds up to. Then **Ask** a
question about the passage, **Explain** or **Summarise** it, or **Rewrite** and **Proofread** it in place: the
answer comes back through the router with the model that wrote it, its price and its time, and the two in-place
actions replace the selection (Word's Undo takes it back).
The analysis runs in the pane, on your computer; nothing leaves it until you ask for a quote or an answer.
Personal data - e-mail addresses, phone and card numbers, IBANs, national ids, IP addresses, API keys - is replaced
by placeholders before a request is sent and put back in the answer, so the model never sees the values.
Word for Windows, Word for Mac and Word on the web (Microsoft 365). Works with the hosted platform at
[opensmartroute.ai](https://opensmartroute.ai) and with your own deployment of the platform.
## Install
The add-in is described by a manifest file the site writes for its own address:
[`/office/manifest.xml`](https://opensmartroute.ai/office/manifest.xml) (a self-hosted deployment serves its own
at the same path - the file names the server it was downloaded from).
- **For yourself (sideload):** in Word on the web open **Insert › Add-ins › More Add-ins › My Add-ins › Upload My
Add-in** and pick the file. In Word for Windows the same dialog takes it (**Home › Add-ins › More Add-ins**); on
a Mac, copy it into `~/Library/Containers/com.microsoft.Word/Data/Documents/wef/` and restart Word.
- **For everyone (administrator):** in the Microsoft 365 admin centre go to **Settings › Integrated apps › Upload
custom apps**, choose *Office Add-in*, point it at the manifest URL above and assign the users or groups. The
add-in then appears in their Word without anything to install, and an update of the manifest reaches them the
same way.
An **OpenSmartRoute** button lands in Word's **Home** tab; it opens the pane. The *Get started* notice Word shows
once explains the three steps: select, read the analysis, ask.
## Sign in
The pane uses the same sign-in as the website, so the dashboard and the pane are one workspace.
- **Sign in** shows a code and opens the browser on the platform's approval page; approve the code while signed
in to your workspace and the pane is connected. The credential stays in the pane's own storage on your computer.
- **Use a key** takes an API key from the dashboard (`osr_live_...`) or the token of a self-hosted server.
- **Sign out** removes it.
Without a sign-in the pane still analyses the selection and asks for an anonymous quote (rate limited, with the
public catalogue and prices); routing an answer needs a workspace.
## The pane
- **Selection** - the local analysis: the task the text asks for (summarisation, translation, rewriting, a
question ...), the language, the complexity and how much reasoning it needs, the words and tokens in the
selection and in the paragraphs around it, the domains, and the personal data found - by type and count - with a
note that it will be redacted. The pane follows the selection as you move it.
- **Quote** - what the platform answers: the recommended model with the cost of this request and the input and
output tokens it was priced on; under it the cheapest model, the best model and the fastest when they differ,
each with cost, expected latency and quality estimate; and a projection for a thousand such requests a month.
- **Ask the router** - a box for your question or instruction (leave it empty for the default of each button) and
the five buttons: **Ask**, **Explain**, **Summarise** show the answer in the pane with **Insert below the
selection** and **Copy**; **Rewrite in place** and **Proofread in place** replace the selection when the answer
is complete. **Cancel** stops a request in flight.
- **Answer** - the model that wrote it, its cost and time, and **Good** / **Not good**, which send the rating back
so the router learns which model to prefer for that kind of writing.
## What is sent
A request carries the selected passage, the paragraphs it touches (context), the document's title, and your
instruction - as one prompt in the OpenAI-compatible `/v1/chat/completions` shape with `model: auto`, so the router
chooses; the answer is at most 1 024 tokens. The pane never sends the rest of the document, other open documents
or anything on your computer. Every personal item is replaced before the quote and before the request - the same
placeholder for the same value everywhere in the prompt - and the answer's placeholders are turned back into the
values in the pane.
## Self-hosting
The pane is part of the platform's web app: a deployment of your own serves it at `/office/taskpane` and writes
its manifest at `/office/manifest.xml` for its own hostname. Office loads the pane inside Word's own frame, so the
web app sends a Content-Security-Policy for `/office/*` that allows Office's hosts as frame ancestors and
Microsoft's CDN for Office.js (the only place add-ins may load it from); nothing else about the deployment
changes. Office requires `https` for everything the manifest names.
## Troubleshooting
- **The pane says "outside Office"** - it was opened in a browser tab. Open it from Word's Home tab.
- **"sign in to continue"** - the credential was removed or expired; **Sign in** again, or **Use a key**.
- **"the workspace is over its quota or rate limit"** - the plan's monthly requests are used up or the anonymous
quote limit was hit; the dashboard's Usage page shows the quota, signing in lifts the anonymous limit.
- **The button is missing from the ribbon** - Word caches add-ins; in Word on the web reload the page, on the
desktop close Word and open it again, and check the manifest was accepted (an upload names any problem).
- **A rewrite changed the formatting** - the replacement is written as plain text with the selection's paragraph
breaks; bold and italic runs inside it are not kept. Undo and select a smaller passage.
---
# User guide
Everything the README leaves out: the full quick-start sequence, running plans end to end, wiring
real providers, the enterprise builder, the research-track modules, routing latency and the
repository layout. API reference material lives in [SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md) (decorator SDK, components,
settings) and [REFERENCE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/REFERENCE.md) (generated; every module and exported name); design in
[ARCHITECTURE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ARCHITECTURE.md).
Contents
1. [Targets](#1-targets)
2. [Routing](#2-routing)
3. [Executing a plan](#3-executing-a-plan)
4. [Real providers](#4-real-providers)
5. [Learning from outcomes](#5-learning-from-outcomes)
6. [Declaring components with decorators](#6-declaring-components-with-decorators)
7. [Enterprise builder](#7-enterprise-builder)
8. [Research-track modules](#8-research-track-modules)
9. [Command line](#9-command-line)
10. [Routing latency](#10-routing-latency)
11. [Package layout](#11-package-layout)
## 1. Targets
Every routable thing is a `RouteTarget` with the same contract: an `id`, a `kind`, declared
`capabilities`, `cost`, `latency_ms`, a `quality_prior`, optional `examples`, `instructions` and a
`handler`. Adding a target never requires retraining; it is routable from its declaration and learns
from its outcomes.
| Kind | What it is | Typical `capabilities` | Wins the primary route? | Executed by |
|---|---|---|---|---|
| `llm` | A model endpoint (optionally per reasoning effort) | `domains`, `min/max_complexity`, `languages`, `context_window` | yes | `chat_handler` / any callable |
| `skill` | A deterministic or narrow capability (SQL, translate, summarise, a SKILL.md package) | `actions`, `domains`, token footprint as `cost` | yes, or fills the `skill` plan slot | your function; `instructions` are disclosed to the model |
| `persona` | System-prompt / behaviour layer (`primary: false`) | `domains`, `actions` | no; fills the `persona` slot on top of a model or agent | `instructions` become the system prompt in `run()` |
| `agent` | An agent harness: coding, research or support runtime with tools, memory and its own loop | `supports_tools`, `tags: [autonomous]`, `actions` | yes | `CallableHarness` / `HTTPHarness` / `SubprocessHarness` |
| `tool` | A single MCP / function tool | `actions`, `input_schema` in metadata | yes | tool call |
| `workflow` | A fixed multi-step pipeline | `actions`, `domains` | yes | workflow engine |
| `human` | Queue, team or expert (Erlang-C capacity maths) | `actions: [escalate]`, `tags: [safety]` | yes; also the abstention target | ticketing / hand-off |
```yaml
# examples/targets.yaml (excerpt)
- id: coding-agent # agent harness: repo access, tests, review loop
kind: agent
capabilities: { domains: [coding], actions: [code_generation, code_review, action],
min_complexity: 0.3, supports_tools: true, tags: [autonomous] }
cost: { usd_per_1k_tokens: 0.008 }
latency_ms: 6000
examples: ["Refactor this class to remove the circular import and run the test suite."]
- id: skill-sql # deterministic skill; `instructions` is its SKILL.md-style body
kind: skill
capabilities: { domains: [data_analysis, coding], actions: [code_generation, extract] }
cost: { usd_per_1k_tokens: 0.001 }
instructions: "Always qualify table names with the schema; never SELECT *."
- id: persona-legal-counsel # never primary; layered on whichever model/agent wins
kind: persona
primary: false
capabilities: { domains: [legal], actions: [reasoning, summarize, qa] }
instructions: "You are cautious in-house counsel. Cite the clause, flag jurisdiction risk."
```
Catalogues load from JSON or YAML (`config.load_targets`), from MCP `tools/list` payloads
(`adapters.tools_from_mcp`), A2A agent cards (`adapters.agent_from_card`), SKILL.md directories
(`adapters.load_skills`), persona folders (`adapters.load_personas`) and vLLM semantic-router
configurations (`adapters.load_semantic_router_config`).
`latency_ms` is the time to an answer whatever its length; a target that answers token by token at a known
pace - a self-hosted model on CPU at five tokens a second, say - also declares `latency_per_token_ms` (200 there),
and the router then expects `latency_ms + latency_per_token_ms x expected output tokens` for *each* request
(`RouteTarget.estimated_latency_ms`): the slow box keeps the one-line classification and loses the long draft,
the hard `max_latency_ms` constraint, the health strategy's SLO fit and the pre-call estimate all read the same
figure. The hosted platform fills both from what it measures (see docs/PLATFORM.md, *Self-hosted models*).
### Agent skills for coding assistants
OpenSmartRoute ships Agent-Skills packages (`SKILL.md`, agentskills.io
format) that teach a coding agent how to use and extend OpenSmartRoute. The product-facing ones -
`osr-routing-catalogue`,
`osr-decorator-sdk`, `osr-enterprise-builder`, `osr-evaluation`, `osr-security-hardening`,
`osr-integrations` and `osr-deploy-serve` - are
published on the documentation site under `/docs/skills/` and on the marketplace; a coding agent loads a folder of such
packages, and they are also valid routing
targets: the same loader the SDK uses (`load_skills`) validates them and the router fills the
plan's skill slot with the right one.
```bash
osr skills # validate + list the SKILL.md packages in a folder
osr -t examples/targets.yaml --skills route "Create a JSONL eval dataset and gate CI on accuracy" --plan
# plan: llm:llm-mid -> skill:osr-evaluation
```
## 2. Routing
```python
from opensmartroute import Router, RouteRequest, RequestConstraints, Objective
router = Router(registry)
# plain text
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, f"{d.confidence:.2f}")
print(d.trace.explain()) # per-strategy scores and rationales
# hard constraints and a per-request objective
req = RouteRequest(
text="Summarize this patient intake note.",
constraints=RequestConstraints(region="eu", data_boundary="private", max_cost_per_1k=0.005),
objective=Objective(quality=1.0, cost=0.5, latency=0.1, quality_floor=0.6),
)
d = router.route(req)
print(d.trace.policy_rejections) # {'llm-frontier': 'cost 0.015 > budget 0.005', ...}
# a composed plan: persona -> skill -> model (+ the tools the model may call)
d = router.route("Review this NDA clause for GDPR liability.", plan=True)
for slot in d.plan.slots:
print(slot.role, slot.target.id, f"{slot.confidence:.2f}")
```
Policy runs before scoring and is never traded off: enable flags, allow / deny lists, kind, region,
data boundary, PII, tenant, cost / latency / token SLOs, modality, language and jailbreak risk.
Strategies then score the survivors and a confidence-weighted ensemble produces a quality estimate;
utility is `w_q * quality - w_c * norm(cost) - w_l * norm(latency)` with a hard `quality_floor`.
With a `ConformalCalibrator`, `RouteDecision.candidate_set` is the smallest set with the requested
coverage and `abstain` is set when no target is safe enough.
What the signals read out of a request is an ontology of twenty-five task types in seven families (`signals.TASK_TYPES`;
`osr route` prints the detected domains and actions on its `signals:` line): the transformations (summarisation, translation, rewriting - which
includes the editing help the extension and the desktop bar send), analysis (extraction, reading a scanned or
photographed document - `document_intelligence`, classification, factual questions, `web_search` for a lookup on
the live web, embeddings), reasoning (explanation, maths, planning, `observability` - the root cause of an incident
from logs, traces, metrics and alerts), coding, generation (creative writing, images, speech), interaction (chit-chat,
support, escalation) and action (`tool_action`, `browsing` - open the page, log in, fill the form, click through,
download - and `desktop_action` - rename, move, paste, type into the window in front). Each type maps to the actions
targets declare (`search`, `ocr`, `browse`, `action` ... - `signals.ACTION_LEXICON`) and requests carry up to three
domains (`signals.DOMAIN_LEXICON`: coding, finance, legal, medical, observability, science, education ...). A human
target is scored as a last resort: unless the request asks to escalate its capability fit is halved
(`capability.human_fallback_penalty`, `OSR_CAPABILITY_HUMAN_FALLBACK_PENALTY`), so a queue with a high quality prior
never wins a plain question on that prior alone.
A generative judge can be consulted only when the ensemble is unsure:
```python
from opensmartroute import LLMJudgeStrategy
router = Router(registry, llm_judge=LLMJudgeStrategy(llm=lambda prompt: my_llm(prompt)), escalate_llm_judge_below=0.6)
```
## 3. Executing a plan
`route()` decides; `run()` decides and executes the whole plan: skill pre-processors run first,
persona and skill `instructions` are composed into the system prompt, the primary target is called,
and an `Outcome` is recorded for every participant so each one learns from the result.
```python
from opensmartroute.adapters import (
CallableHarness,
HTTPHarness,
SubprocessHarness,
chat_handler,
harness_handler,
load_skills,
)
registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o")
registry.get("coding-agent").handler = harness_handler(
SubprocessHarness(["my-coding-agent", "--repo", "."], system_flag="--system")
)
registry.get("research-agent").handler = harness_handler(
HTTPHarness("https://agents.internal/research/run", api_key_env="AGENT_TOKEN")
)
registry.get("skill-sql").handler = lambda req: lookup_schema(req.text) # runs BEFORE the model
for skill in load_skills("examples/skills"): # SKILL.md folders
registry.add(skill)
res = router.run("Is this indemnification clause enforceable in California?", task_id="ticket-812")
res.text # the primary target's answer
res.steps # [persona:persona-legal-counsel, llm:llm-frontier] with per-step latency
res.system_prompt # composed persona + skill instructions
res.outcomes # one Outcome per participant, role="persona"/"skill"/None, shared task_id
```
A skill handler may return a `RouteRequest` (rewrite the request), a `str` (attached as
`context["skill_output"]` and shown to the model) or `None` (instructions only). Async handlers work
through `AsyncRouter.run()`; `EnterpriseRouter.run()` adds health, budgets and telemetry. See
examples/end_to_end.py (`examples/end_to_end.py`) and [SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md#executing-end-to-end).
### Tools the model may call
A model or agent plan also carries `tool` slots: the tool targets (MCP tools, `@tool` handlers) whose
ensemble quality for *this* request clears `RoutingSettings.tool_quality_floor` (default 0.5; measured on a live
catalogue, a tool whose examples match the request scores 0.53-0.75 and an unrelated one stays below 0.47 - the
model still decides whether to call it, so an offered-but-unused tool only costs its spec tokens), at most
`tool_slots` of them (default 2; `Router(reg, tool_slots=0)` or `OSR_ROUTING_TOOL_SLOTS=0` turns the slot
off). `run()` offers them to the model as OpenAI function-calling specs (`execution.tool_spec`: the tool's MCP
`input_schema`, or a single `input` string for handler-only tools) through `context["openai"]["tools"]`.
When the model calls one, the tool target's handler runs (`context["tool_arguments"]` = the parsed arguments,
or the `input` argument as the request text), the result goes back as a `tool` message and the model is asked
again - up to `RoutingSettings.tool_rounds` rounds - so the caller receives the finished answer. A request that
brings its own `tools` keeps its own tool protocol (nothing is added), and a call to a tool the router did not
offer is handed back untouched. `res.steps` shows `tool … offered` / `called` entries and every tool that was
called is credited with a `role="tool"` outcome, so the learners find out which tools actually help.
A skill or tool slot is only filled when the request *names* the companion, whatever its quality score: an
action it declares (a translation for the translate skill), one of its domains, a need for fresh facts for a
tool tagged `web`, or a text close to its own examples (`RoutingSettings.companion_similarity_floor`, default
0.7 on the `similarity` strategy's score). A greeting or a bare word names nothing and gets a model and, at most,
a persona - a companion is a request of its own, and its record on other requests is no reason to run it here.
The trace shows a slot left empty for this reason as `plan.slot … "nothing in the request names one"`.
```python
res = router.run("How many orders shipped last week?")
[(s.role, s.target_id, s.note) for s in res.steps]
# [('tool', 'warehouse:sql_query', 'offered'), ('tool', 'warehouse:sql_query', 'called'), ('llm', 'llm-frontier', 'primary')]
```
## 4. Real providers
The bundled client is stdlib only and speaks the OpenAI chat and embeddings API, which covers OpenAI,
Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq and Mistral.
```python
from opensmartroute import Router, LLMJudgeStrategy
from opensmartroute.adapters import OpenAICompatClient, chat_handler, embedder, judge_fn
from opensmartroute.strategies import CapabilityStrategy, SimilarityStrategy
openai = OpenAICompatClient() # OPENAI_API_KEY from env
ollama = OpenAICompatClient("http://localhost:11434/v1", api_key="ollama")
registry.get("llm-frontier").handler = chat_handler(openai, model="gpt-4o")
registry.get("llm-small").handler = chat_handler(ollama, model="llama3.1")
router = Router(
registry,
strategies=[CapabilityStrategy(), SimilarityStrategy(embedder=embedder(openai, model="text-embedding-3-small"))],
llm_judge=LLMJudgeStrategy(judge_fn(openai, model="gpt-4o-mini")),
escalate_llm_judge_below=0.5,
)
d = router.route(req)
result = router.execute(d, req) # ChatResult(text, prompt_tokens, completion_tokens, latency_ms)
res = router.run(req) # route + execute the full plan
```
`adapters.sentence_transformers_embedder()` and `adapters.OpenTelemetryTelemetry` are lazy imports
behind the `embeddings` and `otel` extras. `osr serve` exposes a FastAPI app with an OpenAI-compatible
`/v1/chat/completions` proxy that routes, executes and learns.
`adapters.AnthropicClient` speaks the Messages API natively behind the same `chat` / `stream_chat` surface and
places **prompt-cache breakpoints** on its own when the caller sent none: on the tool list, on a system prompt
long enough to be cached and on the conversation prefix (the user turn before the current one), so an agent loop
that resends the same tools and history pays Anthropic's cache-read rate for them instead of the input rate;
``prompt_cache=False`` turns it off, markers the caller placed are respected, and the cached and written token
counts come back in ``usage`` (``prompt_tokens_details.cached_tokens``, ``cache_creation_input_tokens``).
`adapters.MockClient` is the provider for tests and demos: deterministic answers without a network, streaming,
embeddings and failure on cue (`OSR_LLM_BASE_URL=mock://` gives every LLM target of `osr serve` one).
### The OpenAI-compatible proxy
`osr serve` (`opensmartroute.server.create_app`) is a drop-in base URL for the official OpenAI SDKs: point a
client at `http://router:8000/v1` and keep the rest of the code.
```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="osr_local_...") # the token from `osr token generate`
client.chat.completions.create(model="osr/auto", messages=[{"role": "user", "content": "..."}])
```
- **Model names.** `osr/auto` (or `auto`) lets the router choose, `osr/auto:cheap|fast|quality|private|agentic|llm`
presets the objective, constraints or kinds, and any target id pins that target. `models: ["llm-a", "llm-b"]`
names the candidates: the router picks among them and, when the chosen handler fails, falls back down the list
(`osr.fallbacks: false` turns that off; `OSR_SERVER_MAX_FALLBACKS` caps the re-routes). Every failed attempt is a
failed outcome for the learners and health breakers; the response's `opensmartroute.fallback_from` lists them.
- **Streaming.** `stream: true` answers with `chat.completion.chunk` server-sent events: a role chunk, one content
delta per provider token (relayed while the upstream is still generating), tool-call deltas, then a final chunk
with `finish_reason`, `usage` and the `opensmartroute` metadata, then `data: [DONE]`. Errors before the first
token keep their HTTP status (404 unknown model, 422 no admissible target, 503 provider down); the response
headers `X-OSR-Target` and `X-OSR-Request-Id` name the decision. A handler that does not stream is chunked after
it returns (`OSR_SERVER_STREAM_CHUNK_CHARS` characters per delta). The response is held for the first token for
at most `OSR_SERVER_STREAM_FIRST_TOKEN_WAIT_S` (15 s); a model still silent after that - a cold load, a long
prompt on a CPU, a queue - gets its stream opened anyway and an SSE comment (`: keepalive`) every
`OSR_SERVER_STREAM_KEEPALIVE_S` (10 s) until it speaks, so an ingress idle timeout never cuts a slow answer and
the client sees the request is alive; a failure after that point arrives in-band as an `error` event with the
status it would have had.
- **Tools and passthrough fields.** `tools`, `tool_choice`, `response_format`, `stop`, `n`, `seed`, `logprobs`,
`top_logprobs`, `parallel_tool_calls` and `user` reach the target's handler through
`request.context["openai"]`; `chat_handler` forwards them to the upstream untouched. Assistant `tool_calls`
and `tool` messages are kept in the conversation, and the response carries `message.tool_calls`,
`message.refusal` and the real `finish_reason` (`tool_calls` when the model called tools). A request that sends
`tools` only routes to LLM targets that declare `capabilities.supports_tools: true`.
- **Prompt compaction.** Before any model or agent handler runs, `execution` compacts the prompt
(`opensmartroute.compress`, `OSR_COMPRESSION_*`, on by default): trailing whitespace, zero-width characters and
runs of blank lines go, consecutive duplicate lines of 24+ characters (pasted logs, repeated boilerplate) collapse
to one, and the same is applied to the history and the composed system prompt. Fenced code blocks are kept
byte-for-byte and no line is rewritten, so the model reads the same words with fewer tokens.
`OSR_COMPRESSION_HISTORY_BUDGET_TOKENS` adds a conversation budget: the oldest turns are dropped until the
history fits, never splitting an assistant `tool_calls` from its `tool` results and never touching the newest
`OSR_COMPRESSION_KEEP_LAST_TURNS` turns. The estimated saving is reported as `tokens_saved` in the response
metadata, `ExecutionResult.tokens_saved` and a `compress` step (`-N tokens`); `OSR_COMPRESSION_ENABLED=0` turns
it off. Tool, workflow and human targets receive the request verbatim. Token estimates come from
`estimate.estimate_tokens`, calibrated against billed gpt-4.1 prompt tokens on prose, chat, logs, code, JSON,
Markdown, URLs, numbers, German and Japanese (mean ratio 1.02, every type within about 20 %).
`examples/precall_check.py` re-measures the whole story against a live endpoint - routing, the tool slot, the
compaction - and prints real tokens, cost and correctness next to an always-frontier baseline.
- **Multi-part content.** Text parts of the current turn join into the routed text; `image_url`, `input_audio` and
`file` parts land in `context["attachments"]` (and `images` / `audio` / `files` for the modality signal, so an
image only routes to a target whose `capabilities.modalities` include `image`). `chat_handler` sends them on
as multi-part content; earlier turns keep their text.
- **Embeddings.** `POST /v1/embeddings` (`input: str | list[str]`) routes `osr/auto` among LLM targets tagged
`embeddings` (`capabilities.tags`) or with `metadata.embeddings: true`; a target id pins one. The target's handler
receives `context["embeddings_input"]` and returns `{"embeddings": [[...]], "model": ..., "prompt_tokens": n}` -
`adapters.openai_compat.embed_handler(client, model)` does exactly that for any OpenAI-compatible endpoint, and
`OSR_LLM_BASE_URL` attaches it automatically to embeddings-tagged targets (the example catalogue ships one,
`embed-small`). Embeddings-only targets are never chosen for a chat completion.
- **Other dialects.** `POST /v1/responses` (the OpenAI Responses API, `opensmartroute.responses`) and
`POST /v1/messages` (the Anthropic Messages API, `opensmartroute.anthropic`) are translated onto the same chat
path, so the OpenAI Agents SDK, the Anthropic SDKs and Claude Code use the router as their base URL unchanged:
content blocks, tools, tool results, stop sequences and the streaming events of each dialect are mapped both
ways, the `opensmartroute` block rides along, and `POST /v1/messages/count_tokens` answers from the router's
token estimate without calling a provider.
- **`GET /v1/models`** lists `osr/auto`, its variants and every executable primary target: those with a handler,
plus instructions-only skills and personas when a model with a handler can run them.
- **Async handlers.** Handlers may be `async def`; the server runs routing and execution in worker threads and
drives coroutine handlers to completion, so sync and async targets mix freely. Handler keywords such as
`on_delta` are only passed to handlers whose signature accepts them (`on_delta` is also in `request.context`).
- **Token scopes.** An entry in `--token` / `OSR_SERVER_AUTH_TOKENS` may be `` (full access), `:ro`
(read-only: every `GET`, `/route`, `/v1/*` and `/mcp`, but 403 on `/feedback` and `/autopilot/cycle`) and carry
an expiry `:exp=` (401 once passed). `GET /whoami` reports the scope of the presented token.
The same building blocks are public in the SDK: `execution.execute_with_fallback` (route -> execute -> re-route on
failure), `execution.ChatStream` / `completion_chunks` (OpenAI chunk events from any handler) and
`server.proxy_request` / `proxy_response` / `embeddings_response` (the body conversions).
This is tested against the real clients, not a look-alike: the test suite starts an `osr serve`
process on the example catalogue and drives it with the official `openai` SDK (models, chat, streaming, a tool
call, embeddings, a rejected token) and the official `mcp` SDK (`POST /mcp` over Streamable HTTP, and `osr mcp`
over stdio).
## 5. Learning from outcomes
```python
from opensmartroute import Outcome
router.learn(
Outcome(
request_id=d.request_id,
target_id=d.target.id,
success=True,
quality=0.9,
cost_usd=0.002,
latency_ms=1800,
domains=d.trace.signals.domains,
)
)
```
One `Outcome` fans out to every learner: Thompson bandits per domain, IRT ability and difficulty,
Bradley-Terry preferences, LinUCB on the signal vector, the Markov / MDP lookahead, the task table and
the personalisation posteriors. `Outcome.task_id` and `step` join delayed, task-level rewards to the
routing decisions that produced them (`learning.TaskCredit`, `learning.TaskPins`). Page-Hinkley
drift detection flags targets whose success rate degrades; every learner forgets with a configurable
`decay` and can be merged across replicas (`learning.merge_learners`). Each `learn()` also records a
`learn.outcome` event on the request's trace and, with `with_audit(sink, outcomes=True)`, an `outcome`
record in the audit chain, so a decision and its result stay joined by `request_id`
([OBSERVABILITY.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/OBSERVABILITY.md); on the hosted platform `POST /api/v1/feedback` does the same).
Offline warm starts: `learning.warm_start_from_matrix()` replays a `(prompt, {target: reward})`
matrix, `learning.warm_start()` copies posteriors from the nearest declared neighbours and
`osr train --from rows.jsonl --out models.json` fits the hashed signal models.
### Implicit feedback: learning from the request and the response
Almost no traffic calls `/feedback`, so the router also reads the evidence it already has
(`learning.implicit`, settings group `implicit`):
- **The answer itself.** Whenever `execute()` (and with it `osr serve`'s `/v1/chat/completions`, `/route`
+ `/execute`, the MCP tool) runs an LLM, agent, skill or persona target whose handler reports no
`quality` of its own, `learning.assess_response` judges the response against the request: a refusal
(`message.refusal`, `finish_reason: content_filter`, "I cannot ..."), an empty answer, a truncated one
(`finish_reason: length`), hedging and self-correction phrases, a coding request answered without any code,
and a call slower than `OSR_IMPLICIT_SLOW_FACTOR` times the target's typical latency all lower the
outcome's `quality` (`OSR_IMPLICIT_REFUSAL_QUALITY`, `OSR_IMPLICIT_TRUNCATED_QUALITY`, ...). An answer with
nothing wrong gets `OSR_IMPLICIT_NEUTRAL_QUALITY` (0.7) - good enough to learn from, not the 1.0 that a bare
`success=True` used to mean, which had every learner reinforcing whatever the router picked simply because
the call returned. `Outcome.source` records the verdict (`response:refusal`, `response:ok`, ...).
- **The next turn.** With `context["session_id"]` (the proxy's `session_id` / `metadata.session_id`),
`learning.ReactionLearner` reads each new turn of a session as feedback on the previous one: a complaint
("that's wrong", "try again", "no, I meant ..."), a near-identical re-ask (token Jaccard >=
`OSR_IMPLICIT_RETRY_SIMILARITY`, a retry) or thanks become an outcome for the previous request's target
(`reaction:negative` 0.2, `reaction:retry` 0.3, `reaction:positive` 0.95) within
`OSR_IMPLICIT_REACTION_WINDOW_S`. The router feeds it through `learn_correction()`, so it reaches every
learner - including the SLM's online step, which now accepts a later correction of a request it already saw.
- **Into the self-improvement loop.** `eval.rows_from_feedback` reconciles several outcomes for one request by
trust (reaction > explicit feedback > response assessment) and drops `response:ok` outcomes - "it did not
fail" says nothing about which target was best, and training on it would only teach the SLM its own
policy. A row that scores a single target *badly* (`learning.negative_only`) trains as "not this one" - a
push towards every other target - in both objectives; it used to become a full positive for the target it
complained about, because a softmax over one score is 1.0.
`OSR_IMPLICIT_ENABLED=false` returns to plain success/failure outcomes; `OSR_IMPLICIT_REACTIONS=false` turns
off the next-turn reading. A handler that returns its own `quality` (a judge, an exact-match check, an agent
harness) is always believed over the assessment.
### The routing SLM and the self-improvement loop
`learning.RouterSLM` is a small routing model in one JSON file: a hashed dual encoder
(`SLMSettings.dim` x `feature_dim`), a two-stage calibrator and a snapshot of every target's price,
latency and quality prior. It has no dependencies, predicts in microseconds and can be trained from
any `EvalRow` source - labelled datasets, your own outcomes, synthetic ontology prompts or the full
router's decisions (`distill_router`, which compresses rules + capability fit + learners + judge into
one model). Calibration fits the softmax temperature on the validation rows and, from 40 rows up, also a
monotone (isotonic) map from the tempered top probability to the observed hit rate, shrunk toward the tempered
value while the calibration set is small (`n / (n + 100)`): a single temperature cannot fix a model that is right
at 0.6 as often as at 0.9, and the ensemble weighs the SLM by the probability it reports. Measured out of
sample on the example catalogue (calibrate on half the holdout, score the other half, ~120 rows): ECE 0.07-0.14
-> 0.03-0.09 and Brier down a third for both encoders, at unchanged accuracy; the ranking never changes (the
winner stays the winner).
```python
from opensmartroute import RouterSLM, SLMStrategy, Router, RouteRequest
from opensmartroute.eval import DatasetCollector, synthetic_rows
from opensmartroute.learning import distill_router
slm = RouterSLM(reg.all(), seed=0)
slm.fit(synthetic_rows(reg.all())) # no data yet: seed prompts scored by capability fit
slm.fit(DatasetCollector("data").corpus()) # collected battle datasets (RouteLLM, LMArena)
student, report = distill_router(Router(reg), texts) # or copy the whole ensemble
print(report.summary()) # accuracy, loss, ECE, Brier, cost/1k
slm.save("slm.json")
router = Router(reg, strategies=[SLMStrategy(RouterSLM.from_file("slm.json"))])
d = router.route(RouteRequest(text="prove that sqrt(2) is irrational"))
print(slm.predict(d.request.text, objective=Objective(cost=2.0))[:3]) # calibrated p, quality, $/1k, utility
```
`SLMStrategy` sits in the ensemble under weight `OSR_WEIGHTS_SLM` and, like the other learners, updates
the model online from every `router.learn(Outcome)`. Its scores and confidence are read from the SLM's
*calibrated* probabilities (the fitted temperature), the same numbers `osr slm predict` prints.
#### Slot models: which persona, skill or tool suits the request
The primary encoder answers one question - which target *answers*. A plan asks more: which persona, which
skill, which tools. Those decisions get their own small encoders, the **slot models** (`RouterSLM.slots`, one per
role in `OSR_SLM_SLOT_ROLES` - `persona`, `skill`, `tool`), trained on the outcomes credited to that slot
(`Outcome.role`; `eval.rows_from_feedback(..., role="skill")`) and saved in the same file, so a promotion or a
rollback moves them with the primary. `SLMStrategy` reads the slot the router is filling
(`signals.extra["role"]`) and votes with that role's model - a marketplace skill with a record is learned exactly
like a catalogue one - and abstains for a role it has no model for; slot outcomes never enter the primary corpus
(a persona is not an answer), and the online update of a planned request keeps the primary's and each slot's
decision apart. The improve cycle trains a slot once it has `OSR_SLM_SLOT_MIN_ROWS` (20) rows and reports each
role under `slots` (rows, accuracy against the constant predictor, skill); `RouterSLM.info()["slots"]` and
`osr slm info` show what is served.
```python
rows = rows_from_feedback(feedback, texts, role="persona") # the persona slot's outcomes, prompt by prompt
report = slm.fit_slot("persona", rows) # its own encoder and calibration
slm.slots["persona"].predict_proba("review this contract clause", slm.slot_candidates("persona"))
```
#### Conversation context
A follow-up says nothing about the task on its own: "why?", "shorter please", "and what about the edge cases?"
after a proof, a translation or a code review each belong with the target that served the turn before. The
SLM reads such a message together with the last `OSR_SLM_CONTEXT_TURNS` (2) user turns of `RouteRequest.history`
(`learning.routing_text`: `" || ..."`, at most `OSR_SLM_CONTEXT_CHARS` (400) of
context, newest first) - `SLMStrategy` does it for every request, `RouterSLM.predict*(text, history=...)` for a
direct call, `EvalRow.history` for training and evaluation rows. Only a *follow-up* gets context
(`learning.is_followup`): a message of at most `OSR_SLM_CONTEXT_MAX_WORDS` (3) words, or one that opens with a
continuation (`learning.slm.FOLLOWUP_RE`: "and", "now", "instead", "try again", "can you elaborate" ...) and stays
under `OSR_SLM_CONTEXT_MAX_TEXT_CHARS` (240). "Translate this paragraph into French" after a coding chat names
its own task and routes on its own words. Context words hash under their own `ctx:` namespace
(`signals.models.CONTEXT_MARK`, weight `CONTEXT_WEIGHT` 0.7; words and bigrams only, in the attention block
too), so "python" said earlier and "python" said now are different features and a model trained without
conversations reads unchanged. `eval.synthetic_followup_rows` builds follow-up conversations (and, with
`switches=`, topic switches) from any labelled rows for training; `explain()` reports the text the encoder
actually read and names context features `ctx:w:...`.
#### The online update is reinforcement learning
An outcome is a reward for the routed target, not a label: the served model learns from it with a REINFORCE
step on its own softmax (`OSR_SLM_ONLINE_POLICY=reinforce`, the default) - the reward's *advantage* over the
target's running mean reward (`OSR_SLM_REWARD_BASELINE_RATE`, the baseline starts at 0.5) raises or lowers the
target's log-probability, so the first good answer of a new target is a surprise worth learning from and the
hundredth is not, and a bad outcome moves probability to the other candidates in proportion to what the model
already gives them. Because the *router* picked the target, not the SLM, the step is importance-weighted by
`p_SLM(target) / propensity` (capped at `OSR_SLM_IPS_CLIP`, 5): `Router.learn` fills `Outcome.propensity` from
its request memory when the caller did not, the same logged propensity `eval.ope` uses, so traffic routed by
the whole ensemble trains the SLM without bias. `OSR_SLM_ONLINE_POLICY=supervised` is the plain label update (a
good outcome is a positive, a bad one spreads the label over the others). `SLMStrategy.status()` reports
`online_policy` and `last_advantage`; the same REINFORCE with a learned baseline also drives the stand-alone
`learning.PolicyGradientStrategy` over hashed features.
#### The bundled model
The package ships a trained model, so an on-prem or air-gapped install routes with a learned SLM before it
has seen a request: `RouterSLM.bundled()` (or the path `"bundled"` anywhere a model file is accepted -
`--slm bundled`, `osr slm info bundled`, `RouterBuilder.with_slm()`, `OSR_SLM=bundled` in the container,
`OSR_PLATFORM_SLM=bundled` on the platform). It is an attention-encoder model trained
on the reference catalogue (`examples/targets.yaml`, 27 targets) and labelled by the **reference ensemble** -
rules, capability fit, cost and quality priors together - rather than by the capability fit alone, whose
near-ties between generalists had sent a quarter of everyday prompts to the on-prem model. Its corpus is the
ontology's seed templates over three seeds plus the **everyday phrasings** of every task type
(`signals.corpus.NATURAL_PROMPTS`: terse chat, office English, pasted context, typos, a second language), the
multicloud benchmark prompts routed by the ensemble, one follow-up conversation per row, the 70 labelled example
rows with their follow-ups - weighted ten times so ground truth outvotes near-tied synthetic labels - and 64 bare
follow-ups with a flat label; a 128-token attention window, learning rate 0.08. The labelled rows cover what the
browser extension's writing help, the desktop bar and the newsroom's writer actually send (grammar fixes, shorter
drafts, e-mail replies, subject lines, voice memos, read-aloud, embeddings, release notes) and what the new agents
and tools are for (search the web for..., OCR this receipt, root-cause this alert, open the portal and download
the statement, rename the files in this folder).
It is measured two ways. The **probe holdout** - a fourth seed's templates and a fifth of the everyday phrasings
the probe never trained on - reads 0.77 (follow-ups 0.76, topic switches 0.77; a template-only holdout had read
0.97 of a model that routed unseen wording at 0.49). The **held-out set** (`examples/eval_heldout.jsonl`, 117
everyday asks naming every target, in none of the training data) reads 0.73 for the model alone, and the
ensemble with it 0.82 against 0.84 without - which is why `OSR_WEIGHTS_SLM` defaults to 0.5 and the strategy
reports its calibrated top probability as its confidence, scaled by that held-out skill (`meta.heldout`). On
the labelled set the model alone routes all 70 rows acceptably and the ensemble with it 69 of 70; ~1.8 MB as an
int8 file. The learning rate matters more than the seed: at the
attention encoder's default 0.2 the holdout swung between 0.71 and 0.88 from one epoch to the next and the file
depended on where early stopping landed, at 0.08 every epoch improves on the last, so the recipe is reproducible -
the repository's training script rebuilds the shipped file bit for bit. The file also carries the plan's **slot models**: a
`persona` slot and a `skill` slot (`RouterSLM.slots`, trained by `fit_slot` on the same seed prompts scored for
every persona and skill of the catalogue plus the labelled rows that name one), so `plan=True` ranks *which*
persona speaks and *which* skill instructs with a learned model from the first request - 0.99 holdout accuracy on
both slots. Its targets are the reference ids. Its calibration - temperature plus the isotonic stage -
is the one fitted on rows the model never trained on (the script's probe fit, everyday phrasings included), not
on its own training rows, so the probability it reports is honest; the final fit runs for the number of epochs
the probe's early stopping chose. Your own targets are added when it is loaded; the ensemble hears the model only
on targets it has compared, so with a different catalogue it starts silent and `osr improve` / the autopilot
- which now start *warm* from it whenever no model file exists yet - train a challenger that knows your
targets on the first cycle (the file records `origin: bundled` through every promotion). Retrain the shipped
file with the script; never edit it by hand.
```python
from opensmartroute import RouterSLM
from opensmartroute.enterprise import RouterBuilder
app = RouterBuilder(reg).with_defaults().with_slm().build() # the packaged model, your targets added
print(RouterSLM.bundled().info()["holdout"]) # what it was measured at
```
The package ships the hand-labelled set the bundled model was measured on next to it
(`opensmartroute/data/labelled.jsonl`, a copy of `examples/eval_dataset.jsonl`), and `RouterSLM.labelled_check()`
scores any model on it - on its own, on the rows whose targets the model can choose between. That is the
**labelled floor** of the self-improvement loop (`SLMSettings.labelled_floor`, on): a holdout is self-agreement
with the router's own decisions on your traffic, and a deployment whose traffic is one kind of job can promote
itself into a model that is right on that job and wrong on everything else while every holdout reads 0.95. A
challenger that routes fewer of the applicable labelled rows acceptably than the champion - beyond
`labelled_tolerance` (5 %) of them - is refused whatever its holdout says (`ImprovementReport.labelled` carries
both scores), and `SelfImprover.floor_check(RouterSLM.bundled())` holds a served champion to the same floor
against the packaged model, adopting it when the champion fell below. Fewer than `labelled_min_rows` (20)
applicable rows - a catalogue of your own - switch the floor off. The floor is also *rehearsed*: the applicable
labelled rows train every challenger next to your traffic, replicated to about `labelled_rehearsal` (25 %) of its
training rows, so a challenger that learns one kind of traffic keeps the reference behaviour instead of being
refused every cycle (the holdout stays your own rows; `ImprovementReport.sources["labelled"]` counts them).
Data comes from three collectors that all degrade gracefully when offline:
- `adapters.ModelCatalogue` merges OpenRouter prices and Hugging Face model cards (downloads, likes,
`model-index` benchmarks) into `ModelCard`s, turns them into `RouteTarget`s (`targets(min_quality=,
max_usd_per_1k=, measured=True)`), prints a cost/quality benchmark and the Pareto frontier.
`refresh(leaderboard=True)` adds Open LLM Leaderboard accuracies (IFEval, BBH, MATH, GPQA, MuSR,
MMLU-Pro) as measured quality; `apply_quality()` takes the Bradley-Terry win rates that
`eval.model_quality` derives from collected battles. Third-party descriptions are risk-scored and
replaced when they look like injections.
- `eval.DatasetCollector` pulls routing datasets from the Hugging Face datasets-server into a JSONL cache.
Fifteen public sources are on by default: the pairwise battles `routellm-battles` (GPT-4-judged), `arena-55k`,
`arena-100k`, `arena-140k` (LMArena human votes, with the code / language / math / hard-prompt tags),
`ppe-human`, `webdev-arena`, `mt-bench-human` and `reward-bench`, the 2025 releases `arena-expert-5k` (domain
experts judging 2025 models; the conversation column is a Python repr, parsed as such), `search-arena-7k`
(search-augmented models on live questions - the names carry their own tier rules), `repochat-arena`
(answers over a repository) and `llama3-70b-battles`, the per-model judge scores of `ultrafeedback`, and
RouteLLM's `routellm-gpt4` ("was Mixtral good enough for this prompt") and `routellm-mmlu` battles.
RouterBench / RouterEval / the raw arena conversations are known but gated: download them with your own
Hub token and load the file with `osr collect --file routerbench=routerbench.jsonl --preset routerbench`
(`.jsonl` / `.json` / `.csv` in the preset's shape, `--tier` folds model columns onto your targets), and any
dataset works through `DatasetSource`. Battles name a moving population of models, so `--tier frontier=llm-frontier --tier
mid=llm-mid --tier small=llm-small` folds every side onto your tiers as it is parsed (`tier_of()`: the
`ARENA_TIERS` table, name rules for the 2024-26 generations, then the parameter count in the name) - the
RouteLLM question "was the cheaper tier good enough?". Collection is polite and resumable: pages are paced,
429s are retried with backoff, and a run the server cuts short keeps its rows and continues from the
recorded offset on the next `osr collect` (`.meta.json`). `rows_from_feedback` turns your
`FeedbackStore` into labelled rows.
- `adapters.WebKnowledge` searches Hugging Face (text-generation repos, per-keyword fallback), Wikipedia and
DuckDuckGo (a self-hosted SearXNG with `SEARXNG_URL`, Brave with `BRAVE_API_KEY`) for new models, datasets and
live context; fetches are https-only,
byte-capped and page text is risk-scored. `read_feeds(urls)` reads RSS / Atom **article feeds** - vendor
blogs, release notes, arXiv listings (`adapters.parse_feed`, stdlib XML, newest `OSR_SLM_FEED_MAX_ENTRIES`
entries within `OSR_SLM_FEED_MAX_AGE_DAYS`) - so the loop hears about a model launch the day it is
announced, not when a benchmark gets around to it.
`learning.SelfImprover` closes the loop as a champion/challenger cycle: refresh the catalogue, discover,
gather evidence, split, train a challenger, calibrate it on the holdout and promote it only when it really
wins there. Articles feed the *discover* step (`feeds=` / `--feed URL` / `OSR_SLM_FEEDS`): each cycle counts
which catalogue models the articles mention (`ImprovementReport.mentions`), looks up Hugging Face cards for
model names the catalogue has never seen (`gemini-2.5-pro` announced in a post becomes a card lookup and a
`discovered_models` entry) and lets priced cards the articles talk about join the candidate targets
(`trending`) even before a leaderboard has measured them - the evidence rows still decide whether the
challenger learns to name them, and the promotion gates below still apply, so an article can make a model
*eligible*, never *preferred*. Three gates stand between a challenger and production: the holdout must hold at least
`OSR_SLM_PROMOTE_MIN_HOLDOUT` rows (a four-row holdout cannot judge anything); holdout accuracy must improve
by at least `OSR_SLM_MIN_GAIN` (or loss must drop at equal accuracy); and the rows the challenger newly gets
right must outnumber the rows it newly gets wrong with a one-sided exact binomial p-value of at most
`OSR_SLM_PROMOTE_ALPHA` (`learning.paired_test`, McNemar's discordant pairs - two flipped rows out of a
hundred are noise, not a win; `1.0` turns the test off). The replaced champion is kept next to the model as
`.prev.json` (`OSR_SLM_KEEP_PREVIOUS`), so `osr slm rollback slm.json` - or `SelfImprover.rollback()`
- undoes a promotion that looked good on the holdout and bad in traffic; the files swap, so a second call
redoes it. Every cycle is appended to a JSONL history with the champion and challenger reports, the
discordant counts and the p-value, and the model file carries a `version` that counts the training runs
behind it.
```bash
osr catalogue --cache data/catalogue.json --hf "llama instruct" --leaderboard --markdown # live cost / quality
osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small
osr -t targets.yaml slm train --out slm.json --cache-dir data --report # every *-tiers cache, holdout metrics
osr -t targets.yaml slm train --out slm.json --from rows.jsonl --encoder attention --warm bundled # fine-tune the packaged model
osr slm predict slm.json "write a python function to parse csv" --cost-weight 1.0 --explain
osr -t targets.yaml improve --slm slm.json --cache-dir data --catalogue data/catalogue.json \
--tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small \
--search "qwen2.5 instruct" --feed https://openai.com/news/rss.xml \
--feed https://huggingface.co/blog/feed.xml --interval 3600 # daemon; --offline for air-gapped
osr slm rollback slm.json # undo the last promotion
```
`--explain` (and `RouterSLM.explain`) prints the hashed features that decided the pick - each word, bigram
or character gram contributes `v_j (W_j . e_t) / tau` to a target's logit, so the list is the exact
decomposition of the logit margin between the pick and its runner-up, plus the learned per-target bias.
The hosted SLM service returns the same block from `POST /predict` with `"explain": true`.
Serve the result: `osr -t targets.yaml --slm slm.json serve`, or in the container `OSR_SLM=/config/slm.json`
(Helm `config.slm`). With `OSR_LLM_BASE_URL` (and `OSR_LLM_API_KEY` / `OSR_LLM_MODEL`) pointing at any
OpenAI-compatible endpoint - OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter - every LLM target gets a
chat handler from `metadata.model`, so `/v1/chat/completions` with `model: "auto"` routes *and* answers.
#### Speed, and what the numbers mean
Training is plain SGD over hashed features; with numpy installed (extra `fast`, included in the serve
image) the same steps run on dense arrays - 18 000 rows train in under a minute, ten times the pure-Python
speed - and prediction projects through a dense mirror of the weights. `OSR_SLM_BACKEND=auto|numpy|python`
chooses; both paths seed identically and write the same model file, so a zero-dependency deployment reads a
numpy-trained model unchanged. The model carries a learned per-target bias (the base rate a unit-norm dot
product cannot express) and trains with inverse-time learning-rate decay (`OSR_SLM_LR_DECAY`) and `l2` 1e-3.
#### The model file: quantised weights
The file is JSON, but its weights are not text: every row of the projection `W`, of the target embeddings `E`
and of the attention encoder's token table is stored as **symmetric per-row integers** - `q = round(x / s)` with
one scale `s = max|x| / 127` per row - packed into a base64 block (`opensmartroute.learning.quant`). At the
default `OSR_SLM_QUANTIZE_BITS=8` that is a fifth of the bytes of the four-decimal floats it replaces with a
relative RMS weight error near 0.5 %; `4` packs two codes per byte (a tenth of the bytes, ~5 % error - measure
before shipping); `0` writes floats. Inference dequantises on load and stays in floating point, so a quantised
model ranks like the float one: the packaged model went from 2.15 MB to 0.58 MB with identical top-1 picks on the
example dataset. Files written by earlier versions (format 1, float rows) load unchanged; a file written at 8
bits carries format 2 and is refused with a clear error by a package that predates it.
```bash
osr slm quantize slm.json --bits 8 --check rows.jsonl # rewrite in place; size, error and top-1 agreement
osr slm quantize slm.json --bits 4 --out slm-4bit.json # smaller still, for edge deployments
osr slm quantize slm-4bit.json --bits 0 --out slm-float.json
osr -t targets.yaml slm train --out slm.json --quantize 0 # train and keep floats in the file
```
`osr slm info` reports `quantization.loaded_bits` (what the file carried) and `quantization.bits` (what the next
save writes).
How long to train is decided by the data, not a constant: from `OSR_SLM_VALIDATION_MIN_ROWS` rows up, `fit`
splits a deterministic `OSR_SLM_HOLDOUT` share off as validation, scores it after every epoch, stops once the
validation loss has not improved for `OSR_SLM_PATIENCE` epochs (at most `OSR_SLM_EPOCHS`) and keeps the best
epoch's weights; the calibrator and the report are fitted on those validation rows, so `--report` is honest
even without a separate holdout. Every row also trains through `OSR_SLM_AUGMENT_VIEWS` (2) *word-dropout
views* - copies with each word dropped at `OSR_SLM_AUGMENT_DROPOUT` (0.15), drawn once per fit so both
arithmetic backends see the same corpus - which teaches the hashed features that a prompt survives losing a
word, exactly how the validation prompts differ from the training ones. `OSR_SLM_LABEL_SMOOTHING` (0, off)
spreads that share of every label over the row's candidates; it changed nothing measurable here and is left
for corpora with noisier labels. On the example catalogue (24 targets, 391 synthetic training rows, 97
holdout, three seeds) the old fixed four epochs reached 0.66-0.70 holdout
accuracy at loss 2.22-2.30; early stopping alone 0.68-0.72 at loss 2.17-2.24; early stopping with the two
views **0.80-0.87 at loss 1.81-1.88** (ECE 0.05-0.11), training in 1.5 s (numpy) or 11 s (pure Python) and
predicting in 0.3 ms. Views are not rows: `trained_rows`, the report and the confidence ramp count the
originals, and rare-feature pruning counts a feature's rows without them. Every history entry records
`epochs`, `best_epoch` and `validation`. The file also records the feature-space signature it was trained
under (`hashed/v1:dim=...`); a model built on a different tokenisation or hash width is refused at load
instead of routing on scrambled weights.
The encoder reads at most `OSR_SLM_MAX_TOKENS` (512) whitespace tokens of a prompt, for scoring, familiarity and
`explain` alike (0 = every token); prompts below the cap are untouched, and the cap is a runtime setting, so an
older model file honours it too. A longer prompt is read through a **reading window** (`learning.window`,
`ReadingWindow`) in two views:
- the *window*: `OSR_SLM_WINDOW_HEAD` (0.4) and `OSR_SLM_WINDOW_TAIL` (0.4) of the budget keep the opening and
the closing tokens - where instructions usually live - and the rest goes to the **middle**: it is cut into
sentences, each is read where it holds the most *instruction cues* (the words a request opens with, taken
from the first word of every task template of the ontology - *write, translate, summarise, extract, fix,
what, why ...* - plus a question mark) and scored for salience (cues per square root of the length), and the
most salient sentences are kept and placed *first*, before the head, because the encoders read an opening
more closely than anything after it. Budget left over goes to `OSR_SLM_WINDOW_RUN`-token (8) runs spaced
evenly through the middle, so bigrams and character grams stay meaningful and the body's own vocabulary
(legal, medical, code) still reaches the model;
- the *ask*: the sentences that ask for something - among the opening and closing spans and the salient
middle - read on their own, at most `OSR_SLM_WINDOW_ASK_TOKENS` tokens (0 = two sentence spans), and added
to the query as a unit vector weighted `OSR_SLM_WINDOW_ASK` (2.0) against the window's: twelve words of
request among five hundred of document are two per cent of a bag of features, and the body decided; now
the request is read the way it would be on its own (0 = window only). A fit and an online step read a long
row through the same two views, on both arithmetic backends, so training and serving agree.
`OSR_SLM_WINDOW_NOVELTY` (1.0, in
0..1) is the share of a sentence's salience that depends on its words being rare in the rest of the prompt,
so a question the document itself repeats on every page (an FAQ, a form) is never mistaken for the request.
Measured with the bundled model on the 70 labelled rows, each buried at the head, the middle or the tail of a
thousand-word document (a finance report without punctuation, legal clauses, FAQ questions, a code dump):
head-and-tail reading routed 7-44 of 70 acceptably, the window alone 29-44 (9 on code, 7 on the FAQ), window
and ask view **63-69 of 70** on every body; the same rows on their own route 70/70 before and after. `explain`
reports the `text` the window view read and the `ask` next to it. The attention block has its own, tighter
window (below) and cuts through the same policy.
A served encoder also remembers the features and query vector of the last `OSR_SLM_ENCODE_CACHE` (512)
prompts it scored (0 = none), dropped on every weight change (an online step, a fit, a reload): the
familiarity gate reads the features the score just computed instead of hashing the prompt a second time,
and a repeated prompt - a retry, an evaluation set, the labelled floor check - is encoded once. Measured on the
bundled model with numpy: `SLMStrategy.score` 735 µs -> 600 µs on a prompt never seen, 130 µs on a repeat;
`predict_proba` 400 µs -> 35 µs on a repeat.
Read `--report` against the constant baseline. Human preference between tiers is noisy: on the ten-source
corpus "always the mid tier" is acceptable for 64 % of holdout prompts and the SLM reaches 64-66 % with a
holdout cross-entropy of 0.98-1.00 (the constant predictor: 1.03), well calibrated (ECE 0.05-0.08). It is much
stronger where the label really depends on the prompt - 94 % on `routellm-gpt4` (baseline 86 %), +12 points
on `arena-100k` - and no better than the prior on sources where the prompt carries no signal. That is what a
learned routing prior is; the ensemble adds rules, capability, similarity and the bandit on top. The
corollary: an SLM trained only on synthetic seed rows is a compressed capability scorer, and in a rule-driven
ensemble it can cost accuracy on labelled traffic (on the 33 example rows: 0.88 without it, 0.82 with it at
weight 1.0). Every report therefore also states the `baseline` - the accuracy of always predicting the most
common acceptable target of those rows - and the `skill` = (accuracy - baseline) / (1 - baseline), the share
of the constant predictor's errors the model actually fixes; `SLMStrategy` speaks as loudly as that skill
(`OSR_SLM_TRUST_BY_SKILL`: confidence x (0.5 + 0.5 x skill) of the last validated fit, so a model no better
than the constant predictor is heard at half volume, one that fixes every error at full). Warm-start with
synthetic rows, but let `osr eval` on your own labelled rows set `OSR_WEIGHTS_SLM` until real outcomes have
trained it. `OSR_SLM_ABSTAIN_MARGIN` makes `SLMStrategy` return
no scores at all when its top two probabilities are closer than the margin (top-1 accuracy on such near-ties
was 22 % against 72 % elsewhere); it is off by default because a near-tie still carries the model's opinion
about every *other* target, and the ensemble breaks the tie anyway - measure before turning it on. The
familiarity gate is on: the share of a prompt's features the model has a trained row for scales its confidence
from nothing at `OSR_SLM_FAMILIARITY_NONE` (0.7) to full at `OSR_SLM_FAMILIARITY_FULL` (0.95; 0 = no gate) -
a model is calibrated on the prompts it saw, and a prompt made of features it never saw is where it is most
confidently wrong. Measured with the production model trained on synthetic rows against the 33 labelled
example rows: 0.879 for the ensemble alone, 0.758 with the SLM ungated, 0.879 gated; on prompts from its own
training distribution nine in ten score above 0.94 familiarity and pass at full volume. The trace shows the
gate (`familiarity 0.51 -> x0.00`), and the same number feeds the input-drift monitor below.
Online learning is bounded so one prompt or one noisy stretch cannot drag the served model: a request's
outcomes apply at most `OSR_SLM_ONLINE_MAX_PER_REQUEST` (3) SGD steps, and a served model takes at most
`OSR_SLM_ONLINE_BUDGET` (5000) steps in total before updates pause until the next fit or promotion re-anchors
it on the full corpus (`SLMStrategy.online_steps` / `online_skipped`). `fit` also drops the projection rows
of features seen in fewer than `OSR_SLM_MIN_FEATURE_COUNT` (2) training rows once the corpus has 50 rows or
more - a feature seen once carries its random seed row and nothing else, so an unseen feature contributes
nothing instead of noise: measured -18 % file size at identical accuracy.
One `SLMStrategy` can serve one model per tenant: `SLMStrategy(slm, resolver=lambda req: models.get(req.context.get("account_id")))`
scores a request with the tenant's own `RouterSLM` when the resolver returns one (the shared model
otherwise), trains that same model from the request's outcomes and reports confidence from that model's
skill. The hosted platform uses it for per-workspace models ([PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md), "Learning").
A corpus of model battles only knows the models it compared. Scored rows therefore train a softmax over the
*scored population* (every target any row scores, plus `OSR_SLM_NULL_TARGETS` learned "none of these"
logits that keep it from saturating), so a tool, skill or agent the data never mentions keeps its catalogue
embedding instead of being pushed to zero, and `SLMStrategy` abstains on such targets - its confidence is
scaled by the share of the request's candidates it has evidence about. Rows that name only an `expected`
target (your own feedback, `examples/eval_dataset.jsonl`) contrast against the whole catalogue, which is how
the model learns the non-LLM targets once outcomes arrive.
#### Where a transformer helps
The hashed encoder is a bag of words, bigrams and character grams: it cannot tell "explain this SQL"
from "SQL that explains itself". Two opt-in encoders add that context, both persisted in the same JSON
file and restored by `RouterSLM.from_file`:
- `OSR_SLM_ENCODER=attention` adds `learning.AttentionEncoder`, a transformer block
(hashed token embeddings, sinusoidal positions, `OSR_SLM_ATTENTION_HEADS` heads of
`OSR_SLM_ATTENTION_HEAD_DIM`, residual, attention pooling) to the query encoder:
$q = W\,\phi(x) + \operatorname{Attn}(x)$. No dependencies, deterministic, hand-derived gradients.
It reads a prompt up to `OSR_SLM_ATTENTION_MAX_TOKENS` (128; word and punctuation tokens) through the same
reading window as the hashed features - head, the salient middle first, tail - and is quadratic in that
window - at 128 tokens prediction on a 150-word prompt still
takes about a millisecond. With numpy (extra `fast`, in every image) the block
runs as dense matrix products over a mirror of its weights: a fit that took four and a half minutes in
pure Python takes 12-30 s, prediction 0.7 ms instead of 6 ms, and both paths draw the same random numbers
and end within float rounding of each other (the model file is identical). The block learns slower than
the linear projection, so it trains on its own budget: `OSR_SLM_ATTENTION_EPOCHS` (40),
`OSR_SLM_ATTENTION_LR` (0.2) and `OSR_SLM_ATTENTION_PATIENCE` (4) replace `epochs`, `lr` and
`patience` when the encoder is on (0 = the shared value). With four heads of sixteen dimensions and that
budget it reaches **0.95-0.97** holdout accuracy on the example catalogue's synthetic rows where the
hashed encoder with views stops at 0.80-0.87 (the views add nothing
measurable to the attention encoder, which sees word order already, and double its fit time); with the
old budget (two heads of eight, twelve epochs at 0.05) it *lost* three points, and on the public arena
corpora it lost five at full size - measure on your holdout (`osr slm train --report`) before switching.
The bundled model and the hosted platform's SLM service use this encoder (`OSR_SLM_ENCODER` on the
trainer). The default is `auto`: a loaded model keeps the encoder it was trained
with - the improver clones a bundled or promoted attention champion and fine-tunes it on the attention
budget - and a fresh model is hashed. Naming an encoder explicitly after a champion was trained does not
clone it - `SelfImprover` builds a fresh model with the configured architecture (`ImprovementReport.encoder`
vs `champion_encoder`) and the promotion gate decides, so the switch takes effect only where it wins.
- `OSR_SLM_EMBEDDER=openai:[@]` appends a *frozen* pretrained embedding as dense features
(`learning.EmbeddingFeaturizer`) from any OpenAI-compatible `/embeddings` endpoint over the stdlib client -
no extra, no download: the embedding model a deployment already serves
(`openai:qwen3-embedding:4b@http://osr-ollama:11434/v1`), Azure OpenAI
(`openai:text-embedding-3-small@https://.openai.azure.com/openai/v1`) or OpenAI itself
(`openai:text-embedding-3-small`). The key comes from the variable `OSR_SLM_EMBEDDER_API_KEY_ENV` names
(`OSR_LLM_API_KEY`; an in-cluster model server needs none), each call waits `OSR_SLM_EMBEDDER_TIMEOUT_S` (20)
and texts are cut at their first 6,000 characters. Every vector is cut to its first `OSR_SLM_EMBEDDER_DIMS` (256)
components and renormalised - what the providers' `dimensions` parameter does for a Matryoshka-trained model
(text-embedding-3, Qwen3-Embedding, nomic, jina v3 keep almost all of their quality there) and what keeps a
training corpus of tens of thousands of rows in memory; `0` keeps the full width. The dense block enters at
`OSR_SLM_EMBEDDER_SCALE` (4) times the weight of one hashed feature - at 1 the hundreds of sparse cues drowned it.
`W` then learns a linear head on top of a real transformer while the hashed features keep the surface cues (code
fences, length, exact vocabulary); a fit embeds its corpus in batches of 64 and a text is embedded once per
process, so a training run costs one pass over its rows. Measured on the reference catalogue's 117 held-out
phrasings with Azure `text-embedding-3-small`: the model alone routes 0.79 of them acceptably against 0.74 for the
hashed encoder, and the ensemble with it 0.83. The file records the embedder's name and the cut, so a reload rebuilds
the same client from settings alone; when the endpoint fails at request time `SLMStrategy` sits that request out
(`SLMStrategy.status()["embedder_errors"]`) and the other strategies decide, so an embedding outage never fails a
route. `OSR_SLM_EMBEDDER=sentence-transformers/all-MiniLM-L6-v2` (extra `embeddings`) runs the transformer
in process instead, and `RouterSLM(embedder=callable)` takes any callable (passed again to
`from_file(embedder=...)`). The packaged model ships without an embedder - it must load with zero dependencies
and no endpoint - so a deployment that wants the dense features trains its own: the hosted platform's SLM
service and `osr slm train` read `OSR_SLM_EMBEDDER`, and `SelfImprover` builds a fresh challenger with the
configured embedder when the champion was trained without one, promoted only when it wins the gate.
- `OSR_SLM_EMBEDDER=keras:/path/head` (extra `tensorflow`) names a saved `learning.NeuralRoutingHead` - a Keras
classifier over the same hashed features, trained on the same rows - whose representation layer becomes the
dense embedder, so the SLM's linear head sees a non-linear view of the prompt next to the raw hashed cues.
The head reads conversation context the same way the SLM does (`context=True`: `EvalRow.history` at training
time, `predict_proba(text, history=...)` at prediction time) and carries an experience-replay memory:
`observe(text, target, reward, history=)` keeps every outcome in a bounded `learning.ReplayMemory`,
`replay_fit()` fine-tunes on it (good outcomes as positives, bad ones as reward-weighted mass on the other
targets), and the memory is saved next to the model, so a restart keeps what live traffic taught.
The same extra offers `learning.TokenCountModel`, which learns billed token counts from `(text, tokens)`
pairs and, once installed (`estimate.set_token_counter`), replaces the heuristic behind `estimate_tokens`
([SDK guide](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md)).
#### Live web context
A router picks the model; it cannot make a model know what happened this morning. `signals.freshness_need`
scores how much a request depends on facts that change - news, prices, weather, scores, "latest", "today", an
explicit "look it up", this year's date - into `Signals.freshness_need`; the capability strategy reads it as a
nudge toward targets tagged `web` / `search` / `browsing` (`OSR_CAPABILITY_WEB_BONUS`), and
`enterprise.WebGroundingMiddleware` acts on it: at or above its `threshold` (0.5) it runs the request's first
sentence through `adapters.WebKnowledge` - keyless Wikipedia (`adapters.wikipedia_search`: encyclopaedic, hours to
days behind the news, but real text for "who / what is X"), DuckDuckGo instant answers and Hugging Face by default;
a self-hosted SearXNG instance for the whole live web when `SEARXNG_URL` is set (`adapters.searxng_search`), Brave
with `BRAVE_API_KEY`, or any provider you register - screens every hit with the injection lexicon (`security.injection_risk`; hits at or above
`OSR_SLM_PAGE_RISK_THRESHOLD` are dropped) and attaches the rest to the request - `context["web_context"]` for the
trace and `context["system"]` as a *Live web context* block the OpenAI-compatible chat handler puts in front of
the model, so the answer is grounded in what the web says today and can cite it. `context["web"] = True` forces
it for one request, `False` disables it; identical queries within `cache_ttl_s` come from memory; a failing or
slow provider never blocks routing (`web.grounded` / `web.empty` / `web.failed` events). Turn it on with
`RouterBuilder.with_web_grounding()`, `harden(router, web_grounding=True)` or `osr serve --web-grounding`
(`OSR_SERVER_WEB_GROUNDING=true`).
#### Knowledge graph and context engineering
Outcomes say which target did well on which task; conversations say what was said; the **knowledge graph**
(`opensmartroute.knowledge`) keeps *facts* - typed, sourced, weighted, expiring triples - about the people who
ask, the targets that answer, the topics that come up and the pages that were consulted, and turns them into the
context a model should see for one request:
- `KnowledgeGraph` stores `Triple(subject, predicate, object)` facts with a confidence, an evidence weight (a fact
stated twice grows instead of duplicating), a source (`catalogue`, `conversation:`, `web:`,
`outcome`, `user`), timestamps and an optional expiry; it answers pattern queries (`query`), neighbourhood walks
(`neighbors`), word search (`search`), forgets on request (`forget`), decays what nobody confirmed (`decay`,
`OSR_KNOWLEDGE_DECAY_PER_DAY`), bounds itself (`OSR_KNOWLEDGE_MAX_TRIPLES`) and persists to one JSONL file.
Interoperability with the open-source graph ecosystem is by *format*, not by dependency: `to_jsonld` (JSON-LD
1.1), `to_ntriples` (RDF for rdflib, Jena, GraphDB, Blazegraph), `to_graphml` (Gephi, yEd, igraph), `to_cypher`
(Neo4j / Memgraph `MERGE` statements) and `to_networkx` (when networkx is installed); `from_jsonld` reads back.
The vocabulary is `knowledge.PREDICATES`; the person-subject ones (`PERSONAL_PREDICATES`) are never exported
to a shared graph unless asked.
- Ingestion is rules, not a model: `extract_facts` reads what a person states about themselves (*my name is*,
*I work at*, *I'm a data engineer based in Toronto*, *we use Python and dbt*, *I prefer*, *answer in French*,
*I'm working on*, *remember that*), `mentions` the entities a message names; `ingest_turn`, `ingest_targets`
(the catalogue as facts), `ingest_decision` (task type and domains `routed_to` the chosen target, the person
`asked_about` them), `ingest_outcome` (`target performed_on task` with the running mean quality and the count)
and `ingest_web_context` (a live search's hits as expiring `cites` facts) write into the graph and return what
they added.
- `ContextEngineer` builds the block: *About the user* (the subject's facts, identity first, then relevance),
*Relevant knowledge* (shared facts whose words overlap the request) and *Live web context* (the hits a
`WebGroundingMiddleware` attached, numbered so the model can cite them); every candidate line has a score and
lines are admitted in score order until `OSR_KNOWLEDGE_BUDGET_TOKENS` (600) is spent, at most
`OSR_KNOWLEDGE_MAX_FACTS` per section and `OSR_KNOWLEDGE_MAX_WEB_HITS` hits. `EngineeredContext` records what
was placed and what was dropped and why (budget, limit, duplicate); `apply` puts the block in front of
`context["system"]` and the accounting under `context["knowledge"]`, so a trace shows the model's context the
way it shows the routing decision.
- `ContextMiddleware` runs the engineer in front of every routed request (`context["user_id"]` /
`["account_id"]` / `["session_id"]` or `profile["id"]` names the person) and teaches the graph from the traffic:
the message's facts before routing, the decision and any web hits after; it emits `context.engineered`.
`KnowledgeStrategy` (`knowledge` in the ensemble, `OSR_WEIGHTS_KNOWLEDGE`) scores candidates by what the graph
knows worked on the request's task type and domains - it abstains for a target with fewer than
`OSR_KNOWLEDGE_STRATEGY_MIN_EVIDENCE` outcomes on the topic and trusts itself fully at
`OSR_KNOWLEDGE_STRATEGY_FULL_EVIDENCE`; outcomes reaching it become new `performed_on` evidence.
- A learned link predictor closes the loop (optional extra `tensorflow`): `learning.KnowledgeEmbedding` trains a
TransE embedding of every entity and predicate in Keras (`fit(graph)`; personal facts left out) and then answers
with numpy alone - `predict_tail("target:llm-small", "has_domain")` (which domain is it probably good at that
nobody declared), `predict_head("performed_on", "task:coding")`, `similar`, `score`; saved as `.npz` + `.json`
or `to_dict()` for a database row. Attached as `graph.embedding` it backs `KnowledgeGraph.suggest`, which
otherwise counts what peers of the same type carry.
```python
from opensmartroute.enterprise import RouterBuilder
router = (
RouterBuilder(registry)
.with_defaults()
.with_web_grounding()
.with_knowledge("state/knowledge.jsonl", budget_tokens=600) # ingests the catalogue; learns from traffic
.build()
)
d = router.route(RouteRequest(text="I work at Acme; how do I test a dbt model?", context={"user_id": "u1"}))
print(d.trace.signals.task_type, router.knowledge.query(subject="person:u1"))
print(router.knowledge.to_cypher())
```
`harden(router, knowledge=True)` (or a `KnowledgeGraph` / a path) does the same for a router that already
exists. The hosted platform runs one graph per workspace as its `knowledge` service
([PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md)).
#### Self-operation
`osr serve --autopilot` runs the `SelfImprover` inside the server (`learning.Autopilot`): a cycle every
`OSR_SLM_AUTOPILOT_INTERVAL_S` seconds, an early one when `learning.DriftMonitor` (Page-Hinkley over
the success and quality of every outcome the router learns from, via `Router.observers`) sees the served
quality drop or when `learning.InputDriftMonitor` (Page-Hinkley over the SLM's top probability and over the
share of each prompt's features it has a trained row for - both fall when traffic moves to prompts it was not
trained on, long before outcomes show it; `OSR_SLM_INPUT_DRIFT_THRESHOLD`, `OSR_SLM_INPUT_DRIFT_MIN_N`) sees
unfamiliar traffic, never closer than `OSR_SLM_AUTOPILOT_MIN_GAP_S`. Feedback is joined back to prompts through
the router's request memory (`OSR_SLM_AUTOPILOT_REMEMBER` texts; `RequestMemory(path=...)` keeps it in a JSONL
file across restarts), which also records the decision the router took on every prompt: prompts without an
outcome become distillation rows (the ensemble's propensities as soft labels, newest `OSR_SLM_DECISION_ROWS_MAX`
per cycle), so the SLM learns the router's behaviour on real traffic first and the outcomes correct it. While
real rows are scarce (fewer than the cycle's `synthetic_below`, 200) synthetic rows over the catalogue fill in,
`OSR_SLM_SYNTHETIC_PER_TEMPLATE` per prompt template (2; the hosted stacks use 8). An
accepted challenger is hot-swapped
into the `SLMStrategy` and written back to disk, a failing cycle is logged and the server keeps serving.
`GET /stats` reports `slm` whenever the ensemble serves one (`SLMStrategy.status()`: rows, version, encoder,
origin, calibration, trust, online steps, abstentions, input drift) and `autopilot` (cycles, promotions, drift, input drift, last report, next run) and
`POST /autopilot/cycle` schedules one now. In the container: `OSR_AUTOPILOT=1` with `OSR_STATE`
mounted (Helm `autopilot.enabled`); `--offline` keeps it off the network. `--slm` is a warm start, not a
prerequisite: without it the loop begins with the bundled model over the catalogue (or the one a previous run
promoted into `/slm.json`) and trains a challenger on the first cycle. Each cycle is traced as an
`autopilot.cycle` span with a `learn.improve` event (and `learn.promote` when the challenger wins), so
`GET /events?name=learn.*` shows the loop at work. The hosted platform runs the same loop from
`OSR_PLATFORM_SLM` / `OSR_PLATFORM_AUTOPILOT` and shows it at `/platform/dashboard/learning`
([PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md), section 10).
```bash
osr -t targets.yaml --slm slm.json --state state serve --autopilot --autopilot-interval 1800 \
--catalogue data/catalogue.json --source arena-55k --tier frontier=llm-frontier --tier small=llm-small \
--feed https://openai.com/news/rss.xml --marketplace-refresh 86400
curl -s localhost:8000/stats | jq '.autopilot, .marketplace'
```
The same server follows the marketplace (`--marketplace-refresh` / `OSR_MARKETPLACE_REFRESH_S`,
[MARKETPLACE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MARKETPLACE.md)): new and re-versioned skills and tools land in the served catalogue on the
next tick, and the autopilot's next cycle trains with them as candidates.
## 6. Declaring components with decorators
Targets, strategies, signal extractors, policy rules, middleware and telemetry can be declared where
they live and assembled by a `ComponentRegistry` (details in [SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md)):
```python
import opensmartroute as osr
@osr.tool("weather", domains=["travel"], actions=["lookup"], cost=0.0, examples=["weather in paris"])
def weather(request, **kw):
"""Current weather for a city.""" # docstring -> target.description
return lookup(request.text)
@osr.strategy(weight=0.8) # fn(request, signals, candidates) -> {id: score}
def geo_affinity(request, signals, candidates):
region = request.constraints.region
return {t.id: 1.0 if region in t.constraints.regions else 0.4 for t in candidates}
@osr.policy_rule # fn(target, request, signals) -> reason | None
def business_hours_only(target, request, signals):
if target.kind is osr.TargetKind.HUMAN and request.context.get("after_hours"):
return "human queues are closed"
router = osr.components.router(registry)
app = osr.components.builder(registry).with_auto_learning().build()
```
The short forms take the rest of the boilerplate away - a rule without `rules.yaml`, code for a catalogue's
targets, a guard and hooks without subclasses, and routing without building a router:
```python
osr.components.load("targets.yaml", "rules.yaml") # the catalogue and its rules as components
@osr.handler("llm-small", "llm-frontier") # one function behind several catalogue targets
def chat(request, **kw):
return provider.complete(request.text)
osr.rule("pii-stays-home", prefer=["llm-onprem"], pin=True, contains_pii=True) # a typo in a condition raises
@osr.guard # None passes, a str refuses (SecurityError), a RouteRequest rewrites
def no_empty(request):
return "empty request" if not request.text.strip() else None
@osr.on_outcome # also @on_decision(request, decision), @on_error(request, error)
def count(outcome):
metrics.inc(outcome.target_id, outcome.success)
d = osr.route("Refund order #123, the customer is upset") # router built from the registry on first use
osr.feedback(d, success=True, quality=0.9)
print(osr.run("Summarise this ticket: ...").text) # route + execute + learn
```
examples/decorator_sdk.py (`examples/decorator_sdk.py`) runs the whole loop offline.
Every tunable (ensemble weights, thresholds, scales) lives in one typed `Settings` object and can be
overridden per process (`osr.configure(...)`) or through `OSR__` environment variables;
`osr settings` prints the effective values.
## 7. Enterprise builder
```python
from opensmartroute.enterprise import (
RouterBuilder,
CacheMiddleware,
TenantMiddleware,
LoggingTelemetry,
MetricsTelemetry,
FileAuditSink,
)
from opensmartroute.security import GuardMiddleware
metrics = MetricsTelemetry()
app = (
RouterBuilder(registry)
.with_rules(rules) # declarative preferences
.with_defaults() # capability + similarity + Thompson bandit
.with_auto_learning(state_dir=".osr-state") # IRT + Bradley-Terry + LinUCB + Markov lookahead
.with_health(latency_slo_ms=3000) # circuit breakers, rate limits, budgets
.with_middleware(
GuardMiddleware(redact=True), # gadget defence + PII redaction
TenantMiddleware({"acme": {"max_cost_per_1k": 0.005, "data_boundary": "private"}}),
CacheMiddleware(ttl_s=30),
)
.with_telemetry(LoggingTelemetry(), metrics)
.with_audit(FileAuditSink("audit.jsonl"))
.build()
)
d = app.route(RouteRequest("Summarize this contract", constraints=RequestConstraints(tenant="acme")))
app.learn(
Outcome(
d.request_id,
d.target.id,
success=True,
quality=0.9,
cost_usd=0.002,
latency_ms=1200,
domains=d.trace.signals.domains,
complexity=d.trace.signals.complexity,
)
)
print(metrics.snapshot()["route_latency_ms"], app.health_snapshot())
```
`with_retrieval(narrow_above=, narrow_to=)` switches to retrieve-then-rank for large catalogues,
`with_queue_awareness()` replaces static latency with a queue-aware estimate, `with_shadow(candidate,
mode="shadow" | "ab")` compares policies with SPRT, `with_calibration()` adds temperature scaling and
conformal candidate sets, `with_fair_share()` enforces tenant weights and `with_state_store()` accepts
Redis, SQL, encrypted, versioned or batched stores. Ports (state, telemetry, audit) are ABCs; see
[ENTERPRISE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ENTERPRISE.md).
## 8. Research-track modules
Each module below implements one roadmap item with tests; papers are cited in
[RESEARCH.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/RESEARCH.md) and formulas in [MATH.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/MATH.md).
### Signals from a draft or a small model
```python
from opensmartroute.signals import DEFAULT_EXTRACTORS, VerbalisedDifficultySignal, DraftResponseSignal, extract_signals
extractors = [
*DEFAULT_EXTRACTORS,
VerbalisedDifficultySignal(source=lambda req: small_model_difficulty(req.text)),
DraftResponseSignal(drafter=lambda req: cheap_model(req.text)),
]
router = Router(registry, extractors=extractors)
```
`VerbalisedDifficultySignal` accepts numbers, `7/10` or words (`trivial` ... `expert`) from
`context["difficulty"]` or a callable and blends them into `complexity`. `DraftResponseSignal`
measures hedging, self-corrections, query overlap and length ratio of a cheap draft.
### Response-side uncertainty
```python
from opensmartroute.signals import UncertaintyGate, EventTrigger, TriggerRule, response_uncertainty
from opensmartroute.strategies import Cascade
gate = UncertaintyGate(sampler=lambda req, n: [cheap_model(req.text) for _ in range(n)], n_samples=4)
cascade = Cascade(quality_gate=gate, threshold=0.6) # escalate when samples disagree
result = cascade.run(req, router.route(req).alternatives)
feats = response_uncertainty(samples, p_true=judge_p_true)
trigger = EventTrigger(
[TriggerRule("semantic_entropy", 0.5, "reroute"), TriggerRule("agreement", 0.5, "ask_human", below=True)]
)
trigger.check(feats) # ["reroute", ...]
```
### Hidden-state Dirichlet probe
```python
from opensmartroute.strategies import HiddenStateStrategy
probe = HiddenStateStrategy(
state_fn=lambda req: host_model.hidden_state(req.text), targets=["small", "large"], dim=4096
)
probe.fit(logged_states, logged_winners, epochs=20) # warm start
router = Router(registry, strategies=[*default_strategies(), probe])
```
The probe's confidence is `1 - K/S` (epistemic uncertainty), so unfamiliar states defer to the other
strategies. `math.DirichletProbe` is usable on its own with any feature vector.
### Elastic models and token budgets
```python
from opensmartroute.strategies import BudgetVariant, TokenBudgetStrategy, expand_elastic
for sib in expand_elastic(
registry.get("reasoner"), [BudgetVariant(1024, quality=0.7, cost_scale=0.4), BudgetVariant(8192, quality=0.9)]
):
registry.add(sib) # reasoner@1024, reasoner@8192
router = Router(registry, strategies=[*default_strategies(), TokenBudgetStrategy()])
```
### Edge and cloud tiers
Tag targets with `metadata: {tier: edge}` or `{tier: cloud}` and add `EdgeCloudStrategy()`. Edge
success is a Thompson posterior per complexity bucket, penalised when decode time eats into the
latency SLO; the cloud tier is penalised by upload time and excluded when the request's
`data_boundary` is `private` or `on_prem` and the target is `public`.
### Auctions
`AuctionStrategy(bid_fn=None, value=0.01)` lets each target bid a claimed success probability and a
price (`default_bid` derives both from the catalogue). Claims are corrected by each bidder's observed
bias; the highest corrected surplus wins and pays the second price. Supply a `bid_fn` for
multi-vendor catalogues where providers self-report.
### Protocol selection, aggregation, self-escalation, hand-off
```python
from opensmartroute.strategies import ProtocolPolicy, MixtureOfAgents, SelfEscalation, wrap_stream
from opensmartroute.learning import HandoffPolicy, MixtureCureModel
policy = ProtocolPolicy() # risk / budget -> protocol
choice = policy.choose(req, decision, decision.trace.signals, uncertainty=gate(req, draft))
if choice.protocol == "aggregate":
res = MixtureOfAgents(k=3, budget_usd=0.05).run(decision, req) # top-k answers, majority vote
for o in res.outcomes(req):
router.learn(o)
monitor = SelfEscalation.for_target(decision.target) # streaming competence posterior
streamed = wrap_stream(provider.stream(req.text), monitor)
if streamed.escalated:
... # restart on a stronger target
handoff = HandoffPolicy(MixtureCureModel(), fallback_target="senior-agent", pins=task_pins)
if handoff.step(task_id, teacher_risk): # permanent hand-off
target = handoff.target_for(task_id, decision.target.id)
handoff.finish(task_id, failed=False) # censored observation
```
### Personalisation
`learning.UserAdaptiveStrategy` reads `context["user_id"]` (or `profile["user_id"]`) and keeps a
Beta posterior per user and target, shrunk toward similar users and the global posterior;
`observe_user()` warm-starts from logged interactions. `learning.SkillAffinity.relevance(profile,
skills)` feeds `retrieval.select_skill_set(relevance=)` so skill sets depend on who is asking.
### Multi-turn conversations
```python
from opensmartroute.learning import HistoryTargetStrategy
router = Router(registry, strategies=[*default_strategies(), HistoryTargetStrategy(turns=6)])
d = router.route(
RouteRequest(
"can you explain more?", history=chat_history, context={"last_target": "llm-small", "last_failed": True}
)
)
```
The same follow-up message routes differently depending on what the conversation has been about:
the strategy learns a logistic model over the joint embedding of the recent history and each target's
catalogue embedding, so a target that keeps failing in legal threads but succeeds in coding threads
is scored accordingly, and an unseen target is still scored through its description and examples.
`last_target` gives the incumbent a small continuity bonus that turns into a penalty when the previous
turn failed.
### Discovery beyond text similarity
```python
from opensmartroute.discovery import SchemaAwareStrategy, CachePreservingSelector, SkillGraph
router = Router(registry, strategies=[*default_strategies(), SchemaAwareStrategy()])
selector = CachePreservingSelector(prefix_size=8, evict_after=5)
tools = selector.select(session_id, candidates, needed=[d.target.id]) # prefix-stable tool order
graph = SkillGraph(load_skills("skills")) # osr-requires / osr-conflicts
ordered, dropped = graph.compose(["report-builder"])
```
### MCP server recommendation and semantic-router import
```python
from opensmartroute.adapters import recommend_servers, load_semantic_router_config
for rec in recommend_servers(
"open a pull request", server_cards, k=2, constraints=req.constraints, allowed_auth=["none", "token"]
):
print(rec.server.name, rec.score, rec.rationale)
imported = load_semantic_router_config("config.yaml") # vLLM semantic-router
router = Router(imported.registry, strategies=[RulesStrategy(imported.rules), *default_strategies()])
```
### Energy and carbon
`math.EnergyModel` fits Wh per target from measured samples; `math.hardware_profile("h100-sxm")`
gives priors when no meter exists. Both convert to gCO2 with a grid factor and feed
`wh_per_1k_tokens` / `gco2_per_1k_tokens` on the target's `cost`, which `Objective(energy=, carbon=)`
trades against quality.
## 9. Command line
Install `osr` with the platform installer (`curl -LsSf https://opensmartroute.ai/install.sh | sh` on
Linux and macOS, `irm https://opensmartroute.ai/install.ps1 | iex` in PowerShell) or with
`uv tool install 'opensmartroute[yaml,server]'` (add `fast` for numpy-accelerated SLM training); `osr --version`
confirms the install.
`osr` takes the catalogue with `-t targets.yaml`, optional rules with `-r rules.yaml` and an optional
routing SLM with `--slm slm.json`; `--help` on any subcommand lists its options. `python -m opensmartroute ...`
is the same command for environments where the script is not on `PATH` (IDE MCP configurations, containers,
`uv run`). Grouped by task:
```bash
# Sign in and manage access tokens
osr login # browser sign-in to the hosted platform
osr login --url https://osr.example.com --profile work # another deployment, kept as a profile
osr login --url http://router:8000 --token osr_local_... # self-hosted osr serve token
echo "$TOKEN" | osr login --with-token # non-interactive (CI)
osr whoami # workspace, plan, edition, token source
osr trace # span tree of one request (server or platform)
osr events --name guard.* --limit 20 # recent tracer events there
osr token generate # mint an osr_local_ token for osr serve
osr token create --name ci # new platform API key (shown once)
osr token list # keys of the signed-in workspace
osr token revoke
osr logout # forget the profile (--all for every one)
# Route, quote and inspect
osr -t examples/targets.yaml -r examples/rules.yaml route "I want a refund for order #123" --plan
osr -t examples/targets.yaml route "Summarise this HR complaint" \
--constraint data_boundary=on_prem --constraint region=eu # hard constraints; --kinds agent,tool
osr -t examples/targets.yaml -r examples/rules.yaml route "" --event order.shipped --plan # textless event
osr -t examples/targets.yaml estimate "Translate this contract" --monthly 100000 # cost per candidate + projection
osr -t examples/targets.yaml targets # the catalogue as the router sees it
osr -t examples/targets.yaml stats # feedback statistics per target
osr settings --json # effective tunables
# Evaluate and audit
osr -t examples/targets.yaml -r examples/rules.yaml eval examples/eval_dataset.jsonl --frontier --calibration
osr -t examples/targets.yaml audit traffic.jsonl --baseline llm-frontier --monthly 3000000 --markdown
# Routing Audit: savings + violations
osr -t examples/targets.yaml ope decisions.jsonl # off-policy estimate (IPS/SNIPS/DR) of logged decisions
osr -t examples/targets.yaml safety --learned-guard # red-team suite
# Learn
osr train --from rows.jsonl --out models.json # fit signal models
osr catalogue --cache data/catalogue.json --hf "llama instruct" # live model prices + quality
osr collect --cache-dir data --tier frontier=llm-frontier --tier mid=llm-mid --tier small=llm-small # Hub datasets
osr collect --cache-dir data --file routerbench=routerbench.jsonl --preset routerbench # a gated file you hold
osr -t examples/targets.yaml slm train --out slm.json --report # routing SLM (see section 5)
osr slm info slm.json # rows, sources, encoder, calibration, fit history
osr -t examples/targets.yaml --slm slm.json route "parse this csv" # SLM inside the ensemble
osr slm eval slm.json rows.jsonl # SLM accuracy on labelled rows
osr slm predict slm.json "parse this csv" --top 3 # what the SLM alone would rank
osr slm info slm.json # SLM metadata, targets and size
osr -t examples/targets.yaml improve --slm slm.json --cache-dir data # self-improvement cycle
# Serve and integrate
osr -t examples/targets.yaml -r examples/rules.yaml serve # http://127.0.0.1:8000/docs (open, local dev)
osr -t examples/targets.yaml serve --generate-token # prints an osr_local_ token; all routes need it
osr -t examples/targets.yaml serve --require-auth --token "$TOKEN" # production: refuse to start without a token
osr -t examples/targets.yaml mcp # MCP server over stdio for an IDE
python -m opensmartroute -t examples/targets.yaml mcp # the same, when `osr` is not on PATH
osr mcp --remote # stdio bridge to the platform you signed in to
osr mcp --url https:///mcp --api-key osr_... # ...or an explicit endpoint and key
osr mcp-manifest tools.json --key-env OSR_MCP_KEY --out signed.json # sign an MCP tools/list (--verify to check)
OSR_LLM_BASE_URL=mock:// osr -t examples/targets.yaml serve # every LLM answers in-process: demos and CI, no key
# Launch a coding agent with the router as its model endpoint (the saved sign-in becomes its key)
osr run --list # the tools and how each one is wired
osr run claude-code # ANTHROPIC_BASE_URL + key -> the router, model auto
osr run codex --model osr/auto:cheap -- "fix the failing test" # anything after -- goes to the tool untouched
osr run gemini-cli --print-env # the variables, for your own shell
# Import another gateway's configuration as a catalogue
osr migrate litellm config.yaml --out ./router # targets + providers.json + aliases policy, and what is left to do by hand
# Size self-hosted model servers (CPU / RAM)
osr capacity show qwen3.5:4b glm-4.7-flash --memory 28Gi --cores 4 # footprint, fits?, tokens/s, warm vs cold latency
osr capacity plan qwen3.5:4b qwen2.5-coder:7b deepseek-r1:8b --memory 8Gi --hosts 3 # pack onto hosts + OLLAMA_* env
osr capacity plan glm-4.7-flash --memory 14Gi --shrink --strict # coarser KV / shorter context to fit; exit 2 if not
osr capacity show --url http://ollama:11434 --memory 28Gi # exact shapes from the server's /api/show
osr capacity ps --url http://ollama:11434 # what the server holds in memory right now
# Manifests, stacks and skills
osr validate examples/ocm # Open Capability Manifests
osr -t examples/targets.yaml export-ocm --out manifests/ # catalogue -> OCM manifests
osr stack init --out stack.yaml # starter declarative stack
osr stack validate examples/stack.yaml # check (imports resolved)
osr stack plan examples/stack.yaml --against deployed.yaml # diff against what runs today
osr stack apply examples/stack.yaml --route "refund order 42?" # build the router, route once
osr stack apply registry://support-desk@1.0.0 --registry $OSR # marketplace template
osr skills # validate SKILL.md packages
```
`osr login` without `--token` runs the OAuth 2.0 device authorization grant against the hosted platform
(community or enterprise edition alike): the CLI prints a short code, opens `/platform/cli/authorize` in the
browser, and after you approve it there the platform mints a workspace API key (`osr_live_...`) for
that machine. Credentials are stored per profile in `~/.config/opensmartroute/credentials.json`
(`%APPDATA%\opensmartroute` on Windows, `OSR_CONFIG_DIR` overrides); flags, then `OSR_API_URL` /
`OSR_API_KEY`, then the saved profile decide which one a command uses. A self-hosted `osr serve` has
no accounts: `osr token generate` (or `serve --generate-token`) mints an `osr_local_...` token, the
server accepts it as `Authorization: Bearer` or `X-API-Key` on every route except the health,
metrics, `/whoami` and OpenAPI paths, and `OSR_SERVER_AUTH_TOKENS` / `OSR_SERVER_REQUIRE_AUTH`
configure the same thing from the environment (see :class:`ServerSettings` in
[SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md#settings)).
The Routing Audit replays logged traffic (`{"text": ..., "model": ..., "cost_usd": ...}` per line)
through the catalogue and reports savings against a baseline target, policy violations the logged
route would have committed, and the route mix; `--min-savings` makes it a CI gate. In production the
same numbers come from `RouterBuilder.with_savings(baseline="max")` and
`ep.savings.report().to_markdown()`.
`osr run ` is the one-line way to put a coding agent or AI client on the router: Claude Code, Codex, the
Gemini CLI, aider, OpenCode (launched), and Cursor, Continue, Zed, LibreChat, Open WebUI (configured in their own
settings - `--print-env` prints what to paste). The tool's provider variables (`ANTHROPIC_BASE_URL`,
`OPENAI_BASE_URL`, `GOOGLE_GEMINI_BASE_URL` and their keys) are set from the saved sign-in or `--url` / `--token`,
`--model` names what the tool asks for (`auto` by default: the router decides per request), and the process is
replaced by the tool's, so `Ctrl-C` and exit codes are the tool's own. Everything after the agent name - or after
`--` - goes to the tool untouched.
`osr migrate litellm ` reads a LiteLLM proxy configuration - its `model_list` (deployment names,
provider models, API bases, credential variables, prices, context windows, embedding mode, rate limits, budgets)
and the fallback chains - and writes, under `--out`, the catalogue (`targets.yaml`; one target per deployment
name with cost, context window, `rate_limit` / `budget` metadata and the `embeddings` tag), `providers.json`
(`providers` with `base_url` / `api_key_env` per distinct endpoint, `models` binding each target to
`provider/model`; a literal key in the config is never copied, the variable it should come from is named) and,
when a deployment name is not a valid target id, a `policy` file with the `aliases` that keep callers' model
names working on the hosted platform. The report lists what was skipped and why, and what a proxy keeps in its
database rather than its file - virtual keys, teams, users, budgets - as the steps left to do by hand.
## 10. Routing latency
Routing is pure Python on the request path. Measured with the reproducible routing benchmark (300 requests
per configuration, 2026-10-09)
on the 27-target example catalogue with its rules, CPython 3.11, x86-64 Windows laptop:
| Configuration | p50 | p95 | p99 |
|---|---|---|---|
| signals + policy + rules + capability + similarity + bandit | 4.6 ms | 6.6 ms | 8.3 ms |
| + IRT + preference + LinUCB + Markov + health | 7.4 ms | 9.1 ms | 14.1 ms |
| + LLM judge (only when `confidence < escalate_llm_judge_below`) | + one provider round-trip | n/a | n/a |
Catalogue size: 64 targets route in 14.5 ms p50 and 256 in 16.7 ms with
`RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24)` (versus 36 ms and 142 ms scoring every
target); 1024 targets route in 24 ms. Cost is linear in `scored targets x strategies`; signal
extraction is under 1 ms and independent of catalogue size. `RouteDecision.trace.elapsed_ms` records
the routing time per request, `MetricsTelemetry` exposes `route_latency_ms` percentiles and the
OpenTelemetry adapter emits the `osr.route_latency_ms` histogram. Full tables in
[ARCHITECTURE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ARCHITECTURE.md#performance-envelope).
## 11. Package layout
```
opensmartroute/
core/ types (RouteTarget, RouteRequest, RouteDecision, Outcome, ...), registry
signals/ lexicon extractors, task ontology, learned hashed models, verbalised difficulty,
draft-response and response-uncertainty features
policy/ hard-constraint filter
strategies/ rules, capability, similarity, bandit, llm_judge, cascade, task_table, defer, progress,
probe (Dirichlet), elastic, edge, auction, protocol, escalation, aggregate
learning/ IRT, preference, LinUCB, Markov, AutoLearner, cold start, task credit,
personal (user adaptation, skill affinity), handoff (mixture cure),
multiturn (history-target joint embeddings)
math/ bandits, irt, preference, markov, calibration, estimators, decision, dirichlet, energy
realtime/ circuit breaker, rate limit, budget, health
enterprise/ RouterBuilder, EnterpriseRouter, middleware, telemetry, audit, stores, ops
security/ InputGuard, learned gadget detector, injection risk, redaction, limits, provenance
adapters/ OpenAI-compatible client, harnesses, MCP (tools + server recommendation), A2A,
frameworks, personas, SKILL.md, semantic-router import, embeddings, OpenTelemetry
eval/ harness, baselines, robustness, frontiers, off-policy evaluation, dataset presets
discovery.py schema-aware matching, cache-preserving selection, skill graph
retrieval.py BM25 + dense retrieval, reciprocal-rank fusion, skill-set selection, meta-tools
execution.py plan executor
sdk.py decorator SDK and plugin discovery
settings.py typed, environment-overridable tunables
config.py load_targets / load_rules (JSON, YAML)
feedback/ outcome store and preference export
router.py orchestrator and plan builder
aio.py AsyncRouter
server.py FastAPI app and OpenAI-compatible proxy
cli.py osr command
```
---
# SDK Guide
## Stability & versioning
- **SemVer.** Everything exported from `opensmartroute` (top-level) is public API.
`opensmartroute.enterprise`, `.security`, `.learning`, `.realtime`, `.math`, `.eval`, `.aio`,
`.adapters`, `.config` are public too but may add fields between minor versions.
- Modules prefixed `_` and anything not in an `__all__` are private.
- The complete list of public modules and names, with one-line summaries, is generated into
[REFERENCE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/REFERENCE.md) from the source.
- Deprecations are announced one minor version ahead with `DeprecationWarning`.
## Installation matrix
`curl -LsSf https://opensmartroute.ai/install.sh | sh` (Linux, macOS) and
`irm https://opensmartroute.ai/install.ps1 | iex` (Windows) install the `osr` CLI with the `yaml` and
`server` extras in an isolated tool environment (uv, pipx or a private venv). For the library, pick
the extras you need:
| Extra | Adds | Use when |
|---|---|---|
| *(none)* | pure stdlib | embedding in any Python 3.10+ service |
| `yaml` | PyYAML | loading `targets.yaml` / `rules.yaml` |
| `server` | FastAPI, uvicorn, pydantic | running `osr serve` |
| `fast` | numpy | faster SLM training and scoring (pure Python otherwise) |
| `embeddings` | sentence-transformers | semantic `SimilarityStrategy` |
| `tiktoken` | tiktoken | exact OpenAI tokenisation in `estimate` |
| `tensorflow` | TensorFlow / Keras | `learning.neural` token-count and routing-head models (large; not part of `all`) |
| `otel` | opentelemetry-api/sdk | `adapters.OpenTelemetryTelemetry` |
| `crypto` | cryptography | AES-GCM encrypted state stores |
| `dev` | pytest, ruff, mypy, bandit | contributing |
| `all` | every extra above except `tensorflow` | the installer's `OSR_EXTRAS=all` |
`osr doctor` prints which extras are present, the bundled routing model, the config directory and the sign-in,
and with `--targets` / `--rules` whether a catalogue loads; it exits 1 when something required is missing.
## Two ways in: a router in your process, or the platform over HTTP
**In your process.** `Router.starter()` is a complete router with nothing configured - the same three targets
and two rules `osr route` uses without `--targets` (a small fast model, a frontier model, an on-premises one;
personal data stays on-prem, hard requests go to the frontier). Decisions take a fraction of a millisecond and
nothing leaves the machine:
```python
from opensmartroute import Router
router = Router.starter()
d = router.route("Prove that sqrt(2) is irrational, step by step.")
print(d.target.id, round(d.confidence, 2)) # llm-frontier 0.9x
print(d.trace.explain())
```
`Router.from_files("targets.yaml", "rules.yaml")` loads your own catalogue; `osr stack init --out stack.json`
writes the starter one to disk to edit.
**Over HTTP.** `OpenSmartRoute` is the client of the hosted platform and of a self-hosted `osr serve`: the
same surface as the TypeScript SDK, standard library only. With no arguments it reads `OSR_API_KEY` /
`OSR_API_URL`, then the profile `osr login` saved; without any key it can still quote (`estimate()`) and read
the deployment (`info()`).
```python
from opensmartroute import OpenSmartRoute, RateLimitError
osr = OpenSmartRoute() # or OpenSmartRoute(api_key="osr_live_...", base_url="https://osr.example.com")
d = osr.route(
"Refund order #123, the customer is upset",
constraints={"data_boundary": "private"},
objective={"cost": 0.4, "quality": 0.6},
)
print(d.target_id, d.confidence, d.explanation) # d.raw holds everything the server sent (d["ranked"] ...)
answer = osr.route("Summarise this ticket: ...", execute=True).text # run the chosen target too
osr.feedback(d, success=True, quality=0.9) # close the loop: the learners improve on your traffic
print(osr.ask("What is the capital of France?")) # OpenAI-compatible chat with model="auto"
for chunk in osr.chat("Write a haiku about routers", stream=True):
print(chunk["choices"][0]["delta"].get("content", ""), end="")
quote = osr.estimate("Translate this contract", monthly_requests=10_000) # no key needed
print(quote["recommended"]["target_id"], quote["monthly"]["savings_usd"])
```
What the client does on its own: a timeout on every call (`OSR_CLIENT_TIMEOUT_S`, 60 s), bounded retries with
backoff after a connection failure, a 408, a 429 or a 502-504 - `Retry-After` is honoured, a 4xx that is final is
never retried (`OSR_CLIENT_MAX_RETRIES`, 2) - and typed errors: `AuthenticationError` (401/403, says how to sign
in), `RateLimitError` (429, with `.retry_after`), `APIError` (`.status`, `.detail`, `.request_id` for a support
ticket). No error message and no `repr()` ever contains the key. An API key is refused over plain `http://` to
anything but a private host (loopback, RFC 1918, a bare service name, `.internal`, `.svc` ...) unless you pass
`allow_insecure_http=True` or set `OSR_CLIENT_ALLOW_INSECURE_HTTP=1`. An `osr_local_...` token puts the REST
routes at the root for `osr serve` (`api_prefix=""` does the same by hand); `tenant=` and `app=` are sent on
every call; `osr.rest`-style access is `osr.get(path)` / `osr.post(path, body)` for anything not wrapped.
Every tunable is a `ClientSettings` field (`OSR_CLIENT_*`).
## Connecting real providers
`adapters.OpenAICompatClient` is a stdlib HTTP client for any OpenAI-compatible API (OpenAI, Azure
OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq, Mistral). It has explicit timeouts, exponential
backoff on 408/429/5xx, and maps failures to SDK errors (`ConfigurationError` for 401/403,
`TargetUnavailableError` for connectivity/5xx, `ExecutionError` for the rest). The API key is read
from `OPENAI_API_KEY` (or `OPENAI_API_KEY_FILE`) and never appears in logs or exceptions.
```python
from opensmartroute.adapters import OpenAICompatClient, chat_handler, embedder, judge_fn
openai = OpenAICompatClient() # https://api.openai.com/v1
local = OpenAICompatClient("http://localhost:11434/v1", api_key="ollama") # Ollama
registry = TargetRegistry(
[
RouteTarget(
"gpt-4o-mini",
TargetKind.LLM,
cost={"usd_per_1k_tokens": 0.00015},
handler=chat_handler(openai, model="gpt-4o-mini"),
),
RouteTarget(
"llama-local",
TargetKind.LLM,
cost={"usd_per_1k_tokens": 0.0},
constraints=TargetConstraints(data_boundary="on_prem"),
handler=chat_handler(local, model="llama3.1"),
),
]
)
router = Router(
registry,
strategies=[CapabilityStrategy(), SimilarityStrategy(embedder=embedder(openai, model="text-embedding-3-small"))],
llm_judge=LLMJudgeStrategy(judge_fn(openai, model="gpt-4o-mini")),
escalate_llm_judge_below=0.5,
)
decision = router.route(request)
result = router.execute(decision, request) # ChatResult(text, tokens, latency_ms); outcome auto-recorded
```
`chat_handler` builds the message list from `request.history` (user, assistant - including `tool_calls` -
and `tool` turns are kept verbatim), and uses `request.context["system"]` as the system prompt — which is
how a **persona** slot in a `RoutePlan` changes model behaviour without a separate deployment. It forwards
`context["openai"]` (`tools`, `tool_choice`, `response_format`, `stop`, `seed`, ...) and `context["on_delta"]`
(a streaming callback) to the upstream, and the `ChatResult` carries `tool_calls`, `finish_reason` and
`refusal` beside the text and token counts. `embed_handler(client, model)` is the counterpart for
embeddings-tagged targets (what answers `/v1/embeddings`), and `attach_chat_handlers(targets, client)` gives
every LLM target the right one from `metadata.model` - the call behind `OSR_LLM_BASE_URL` in `osr serve`
(an `api.anthropic.com` base URL there picks the Anthropic client automatically).
`adapters.AnthropicClient` speaks Anthropic's **Messages API natively** behind the same `chat` /
`stream_chat` surface, so `chat_handler`, `judge_fn` and every other adapter route to Claude without an
OpenAI-compatible gateway in between: chat-shaped messages, OpenAI function specs, `tool_choice`, tool
results, images and streaming are translated to `POST /v1/messages` (`x-api-key` + `anthropic-version`
headers) and back; `reasoning_effort` (or an explicit `thinking` dict) turns on extended thinking and the
thinking text lands on `ChatResult.reasoning`. Fields the Messages API has no equivalent for are dropped
rather than sent to fail, and `embed()` raises a clear error (Anthropic serves no embeddings endpoint).
```python
from opensmartroute.adapters import AnthropicClient, chat_handler
claude = AnthropicClient() # https://api.anthropic.com/v1, key from ANTHROPIC_API_KEY
target = RouteTarget("claude", TargetKind.LLM, handler=chat_handler(claude, model="claude-sonnet-4-5"))
```
Optional adapters (lazy-imported):
```python
from opensmartroute.adapters import sentence_transformers_embedder, OpenTelemetryTelemetry
SimilarityStrategy(embedder=sentence_transformers_embedder("all-MiniLM-L6-v2"))
RouterBuilder(reg).with_telemetry(OpenTelemetryTelemetry())
```
TensorFlow / Keras models (`pip install 'opensmartroute[tensorflow]'`, `opensmartroute.learning.neural`) sharpen
two learnable parts of every request without touching the zero-dependency core:
```python
from opensmartroute.estimate import estimate
from opensmartroute.learning import NeuralRoutingHead, RouterSLM, TokenCountModel
# tokenisation: learn the residual between the heuristic and the provider's billed prompt_tokens
tokens = TokenCountModel(family="o200k")
report = tokens.fit(pairs) # [(text, billed_tokens), ...] from your usage rows
report["mape_heuristic"], report["mape_model"]
tokens.save("models/tokens-o200k") # .keras + .json sidecar
TokenCountModel.load(
"models/tokens-o200k"
).install() # estimate_tokens(), the admission bound and quotes count through it
# the decision: a dense head over the SLM's hashed features, and its representation as the SLM's dense embedder
head = NeuralRoutingHead(hidden=64)
head.fit(rows, targets=router.registry.all()) # the same EvalRows `osr slm train` reads
head.predict_proba("Refactor this function", candidates=["coder", "writer"])
head.predict_proba("why?", history=[{"role": "user", "content": "prove that sqrt 2 is irrational"}]) # context
head.observe("why?", "frontier", reward=1.0, history=...) # experience replay: every outcome is remembered ...
head.replay_fit() # ... and replayed between full fits (saved with the model)
slm = RouterSLM(router.registry.all(), embedder=head.embedder()) # or OSR_SLM_EMBEDDER=keras:models/head
```
The head reads a follow-up with the user turns it continues, exactly as the SLM does (`learning.routing_text`;
`EvalRow.history` on training rows), and its `learning.ReplayMemory` (bounded, oldest dropped first) travels in the
`.json` sidecar, so a restart keeps what live traffic taught.
`estimate.set_token_counter(fn)` takes any `str -> int` counter (a tokenizer library works too); a counter that
raises falls back to the heuristic. `OSR_ESTIMATE_TOKENIZER` names one without code: `tiktoken:o200k_base` (or any
tiktoken encoding; `pip install 'opensmartroute[tiktoken]'`) counts exactly the way OpenAI bills, `keras:` loads a saved
`TokenCountModel`; a counter that cannot be built is logged and the heuristic stays. Feature vectors
(`learning.text_stats`, the hashed featurizer) are pure Python, so a model trained on one machine reads the same
features on the router that loads it.
### Knowledge graph and context engineering
`opensmartroute.knowledge` is the SDK's knowledge management layer: a `KnowledgeGraph` of typed, sourced, weighted,
expiring facts (JSONL on disk; JSON-LD, N-Triples, GraphML, Cypher and NetworkX out, JSON-LD in), rule-based
ingestion of what a person states, of the catalogue, of decisions, outcomes and live web hits, and the
`ContextEngineer` that chooses, ranks and trims what a model should know for one request into a token budget with
full accounting ([GUIDE.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/GUIDE.md#knowledge-graph-and-context-engineering)). `RouterBuilder.with_knowledge()`
wires the `ContextMiddleware` and the `KnowledgeStrategy`; the graph is `router.knowledge`.
```python
from opensmartroute.knowledge import ContextEngineer, KnowledgeGraph, ingest_targets, ingest_turn
from opensmartroute.learning import KnowledgeEmbedding # TensorFlow to train, numpy to use
g = KnowledgeGraph("state/knowledge.jsonl")
ingest_targets(g, registry.all())
ingest_turn(g, "person:u1", "I work at Acme and we use Python and dbt")
ctx = ContextEngineer(g, budget_tokens=400).build(RouteRequest(text="test a dbt model"), subject="person:u1")
ctx.text, ctx.tokens, ctx.sections(), [ln.reason for ln in ctx.dropped]
emb = KnowledgeEmbedding(dim=32)
report = emb.fit(g) # TransE over the shared facts: {mrr, hits_at_10, ...}
g.embedding = emb # KnowledgeGraph.suggest() asks it first
emb.predict_tail("target:llm-small", "has_domain") # [("domain:finance", 0.81), ...]
emb.save("models/knowledge")
KnowledgeEmbedding.load("models/knowledge") # .npz + .json, numpy only
```
## Loading configuration
```python
from opensmartroute.config import load_targets, load_rules
registry = load_targets("targets.yaml") # or .json
rules = load_rules("rules.yaml")
```
Both raise `ConfigurationError` with the file, the offending entry and the reason. Only
`yaml.safe_load` is used.
## Executing end to end
`Router.route()` returns a decision; `Router.run()` (also on `AsyncRouter` and `EnterpriseRouter`)
routes **and** executes the resulting plan, returning an `ExecutionResult`. `execute(decision, request)`
is still available and returns just the primary handler's raw response.
```python
res = router.run("Write SQL for monthly active users by region", task_id="job-17")
res.text # best-effort text of the primary response
res.response # the raw handler response (ChatResult, HarnessResult, dict, str, ...)
res.target_id # primary target
res.steps # [ExecutionStep(role, target_id, latency_ms, ok, note), ...]
res.system_prompt # what was composed into context["system"] (None if nothing was added)
res.effective_request # the RouteRequest the primary handler actually received
res.outcomes # one Outcome per participant, already passed to router.learn()
```
What `run()` does, in order:
1. `route(request, plan=True)` builds the plan: primary + optional `persona` and `skill` slots. A
slot is only filled when its best candidate's quality estimate is at least
`Router(slot_quality_floor=0.5)` and (for `run`) its confidence is at least `min_slot_confidence`.
2. **Skill pre-processing.** If the `skill` slot's target has a `handler`, it is called with the
request and may return
- a `RouteRequest` — replaces the request the primary sees (rewrite / enrich);
- a `str` — attached as `context["skill_output"]` and appended to the system prompt;
- `None` — the skill contributes instructions only.
Exceptions are recorded as a failed step (and a failed `Outcome`) but do not abort the run.
3. **Prompt composition.** `context["system"]` is set to: any existing `system` + `# Persona: `
+ `RouteTarget.instructions` of the persona + `# Skill: ` + skill instructions + the
primary's own `instructions` (for `llm`/`agent` targets) + `# Skill output`. `chat_handler` and the
harness adapters read this key.
4. **Primary call.** `target.handler(request, **kw)`. Missing handler → `ExecutionError`. In the sync
path a coroutine result raises `ExecutionError("use aexecute()")`; `AsyncRouter.run()` awaits both
skill and primary handlers.
5. **Outcomes.** Success is read from `response.success` / `response["success"]` (default `True`);
cost from `response.cost_usd` or `total_tokens × unit_cost`; quality from `response.quality`.
One `Outcome` is created for the primary (`role=None`) and one per slot (`role="persona"|"skill"`),
all carrying `task_id` (argument or `request.context["task_id"]`) so delayed feedback can be
joined later. Each is passed to `learn`; on a handler exception, failure outcomes are recorded
and the exception re-raised.
### Targets carry `instructions`
```yaml
- id: persona-legal-counsel
kind: persona
primary: false
instructions: |
You are cautious in-house legal counsel. Cite the governing clause...
- id: skill-sql
kind: skill
instructions: "Always qualify table names with the schema; never SELECT *."
```
`instructions` is the persona's system prompt, a skill's body (SKILL.md style) or an agent's
standing instructions. It is disclosed to the model only when the target is part of the plan.
### Agent harnesses
An agent harness is any runtime that takes a task and drives its own loop (coding agent, research
agent, OpenHands, SWE-agent, Claude Code, LangGraph graph, ...). Adapters live in
`opensmartroute.adapters` and all produce a `HarnessResult(text, success, steps, prompt_tokens,
completion_tokens, cost_usd, latency_ms, quality, artifacts, raw)`:
```python
from opensmartroute.adapters import CallableHarness, HTTPHarness, SubprocessHarness, harness_handler
# in-process: fn(task, context, history) -> str | dict | HarnessResult
registry.get("research-agent").handler = harness_handler(CallableHarness(my_graph.invoke))
# HTTP: POST {"task", "context", "history"} -> JSON {"text"|"output", "success", "usage", "cost_usd"}
registry.get("support-agent").handler = harness_handler(
HTTPHarness("https://agents.internal/support/run", api_key_env="AGENT_TOKEN", timeout_s=300)
)
# CLI: task on stdin, answer on stdout (or JSON with json_output=True); system prompt via a flag
registry.get("coding-agent").handler = harness_handler(
SubprocessHarness(["my-agent", "--repo", "."], system_flag="--system", timeout_s=900)
)
```
Non-zero exit codes and `"success": false` become failed outcomes; HTTP 408/429/5xx, timeouts and
unreachable hosts raise `TargetUnavailableError` (so `HealthRegistry` breakers trip); other HTTP
errors raise `ExecutionError`; a missing binary or non-http URL raises `ConfigurationError`.
Implement the `AgentHarness` protocol (`run(task, *, context, history) -> HarnessResult`) for anything
else.
### Skills from `SKILL.md`
Folders following the [Agent Skills](https://agentskills.io/specification) layout load directly:
```python
from opensmartroute.adapters import load_skill, load_skills, skill_from_markdown
for skill in load_skills("skills/"): # skills//SKILL.md, name must match the folder
registry.add(skill)
```
Frontmatter `name`/`description` become `id`/`examples`; the Markdown body becomes `instructions`
(progressive disclosure: only the description is used for routing, the body is shown to the model
once selected). Routing metadata can be added under `metadata:` — `osr-domains`, `osr-actions`,
`osr-languages`, `osr-tags`, `osr-quality-prior`, `osr-primary` (`"false"` for slot-only skills),
`osr-latency-ms`. `license`, `compatibility`, `allowed-tools` and other metadata are preserved in
`RouteTarget.metadata`. Give the returned target a `handler` to make it an executable pre-processor
or primary skill.
## Learning from outcomes
Every learner (bandits, IRT, preference, LinUCB, Markov, task table, judge calibration) consumes the
same `Outcome`:
```python
from opensmartroute import Outcome
d = app.route(req)
resp = call_provider(d.target, req)
app.learn(
Outcome(
request_id=req.request_id,
target_id=d.target.id,
success=True,
quality=grade(resp),
cost_usd=resp.cost,
latency_ms=resp.ms,
)
)
```
- **Plans**: `Router.run()` records one `Outcome` per participant with `role="skill" | "persona" |
"primary"`; learners keep a target's record as a skill separate from its record as the primary
answerer (`signals.extra["role"]`).
- **Cascades**: `Cascade.run(request, d.trace.ranked)` returns a `CascadeResult`; feed every step to the
learners with `app.learn_cascade(request, result)` (rejected steps are failures with the gate score as
quality, the accepted step a success). `result.total_cost_usd` is the spend across steps.
- **Delayed / task-level rewards**: set `Outcome.task_id` (and `step`) so a reward observed at the end
of a multi-step task credits every call in the trajectory (`Router(task_credit=...)`).
- **Offline warm start**: `learning.warm_start_from_matrix(rows, strategies, weight=0.5)` replays a
full-information reward matrix `(prompt, {target_id: reward})` from an offline evaluation into every
learner before serving traffic; `learning.warm_start()` seeds a brand-new target from its neighbours.
- **Replicas**: `learning.merge_learners(local, remote)` folds another replica's learners into yours
(sufficient statistics only, no prompts); `AutoLearner.refresh()` re-loads a writer's snapshot from a
shared `StateStore`. Corrupt state is quarantined on load (`AutoLearner.quarantined`) instead of
raising.
- **Objective**: `Objective(quality, cost, latency, quality_floor, energy, carbon)`; `energy` and
`carbon` weigh `RouteTarget.unit_energy` (`cost["wh_per_1k_tokens"]`) and `unit_carbon`
(`cost["gco2_per_1k_tokens"]` or energy x `cost["gco2_per_wh"]`).
- **Cold-start exploration**: a target declared with only `id`, `kind` and `cost` is never ranked
first and so never earns an outcome. `Router(explore_rate=0.1, explore_min_samples=200)` (or
`RoutingSettings.explore_rate` / `explore_min_samples`) serves the least-seen viable candidate with
probability `explore_rate` until it has that many outcomes; `decision.propensities` includes the
exploration mass (so off-policy estimates stay unbiased) and `signals.extra["explored"]` marks the
request. Off by default.
- **Measured exit criteria**: `opensmartroute.eval.criteria` turns each offline-measurable routing
guarantee into a simulation of the real router (`cold_start_ratio()`,
`effort_token_savings()`, `multi_round_vs_best_single()`, `match_at_1_at_scale()`,
`conformal_coverage()`, `knapsack_never_exceeds_cap()`, `ope_within_live_ci()`) and
`opensmartroute.eval.agentic` adds the tau-bench-style one (`AgentTask`, `load_agentic_tasks()`,
`synthetic_agentic_tasks()`, `task_routing_frontier()`: per-step routing through `ProgressRouter`
against every always-use-agent-X policy on the accuracy-latency frontier).
On your own data, `osr eval DATASET
--multi-round` runs the multi-round comparison on scored rows, `osr eval DATASET --effort` the
effort-routing token comparison on rows that carry per-target `scores` and `tokens` (`EvalRow.tokens`),
and `osr eval TASKS.jsonl --agentic` the task-routing frontier on tasks with per-agent per-step
`success` and `latency_ms` tables.
- **Contrastive and policy-gradient learners**: `learning.ContrastiveRouter().fit(rows, targets)`
trains on scored `EvalRow`s against each row's acceptable set (`objective="distilled"` uses
reward-softened labels); wrap it in `learning.ContrastiveStrategy(model)` to keep learning online.
`learning.PolicyGradientStrategy()` learns a routing policy directly from `decision_reward()`
(quality minus weighted cost and latency) with no quality predictor in between;
`learning.decision_regret(rows, targets, choose, predict)` compares decision regret with prediction
error on the same rows.
- **Headroom before you learn**: `eval.headroom.routing_headroom(rows, targets)` tells you how much of
the oracle gap is measurable on your data; `target_diversity`, `min_catalogue`, `scaling_curve` and
`learnability_by_difficulty` answer whether, and where, a learned router can beat the best single
target.
- **Other target kinds**: `strategies.SemanticCache.target()` turns a semantic cache into a
`DESTINATION` target scored by `SemanticCacheStrategy`; `strategies.AnnotatorPool` +
`HumanRoutingStrategy` route among `HUMAN` annotators with Dawid-Skene skill estimates and
`select_quorum()`; `strategies.MemoryRouter` routes *what to remember* across memory tiers under a
token budget; `strategies.ModalityStrategy` / `ModalityEscalation` handle text-first escalation to
multimodal targets; `strategies.SpeculativeCascade` runs draft and strong models in parallel when the
learned acceptance rate makes that cheaper.
## Error handling
All exceptions derive from `OpenSmartRouteError` and carry `.code` and `.details`:
```python
from opensmartroute import OpenSmartRouteError, NoRouteError, SecurityError
try:
d = app.route(req)
except NoRouteError as e:
# e.details["rejections"] -> {target_id: reason}
fallback()
except SecurityError as e:
reject_request(e.to_dict())
except OpenSmartRouteError:
log.exception("routing failed")
```
The SDK never raises bare `Exception`/`RuntimeError`; strategy failures (e.g. a broken LLM judge)
degrade to neutral scores rather than failing the route.
## Threading & async
- `Router` and `EnterpriseRouter` are safe to share across threads. `Router.lock` (a
re-entrant lock) guards strategy scoring, every learner update (`learn`, `learn_correction`,
`credit_task`), task pins, the request memory, retrieval narrowing and the exploration RNG,
so a strategy's `score` never observes a half-applied `update`. The optional LLM judge is
scored *outside* the lock (network I/O) - put it behind `escalate_llm_judge_below` so it
runs rarely, and set `TimeoutMiddleware`.
- `AutoLearner` takes the same lock (`EnterpriseRouter` / `RouterBuilder` pass `router.lock`;
pass it yourself when wiring by hand) so `learn`, `reset_target`, `save`, `load` and
`refresh` serialise against routing. `save()` deep-copies a sequence-numbered snapshot under
the lock and writes it outside; a separate I/O lock orders writers and readers so a slow save
never overwrites a newer snapshot and two saves never race on the state files.
- Per-module state elsewhere has its own lock: `SemanticCache`, `MemoryRouter`,
`AnnotatorPool`, `ModalityEscalation` (posterior) and `SpeculativeCascade` (acceptance
posteriors, history, failure counters). Embedding / hashing runs outside those locks.
- `SpeculativeCascade` runs `concurrency` speculative requests at once (pool of
`2 * concurrency` workers); `draft_timeout_s` abandons a slow draft and serves the strong
answer; a draft that raises escalates instead of failing (`details["draft_error"]`); a strong
future that is no longer needed is cancelled or its late failure swallowed. `stats()`
reports runs, modes, failure counts and per-bucket acceptance.
- `AsyncRouter` wraps any router for `await`-based apps and awaits coroutine handlers;
`route` / `learn` run in the default executor under the same lock.
- Routing has no I/O of its own.
### Multi-round execution
`ProgressRouter.run_step` routes through `router.route` (middleware and telemetry apply) and
executes with `execution.execute`; a handler that raises is recorded as a failed step for its
target *before* the exception propagates, so the next step already denies it after
`failures_before_switch`. Step cost falls back to `target.unit_cost x estimated tokens` when the
handler reports none, so `budget_usd` is tracked even for plain-string handlers. `arun_step` is
the coroutine twin.
`MultiRoundExecutor` (route -> execute -> judge -> refine) learns **once per round** with the
judged quality (no provisional learn + correction double count); a round whose handler raised
becomes a `Round` with `error` set, learns a failure for that target and the loop continues on a
different target. `stop_reason` is one of `accepted`, `budget`, `deadline` (`deadline_ms`) or
`max_rounds`; `return_best=True` (default) returns the highest-quality round when nothing reached
the threshold; `credit_task=True` delivers the final quality as the task-level reward through
`router.credit_task`, which re-credits every step (learners then see one judged update per round
plus one credited update). `arun` awaits coroutine handlers and an `async` judge.
## Type checking
The package ships `py.typed`; `mypy --strict`-friendly signatures on the public surface.
## Configuration
```yaml
# targets.yaml
targets:
- id: llm-frontier
kind: llm
capabilities: { domains: [math, coding], min_complexity: 0.5, supports_tools: true }
constraints: { data_boundary: public, pii_allowed: false, regions: [us, eu] }
cost: { usd_per_1k_tokens: 0.015 }
latency_ms: 2500
quality_prior: 0.93
examples: ["Prove that sqrt(2) is irrational."]
```
```yaml
# rules.yaml (Arch-Router-style preferences)
rules:
- name: escalation-intent
when: { actions: [escalate] }
prefer: [human-escalation]
pin: true # restrict candidates, not just boost
```
Secrets are **never** in YAML. Handlers read them at call time:
```python
from opensmartroute.security import load_secret
api_key = load_secret("OPENAI_API_KEY") # env var or OPENAI_API_KEY_FILE
```
## Extending
### Decorator SDK
Every routing component - targets, strategies, signal extractors, policy rules, middleware and
telemetry sinks - can be declared with a decorator and assembled by a `ComponentRegistry`. The
decorators are transparent (they return the decorated object unchanged) and the registry stores
*blueprints* (factories), so one declaration can back many independent routers.
```python
import opensmartroute as osr
@osr.tool("weather", domains=["travel"], actions=["lookup"], cost=0.0, examples=["weather in paris"])
def weather(request, **kw):
"""Current weather for a city.""" # first paragraph -> target.description
return lookup(request.text)
@osr.agent("planner", domains=["travel"], min_complexity=0.3, quality_prior=0.9)
def planner(request, **kw):
"""Plan a multi-day trip."""
return plan(request.text)
@osr.strategy(weight=0.8) # fn(request, signals, candidates) -> {id: score}
def geo_affinity(request, signals, candidates):
region = request.constraints.region
return {t.id: 1.0 if region in t.constraints.regions else 0.4 for t in candidates}
@osr.strategy # or a Strategy subclass; name inferred -> "recency"
class RecencyStrategy(osr.Strategy):
def score(self, request, signals, candidates): ...
@osr.signal(order=-1) # runs before the built-in extractors
def urgency(request, signals):
return {"urgent": "asap" in request.text.lower()} # unknown keys land in signals.extra
@osr.policy_rule # fn(target, request, signals) -> reason | None
def business_hours_only(target, request, signals):
if target.kind is osr.TargetKind.HUMAN and request.context.get("after_hours"):
return "human queues are closed"
return None
@osr.middleware # fn(request, next_) -> RouteDecision
def tag_tenant(request, next_):
request.context.setdefault("tenant", "public")
return next_(request)
router = osr.components.router() # Router: targets + strategies + signals + rules
app = osr.components.builder().with_auto_learning().build() # EnterpriseRouter variant
```
Most applications need none of the plumbing above. The one-liners below cover the common cases - a rule
without a `rules.yaml`, a hook without a `Telemetry` subclass, a guard without a middleware chain, code behind a
catalogue's targets - and `osr.route()` / `osr.run()` route through the registry without building a router by hand:
```python
import opensmartroute as osr
osr.components.load("targets.yaml", "rules.yaml") # optional: a catalogue and its rules as components
@osr.handler("gpt-4o", "gpt-4o-mini") # code for targets the catalogue declares
def chat(request, **kw):
return provider.complete(request.text, model=kw.get("model"))
@osr.persona("legal-counsel") # first paragraph routes it, the rest is its system prompt
def legal(request, **kw):
"""Contract and compliance questions.
You are cautious in-house legal counsel. Cite the governing clause before answering."""
osr.rule("pii-stays-home", prefer=["onprem"], pin=True, contains_pii=True) # what rules.yaml holds, in code
@osr.rule(prefer=["human-desk"], weight=0.8, domains=["support"]) # a predicate when `when` cannot say it
def angry_customer(request, signals):
return "unacceptable" in request.text.lower()
@osr.guard # fn(request) -> None (pass) | str / False (refuse: SecurityError) | RouteRequest (rewrite)
def no_empty(request):
return "empty request" if not request.text.strip() else None
@osr.on_decision # one hook each; on_outcome(outcome), on_error(request, error) likewise
def log_decision(request, decision):
log.info("%s -> %s (%.2f)", request.request_id, decision.target.id, decision.confidence)
@osr.signal(name="urgent") # a scalar lands under the extractor's name in signals.extra
def urgent(request, signals):
return "asap" in request.text.lower()
d = osr.route("Refund order #123, the customer is upset") # the registry builds and caches its router
print(d.target.id, d.trace.explain())
osr.feedback(d, success=True, quality=0.9) # Outcome built from the decision and learned
res = osr.run("Summarise this ticket: ...") # route + execute the handler + learn
osr.feedback(res, quality=grade(res.text)) # a later judgement of a run is recorded as a correction
d = await osr.aroute("...") # the same loop for async applications; `async def` handlers are awaited
res = await osr.arun("...")
```
`osr.components.live()` is the router behind `route` / `run` / `learn` / `feedback` (`alive()` wraps it in an
`AsyncRouter` for `aroute` / `arun` / `alearn`): a plain `Router`, or an
`EnterpriseRouter` as soon as a guard, middleware or hook is registered so they apply. It is built on first use
and rebuilt when something is registered afterwards (learned state restarts) - declare first, route later.
A rule condition that is not one of `RULE_CONDITIONS` (`domains`, `actions`, `all_actions`, `language`,
`min_complexity`, `max_complexity`, `contains_pii`, `needs_tools`, `event`, `workflow`, `context`) raises
`ConfigurationError` instead of matching silently; a `@handler` for an id no catalogue declares raises when
the targets are materialised, naming the ids it knows.
The same declarations ride along with the CLI and the served router. `osr --components my_components route "..."`
(repeatable; `my_components:setup` also calls `setup(registry)`) imports the module so its decorators register,
then adds what they declared to the router built from `--targets` / `--rules`: targets and `@handler` bindings,
strategies, rules, signals and policy rules - `@guard`, middleware and the hooks make `osr serve` wrap it in an
`EnterpriseRouter`. `OSR_SERVER_COMPONENTS=pkg.module,pkg.other:setup` does the same for the container and the
Helm chart. In code the step is `osr.components.apply(router)` for a `Router` that already exists (one from
`Router.from_files`), plus `enterprise.harden(router, middleware=osr.components.middlewares(),
telemetry=osr.components.telemetry_sinks())` when middleware or hooks are declared.
| Decorator | Accepts | Materialised as |
|---|---|---|
| `@target(id, kind, ...)`, `@tool`, `@skill`, `@agent`, `@llm`, `@persona`, `@human` | handler `fn(request, **kw)`; docstring = description + instructions | `RouteTarget(handler=fn)` (`@persona` is `primary=False`) |
| `@handler(*target_ids)` | handler `fn(request, **kw)` | bound to the catalogue's targets when materialised |
| `rule(name, prefer=, avoid=, pin=, weight=, **when)` / `@rule(...)` | nothing, or a predicate `fn(request, signals) -> bool` | `Rule` / `FunctionRule` in one `rules` strategy in front of the ensemble |
| `@strategy(name=, weight=, order=)` | `Strategy` subclass or scorer function | `Strategy` / `FunctionStrategy` |
| `@signal(name=, order=)` | `SignalExtractor` subclass or `fn(request, signals) -> mapping \| value` | `SignalExtractor` / `FunctionSignal` |
| `@policy_rule(name=, order=)` | `fn(target, request, signals) -> str \| None` or `Policy` subclass | rule appended to `Policy.with_rules` |
| `@guard(name=, order=-100)` | `fn(request) -> None \| str \| False \| RouteRequest` | `FunctionGuard` middleware, first in the chain |
| `@middleware(name=, order=)` | `Middleware` subclass or `fn(request, next_)` | `Middleware` / `FunctionMiddleware` |
| `@on_decision`, `@on_outcome`, `@on_error` | `fn(request, decision)`, `fn(outcome)`, `fn(request, error)` | `FunctionTelemetry` |
| `@telemetry(name=, order=)` | `Telemetry` subclass | `Telemetry` |
- `osr.components` is the process-wide registry the top-level decorators bind to. Create your own
`ComponentRegistry()` for isolation (tests, multi-tenant apps) - its methods are the same
decorators, and `route` / `run` / `learn` / `feedback` / `live` (and their `a`-prefixed async forms) work on
it the same way.
- Duplicate names raise `ConfigurationError` (`register(..., replace=True)` to override);
`unregister`, `clear`, `merge`, `add(*instances)` (targets, strategies, rules - a `RulesStrategy` folds into
the declared rules - extractors, middleware,
telemetry, policy rules) and `load(targets, rules)` manage the catalogue.
- `RouterBuilder.with_components(registry, targets=, strategies=, rules=, signals=, policy=, middleware=,
telemetry=)` wires a registry into an existing builder selectively; declared rules merge into the
builder's `with_rules(...)` strategy when there is one.
- Plugins: `registry.include("pkg.module")` imports a module so its decorators run
(`"pkg.module:setup"` calls `setup(registry)` afterwards); `registry.discover()` loads every
installed distribution advertising an `opensmartroute.plugins` entry point.
### Subclassing
```python
from opensmartroute import Strategy, StrategyScore
class GeoAffinity(Strategy):
name = "geo"
def score(self, request, signals, candidates):
region = request.context.get("region")
return {
t.id: StrategyScore(1.0 if region in t.constraints.regions else 0.5, f"region={region}") for t in candidates
}
app = RouterBuilder(reg).with_defaults().with_strategy(GeoAffinity(), weight=0.8).build()
```
Custom `Policy`, `SignalExtractor`, `Middleware`, `Telemetry`, `StateStore`, `AuditSink` follow
the same pattern — subclass, implement, pass to the builder (or register with a decorator).
## Settings
Every tunable lives in one immutable `Settings` object, grouped by consumer
(`routing`, `policy`, `rules`, `capability`, `bandit`, `weights`, `slm`, `implicit`, `learning`, `server`,
`observability`, `marketplace`, `compression`, `estimate`, `inference`, `knowledge`, `client`). Components resolve the process-wide settings unless
given an explicit instance.
```python
import opensmartroute as osr
from opensmartroute.settings import RoutingSettings, WeightSettings
osr.configure(routing=RoutingSettings(narrow_above=200), weights=WeightSettings(rules=3.0))
router = osr.Router(reg) # uses the configured settings
isolated = osr.Router(reg, settings=osr.Settings()) # library defaults, unaffected
print(osr.get_settings().env_keys()) # every honoured OSR_* variable
```
Environment overlay: `OSR__` (e.g. `OSR_ROUTING_SOFTMAX_TEMPERATURE=0.2`,
`OSR_WEIGHTS_CAPABILITY=1.5`, `OSR_POLICY_DATA_BOUNDARIES=public,internal,confidential`).
Values are parsed against the default's type; a bad value raises `ConfigurationError` with
`details={"env": key}`. `osr.configure()` with no arguments re-reads the environment.
`osr settings [--json]` prints the effective values, their `OSR_*` keys and which ones the
environment overrides.
## Naming conventions
`opensmartroute.branding` is the single source of truth for every brand-derived name:
| Convention | Value | Helper |
|---|---|---|
| Package / import | `opensmartroute` | `branding.PACKAGE` |
| CLI | `osr` | `branding.CLI` |
| Environment variables | `OSR__` | `branding.env_key("routing", "narrow_to")` |
| Error codes | `OSR_` (`OSR_NO_ROUTE`, `OSR_CONFIG`, ...) | `branding.error_code("no_route")` |
| Metadata / frontmatter keys | `osr- 
-brightgreen?logo=pytest&logoColor=white){kind=icon}