# OpenSmartRoute serve and deploy ## HTTP API (`curl -LsSf https://opensmartroute.ai/install.sh | sh`, or `pip install opensmartroute[server,yaml]`) ``` osr -t targets.yaml [-r rules.yaml] [-s /state] [-m models.json] serve --host 0.0.0.0 --port 8000 osr -t targets.yaml serve --generate-token # prints an osr_local_ token once; all routes need it osr -t targets.yaml serve --require-auth --token $T # production: refuse to start without a token osr -t targets.yaml serve --guard --redact-pii --output-guard redact # screen prompts, mask PII, rewrite leaked secrets / PII in answers ``` Guards on the served router: `--guard` (input guard: gadget / steering detection, size limits; `--learned-guard` swaps in the trained detector), `--redact-pii` (implies `--guard`), `--output-guard off|flag|redact|block` (the answer side: leaked secrets, PII, relayed injection; `OSR_SERVER_OUTPUT_GUARD`, default `off` on `osr serve`). Verdicts become `guard.*` / `output_guard.*` trace events and `output_guard` metadata on proxied responses. `--web-grounding` (`OSR_SERVER_WEB_GROUNDING=1`) searches the web for requests about current facts (`signals.freshness_need`) and hands the injection-screened hits to the model as a *Live web context* system block (`web.grounded` / `web.empty` / `web.failed` events). Code next to the catalogue: `osr --components my_components serve` (repeatable; `module:setup` also calls `setup(registry)`) or `OSR_SERVER_COMPONENTS=pkg.module,pkg.other:setup` imports Python modules whose `@tool` / `@handler` / `rule()` / `@guard` / `@on_outcome` declarations (osr-decorator-sdk) join the served router - handlers bound to catalogue targets, rules merged, guards and hooks running through the hardened router. The module must be importable from the server's `PYTHONPATH` (mount it next to `/config` in the container). | Method | Path | Body -> response | |---|---|---| | GET | `/healthz` | `{"status": "ok", targets, uptime_s}` (liveness) | | GET | `/readyz` | readiness; 503 until the catalogue has targets and a probe request routes end to end | | GET | `/metrics` | Prometheus text: `MetricsTelemetry.prometheus()` (`osr_route_decisions_total{target}`, `osr_route_latency_ms_bucket`, `osr_outcome_*`, `osr_decision_cache_*`) + tracer counters + `osr_targets` | | GET | `/whoami` | `{service, edition: "self-hosted", version, auth_required, authenticated, targets}` - what `osr whoami` prints | | GET | `/targets` | catalogue (`registry.to_dicts()`) | | GET | `/stats` | feedback aggregates (+ `health` from an EnterpriseRouter, + `autopilot` status when enabled) | | POST | `/route` | `{text, context, history, objective, constraints, kinds, top_k=3, plan=false}` -> `RouteDecision.to_dict()`; 422 when no route | | POST | `/feedback` | `{request_id, target_id, success, quality, cost_usd, latency_ms, domains, complexity, preferred_over}` -> `{"status":"recorded"}` | | POST | `/autopilot/cycle` | (with `--autopilot`) schedule an improvement cycle now -> `{scheduled, status}` | | GET | `/v1/models` | OpenAI list: `auto` + every primary target id | | POST | `/v1/chat/completions` | OpenAI body; `model: "auto"` or `"opensmartroute"` routes (plan on), a target id pins; `stream: true` SSE; 404 unknown model, 422 no route, 502 execution error | | POST | `/v1/responses`, `/v1/messages` (+ `/v1/messages/count_tokens`) | OpenAI Responses API and Anthropic Messages API translated onto the chat path (`opensmartroute.responses`, `opensmartroute.anthropic`); Anthropic errors wear `{"type": "error", ...}` | | POST | `/v1/embeddings`, `/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1/images/generations` | OpenAI-compatible embeddings and media over tagged targets (`embeddings`, `speech`, `transcription`, `actions: [image]`) | | POST | `/v1/moderations` | OpenAI moderation shape over the router's guards (`security.moderate`): injection, steering, gadget, PII, oversize - no provider call | | POST | `/mcp` | Model Context Protocol (Streamable HTTP), same tools as `osr mcp` | Python: `from opensmartroute.server import create_app; app = create_app(router, proxy_aliases=..., autopilot=..., auth_tokens=[...])` - mount it inside your own FastAPI app or pass an `EnterpriseRouter`-built `router`. ## Access tokens - Open by default (sidecar on a private network). With one or more tokens (`--token`, `--generate-token`, `OSR_SERVER_AUTH_TOKENS=a,b`, `OSR_SERVER_AUTH_TOKENS_FILE=/run/secrets/tokens`, `ServerSettings.auth_tokens`) every path except `/healthz`, `/readyz`, `/metrics`, `/whoami`, `/docs`, `/openapi.json`, `/redoc` needs `Authorization: Bearer ` or `X-API-Key: ` (`server.OPEN_PATHS`, constant-time compare); otherwise 401 + `WWW-Authenticate: Bearer`. `--require-auth` / `OSR_SERVER_REQUIRE_AUTH=1` fails start-up without a token. - Tokens are opaque strings; `osr token generate [--count N]` mints `osr_local_...` ones (`credentials.generate_token`). Sign a CLI in with `osr login --url http://host:8000 --token ` (stored per profile under `~/.config/opensmartroute/credentials.json`, `OSR_CONFIG_DIR` overrides); `osr whoami` calls `/whoami`. There is no account or key registry on a self-hosted server - rotation is "add the new token, restart, remove the old one". - Helm: `auth.tokens: [..]` (chart-managed Secret), `auth.existingSecret` (key `tokens`, comma separated), `auth.required`. Docker: pass the same `OSR_SERVER_*` variables; the entrypoint needs no change. ## Self-operation (`serve --autopilot`) ``` osr -t targets.yaml [--slm slm.json] -s /state serve --autopilot [--autopilot-interval 3600] \ [--cache-dir DIR] [--catalogue cat.json] [--source NAME ...] [--tier TIER=TARGET ...] [--search Q ...] \ [--feed URL ...] [--offline] [--marketplace-refresh 86400 [--marketplace-url URL]] ``` `--slm` is optional: without it the loop starts warm - the model shipped inside the package (`--slm bundled` names it explicitly; `RouterSLM.bundled()`) over the catalogue, or the model a previous run promoted into `/slm.json`, joins the ensemble and the first cycle trains a challenger on your rows. `learning.Autopilot` runs `SelfImprover` cycles in a daemon thread: on the schedule (`OSR_SLM_AUTOPILOT_INTERVAL_S`), early when `DriftMonitor` (Page-Hinkley on outcome success / quality, fed through `Router.observers`) alarms, or on `POST /autopilot/cycle`; never closer than `OSR_SLM_AUTOPILOT_MIN_GAP_S`. Feedback becomes training rows through the router's request memory (`OSR_SLM_AUTOPILOT_REMEMBER`); an accepted challenger is hot-swapped into the served `SLMStrategy` and saved (to the served file when writable, else `/slm.json`). A failing cycle is counted in `/stats` -> `autopilot.errors`, never fatal. Container: `OSR_AUTOPILOT=1` (+ `OSR_CACHE_DIR`, `OSR_CATALOGUE`, `OSR_AUTOPILOT_ARGS="--offline ..."`); the entrypoint prefers a previously promoted `$OSR_STATE/autopilot/slm.json` over `OSR_SLM` on restart, and `OSR_SLM` may stay empty. Helm: `autopilot.enabled`, `autopilot.intervalSeconds`, `autopilot.args` (needs `persistence.enabled`; `config.slm` is an optional warm start). `--feed URL` / `OSR_SLM_FEEDS` add RSS / Atom article feeds (vendor blogs, release notes) the cycle reads: mentioned catalogue models become `trending` candidates, unknown announced names become card lookups. ## Following the marketplace and compacting prompts `serve --marketplace-refresh SECONDS` (or `OSR_MARKETPLACE_REFRESH_S`) starts `adapters.MarketplaceSync`: the followed listings (`OSR_MARKETPLACE_KINDS` default `skill,tool`, `OSR_MARKETPLACE_QUERY`, `OSR_MARKETPLACE_TAG`, `OSR_MARKETPLACE_LIMIT`, `OSR_MARKETPLACE_SLUGS`, `OSR_MARKETPLACE_URL`) are upserted into the served catalogue on that schedule - no restart, no cron. `GET /stats` -> `marketplace`, `POST /marketplace/sync` (write path; read-only tokens get 403) runs one now. Helm: `marketplace.refreshSeconds` (+ `url`, `kinds`, `query`, `tag`, `limit`, `slugs`) and `autopilot.feeds`. Every model call is compacted first (`OSR_COMPRESSION_*`; whitespace, repeated lines, optional history budget; fenced code untouched) and the proxy metadata reports `tokens_saved`. Plans offer tool targets to the model as functions and run its calls (`OSR_ROUTING_TOOL_SLOTS`, default 2; 0 turns it off). ## Container ```dockerfile # deploy/Dockerfile installs .[yaml,server,fast] (numpy: autopilot retrains 10x faster); defaults OSR_TARGETS=/config/targets.yaml OSR_STATE=/state; EXPOSE 8000 docker build -t opensmartroute -f deploy/Dockerfile . docker run -p 8000:8000 -v $PWD/examples:/config:ro -v osr-state:/state \ -e OSR_TARGETS=/config/targets.yaml -e OSR_RULES=/config/rules.yaml opensmartroute ``` `deploy/entrypoint.sh` reads `OSR_TARGETS` (required), `OSR_RULES`, `OSR_STATE`, `OSR_MODELS`, `OSR_SLM` (routing SLM from `osr slm train`, joins the ensemble as the `slm` strategy), `OSR_HOST` (default `0.0.0.0`), `OSR_PORT` (default `8000`) and execs `osr ... serve`. Health check hits `/healthz`. Mount `/state` on a persistent volume or learner state is lost on restart. The `/v1/chat/completions` proxy only *executes* when LLM targets have handlers. Set `OSR_LLM_BASE_URL` to any OpenAI-compatible endpoint (OpenAI, Azure OpenAI, vLLM, Ollama at `http://host.docker.internal:11434/v1`, OpenRouter, LiteLLM) plus `OSR_LLM_API_KEY` (or `OSR_LLM_API_KEY_ENV=`); each LLM target's `metadata.model` is the upstream model name and `OSR_LLM_MODEL` the default for targets without one. Without it the proxy still routes but LLM targets answer 502 "has no handler". Verified end to end: `X-OSR-Target` / `X-OSR-Request-Id` headers, `model: "auto"`, `"auto:cheap"`, a pinned target id, 404 unknown model, 400 stream. ## Sizing a self-hosted model server (CPU / RAM) `opensmartroute.inference` (`osr capacity`) turns "does this model fit and how fast will it answer" into arithmetic before a deploy: weights = parameters x bits per weight (`Q4_K_M` ~4.85, `Q8_0` 8.5, `F16` 16, `MXFP4` 4.25); KV cache = 2 x layers x KV heads x head size x bytes (`f16` 2, `q8_0` ~1) x `OLLAMA_CONTEXT_LENGTH` x `OLLAMA_NUM_PARALLEL`; plus ~10 % runtime overhead, minus 12 % host headroom. Decode is bandwidth bound (tokens/s ~ GB/s / active bytes per token - an MoE reads only its active experts), prefill compute bound (2 FLOPs per active parameter per token), a cold load = weights / store read rate (a network share: 0.05-0.3 GB/s). ```bash osr capacity show qwen3.5:4b glm-4.7-flash --memory 28Gi --cores 4 # footprint, fits?, tok/s, warm vs cold latency osr capacity plan qwen3.5:4b qwen2.5-coder:7b --memory 8Gi --hosts 2 # first-fit-decreasing packing + OLLAMA_* env osr capacity plan glm-4.7-flash --memory 14Gi --shrink --strict # coarser KV / shorter ctx / fewer slots; exit 2 if unplaced osr capacity show --url http://ollama:11434 --memory 28Gi # exact shapes from /api/show and /api/tags osr capacity ps --url http://ollama:11434 # /api/ps: loaded models, CPU or GPU, resident or expiring ``` `ModelSpec.from_tag` reads size / `-a3b` active share / quant from an Ollama tag (`KNOWN_TAGS` covers names without a size: `gpt-oss`, `glm-4.7-flash`, `glm-ocr`); `OllamaClient(url).specs()` gets the exact shape. `plan()` / `recommend()` / `ollama_env()` are the library entry points; `OSR_INFERENCE_*` (context, parallel, kv_type, headroom_ratio, bandwidth_gbps, load_gbps, keep_alive ...) set the defaults. Rules of thumb the planner encodes: one warm model per CPU node beats several evicting each other; `OLLAMA_KEEP_ALIVE=-1` once the set fits; `q8_0` KV halves cache memory with no measurable loss; never let `context x parallel` push a model out of RAM. `tests/test_inference.py::test_curated_azure_set_fits_its_container_apps` guards the Azure sizing. Cold loads are removed twice on the hosted platform: the model servers preload their set at start (`OSR_OLLAMA_PRELOAD=1` -> `ollama run MODEL ""` after the pulls, with `OLLAMA_KEEP_ALIVE=-1`), and the providers owner (`ProviderHealthChecker`, `OSR_PLATFORM_PROVIDERS_KEEP_WARM_MAX`) reads `/api/ps` on every health pass and loads any mapped model that is missing when the provider maps at most that many distinct models (1 per CPU app, the resident count on a GPU). `GET /api/v1/providers` exposes `providers[].loaded` and `models[].warm`; the console provider card shows "in memory: ...". Test pattern: `platform/api/tests/test_model_residency.py` (stub Ollama with `/v1/models`, `/api/ps`, `/api/chat`). ## The model server (`platform/models`, `osr-models`) The platform's own model host (docs/MODELS.md): Ollama `/api/*` + OpenAI `/v1/*` + Anthropic `/v1/messages` over one `llama-server` process per loaded model (CPU or CUDA). `osr_models/store.py` reads and writes Ollama's on-disk layout (`blobs/sha256-*`, `manifests////`), `sources.py` pulls from ollama.com and Hugging Face (`hf.co//:` or `/.gguf`) with resume, digest check and a directory lock shared between replicas, `gguf.py` reads the header for `/api/show` and the fit, `scheduler.py` owns residency (fit against the memory budget, LRU eviction of *idle* instances only, `keep_alive`), slots and the bounded queue (`Saturated` -> `503` + `Retry-After`), `engine.py` builds the `llama-server` command line, `api.py` is the FastAPI surface, `cli.py` the `osr-models` CLI (the image aliases it as `ollama`, so `resources.bicep`'s start-up script runs unchanged). Settings: `OSR_MODELS_*` first, the `OLLAMA_*` names second (`settings.py`). Image: `platform/models/Dockerfile` (CPU; `BASE_IMAGE`/`LLAMA_FLAVOUR` build args for CUDA) pinned to a llama.cpp release (`LLAMA_BUILD`, also in `azure-pipelines/models.yml`). Tests: `platform/models/tests` (engine-backed ones run when `OSR_MODELS_LLAMA_SERVER` and `OSR_MODELS_TEST_GGUF` are set); `platform/models/scripts/probe.py ` is the live end-to-end probe. Deploy: compose `llm` profile builds it (`OSR_MODELS_IMAGE` overrides), Azure through `OSR_OLLAMA_IMAGE` (`ollamaImage`) on the next Infra run after the Models pipeline pushed `/opensmartroute/models-:ci-`. ## Helm ``` helm install osr deploy/helm/opensmartroute -f values.yaml ``` `values.yaml` covers image, replicas, HPA, ingress, service account, a ConfigMap for `targets.yaml`/`rules.yaml`, extra env/`envFrom` for secrets (`OSR_MCP_KEY`, encrypted-state key), a PVC for `OSR_STATE` and `hardening.*` (`guard`, `learnedGuard`, `redactPii`, `outputGuard: off|flag|redact|block`, `health`, `auditPath`, `originPolicy`, `resourceLimits` -> the matching `OSR_SERVER_*` variables, all off by default). With more than one replica use a shared `StateStore` (Redis/SQL) via a small Python entrypoint, or run one writer and read-only replicas. ## Tuning without code Every tunable is an `OSR__` variable - run `osr settings` to list them with current values. Common ones: `OSR_ROUTING_SOFTMAX_TEMPERATURE`, `OSR_ROUTING_NARROW_ABOVE`, `OSR_POLICY_JAILBREAK_THRESHOLD`, `OSR_WEIGHTS_RULES`, `OSR_WEIGHTS_CAPABILITY`, `OSR_BANDIT_CONFIDENCE_WINDOW`. Bad values fail fast at startup with `ConfigurationError`. ## Operating - Route latency is sub-millisecond for local strategies; only an LLM judge adds network time - keep `escalate_llm_judge_below` low and use `TimeoutMiddleware`. - Post `Outcome`s to `/feedback` from the caller that observed the result; `/stats` shows them. - Watch `route_latency_ms.p95`, `policy_rejections_total`, `errors_by_type` from `MetricsTelemetry.snapshot()`; `GET /metrics` renders the same store as Prometheus text (`MetricsTelemetry.prometheus(namespace="osr", extra=...)`). Scrape it with a `ServiceMonitor` or `prometheus.io/scrape` annotations. `RouterBuilder.with_cache(ttl_s, max_size)` adds a decision cache whose hits and misses appear as `osr_decision_cache_*`; `with_audit(sink, outcomes=True)` chains outcomes too. - Tracing: `GET /metrics` also carries the tracer's `osr_spans_total` / `osr_span_duration_ms` per stage, `GET /trace/{request_id}` returns every span and event of one request (`X-OSR-Trace-Id` on each response; send `traceparent` to join your own trace), `GET /events?name=route.*` filters the buffer. From a terminal: `osr trace ` renders that trace as a stage-by-stage timeline (`--json` for the payload) and `osr events --name output_guard.* --level warning --limit 50` lists events, both against the saved `osr login` credentials (`--url` / `--token` / `--profile` override; work against `osr serve` and the platform). `OSR_OBSERVABILITY_LOG_EVENTS=true` for JSON lines, `OSR_OBSERVABILITY_EVENTS_FILE=/state/events.jsonl` for a file, `OSR_OBSERVABILITY_OTEL=true` (+ `otel` extra) for OpenTelemetry, `OSR_OBSERVABILITY_SAMPLE_RATE` under load. See `docs/OBSERVABILITY.md`. The hosted platform image reads the same variables; there the buffer is workspace-scoped (`GET /api/v1/trace/{request_id}` with outcomes and audit records, `GET /api/v1/events`, public `GET /api/v1/status` for status pages) - `osr-platform` skill. - Use `osr -t targets.yaml eval data.jsonl --min-accuracy 0.9` in the image build pipeline so a catalogue change cannot ship a regression. - Secrets: env or `_FILE` mounts via `load_secret`; never bake them into the image or YAML.