<!-- OpenSmartRoute: Deploy with Docker and Helm. https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy -->
<p align="center">
  <a href="https://opensmartroute.ai">
    <picture>
      <source media="(prefers-color-scheme: dark)" srcset="https://opensmartroute.ai/brand/logo-horizontal-reverse-1240.png">
      <img src="https://opensmartroute.ai/brand/logo-horizontal-1240.png" alt="OpenSmartRoute" width="440">
    </picture>
  </a>
</p>

# Reference deployments

OpenSmartRoute runs in three shapes. All three use the same `Router`; only where it lives differs.

| Shape | When | How |
|---|---|---|
| **Library** | one Python service, in-process decisions (~0.3 ms / route) | `pip install opensmartroute` and call `Router.route()`; see [docs/SDK.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/SDK.md) |
| **Sidecar** | polyglot services, per-pod learner state | run the image next to your app: `docker run -p 8000:8000 -v ./examples:/config crosrua2c6onyidguw.azurecr.io/opensmartroute:2.1.0` and call `POST /route` |
| **Control plane** | many callers, shared catalogue, shared learning | the Helm chart in helm/opensmartroute (`deploy/helm/opensmartroute`) behind a `Service`/`Ingress`; state in Redis/SQL via `RouterBuilder.with_state_store` |

## Image

Dockerfile (`deploy/Dockerfile`) builds a non-root, read-only-rootfs image with the `yaml` and `server` extras:

```sh
docker build -t opensmartroute:2.1.0 -f deploy/Dockerfile .
docker run --rm -p 8000:8000 opensmartroute:2.1.0                              # built-in starter catalogue
docker run --rm -p 8000:8000 -v "$PWD/examples:/config:ro" opensmartroute:2.1.0   # your targets.yaml / rules.yaml
curl -s localhost:8000/route -H 'content-type: application/json' -d '{"text":"summarise this contract"}'
```

With nothing mounted at `/config/targets.yaml` the container serves the CLI's built-in starter catalogue
(`llm-small`, `llm-frontier`, `llm-onprem`) and says so in its log; mount a folder with `targets.yaml` (and
`rules.yaml`, set `OSR_RULES=/config/rules.yaml`) to replace it.

Configuration is by environment (`OSR_TARGETS`, `OSR_RULES`, `OSR_MODELS`, `OSR_SLM`, `OSR_STATE`,
`OSR_HOST`, `OSR_PORT`) so the same image serves both sidecar and control-plane roles;
entrypoint.sh (`deploy/entrypoint.sh`) translates them into `osr serve` arguments. `OSR_SLM` points at a
routing SLM file from `osr slm train` (or `bundled` for the model shipped inside the package); it joins the
strategy ensemble and is reported as the `slm` strategy in every decision trace.

The OpenAI-compatible proxy (`/v1/chat/completions`) executes LLM targets when the container knows an
upstream: set `OSR_LLM_BASE_URL` (OpenAI, Azure OpenAI, vLLM, Ollama, OpenRouter, LiteLLM ...) and
`OSR_LLM_API_KEY` (or `OSR_LLM_API_KEY_ENV` naming another variable / `<NAME>_FILE` mount). Each LLM
target's `metadata.model` is the upstream model name; `OSR_LLM_MODEL` is the default for targets
without one. Targets tagged `embeddings` get an embeddings handler instead and answer `/v1/embeddings`; targets
tagged `transcription` / `speech` and image tools (`actions: [image]`) with a `metadata.model` get the media
handlers and answer `/v1/audio/transcriptions`, `/v1/audio/speech` and `/v1/images/generations` on the same
provider; `/v1/responses` (the OpenAI Responses API) and `/v1/messages` (the Anthropic Messages API, for Claude
Code and the Anthropic SDKs) ride on the chat path.
Without a base URL the proxy still routes but answers 502 for LLM targets.

Access control: the image starts open (every endpoint reachable) so it can act as a sidecar on a
private network. To require a token set `OSR_SERVER_AUTH_TOKENS=<t1>,<t2>` (or mount the list and
point `OSR_SERVER_AUTH_TOKENS_FILE` at it) and, for a hard guarantee, `OSR_SERVER_REQUIRE_AUTH=1` so
the container refuses to start without one. Mint tokens with `osr token generate`; clients send
`Authorization: Bearer <t>` or `X-API-Key: <t>`, and `osr login --url https://<host> --token <t>`
stores one for the CLI (`osr whoami` then shows `edition self-hosted`). A `<t>:ro` entry is a read-only token
(dashboards, CI): it can read everything and ask for decisions but cannot teach the learners or run a target.
`/healthz`, `/readyz`, `/metrics`, `/whoami` and the OpenAPI documents stay open for probes and discovery.

Hardening: the served router takes the production controls from the environment (each one is also a
`serve --...` flag) - `OSR_SERVER_GUARD=1` (input guard: gadget / steering detection, size limits),
`OSR_SERVER_REDACT_PII=1` (PII redacted before routing, originals only reach the executor),
`OSR_SERVER_OUTPUT_GUARD=flag|redact|block` (every answer screened for leaked secrets, PII and instruction injection),
`OSR_SERVER_HEALTH=1` (circuit breakers per target), `OSR_SERVER_AUDIT_PATH=/state/audit.jsonl` (hash-chained
decision trail), `OSR_SERVER_ORIGIN_POLICY=1` (tool-parameter provenance on state-changing tools),
`OSR_SERVER_RESOURCE_LIMITS=1` (per-task step / call / token / cost / wall-clock ceilings) and
`OSR_SERVER_STATE_STORE=redis://...|postgresql://...|sqlite:///state/learners.db` (+ `OSR_SERVER_STATE_KEY_ENV=<VAR>` to encrypt it)
for the learner state instead of files under `OSR_STATE`. Set them in the chart's `env`; the container reports
`Hardening: ...` on start.

Self-operation: `OSR_AUTOPILOT=1` (with `OSR_SLM` and a writable `OSR_STATE`) runs the SLM's
improvement loop inside the container - every `OSR_SLM_AUTOPILOT_INTERVAL_S` seconds (default one
hour) and early when the drift monitor sees the served success rate drop, it gathers evidence
(feedback joined to prompts, cached datasets, seeds), trains a challenger, and hot-swaps it into the
ensemble when it beats the champion on the holdout. The promoted model is written to
`$OSR_STATE/autopilot/slm.json` (or `OSR_CACHE_DIR`), which the entrypoint prefers over `OSR_SLM` on
the next start. `OSR_AUTOPILOT_ARGS` passes extra flags (`--offline`, `--source NAME`,
`--tier TIER=TARGET`, `--search QUERY`, `--feed URL`); `OSR_CATALOGUE` names a catalogue JSON to refresh;
`OSR_SLM_FEEDS=https://...,https://...` lists article feeds (vendor blogs, release notes) the loop reads for new
models. `GET /stats` shows the `autopilot` status and `POST /autopilot/cycle` schedules a cycle now.

Following the marketplace: `OSR_MARKETPLACE_REFRESH_S=86400` (with the optional `OSR_MARKETPLACE_KINDS`,
`OSR_MARKETPLACE_QUERY`, `OSR_MARKETPLACE_TAG`, `OSR_MARKETPLACE_LIMIT`, `OSR_MARKETPLACE_SLUGS`,
`OSR_MARKETPLACE_URL`) makes the server upsert the followed listings into its catalogue on that schedule - a
listing the marketplace dropped is removed, a target you edited locally is kept, and `OSR_MARKETPLACE_PRUNE=1`
also removes what no sync lists any more;
`GET /stats` shows the `marketplace` status and `POST /marketplace/sync` runs one now. Prompt compaction before
every model call is on by default (`OSR_COMPRESSION_ENABLED=0` turns it off; `OSR_COMPRESSION_HISTORY_BUDGET_TOKENS`
adds a conversation budget); tool slots in plans are governed by `OSR_ROUTING_TOOL_SLOTS` (0 = never offer tools).

Observability: every request is traced (spans and events per stage, see
[docs/OBSERVABILITY.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/OBSERVABILITY.md)). `GET /healthz` is the liveness probe (the process is
up) and `GET /readyz` the readiness probe (503 until the catalogue is loaded and a probe request routes end
to end). The image's Docker `HEALTHCHECK` and the Helm chart's `readinessProbe` use `/readyz`, so a
container is only reported healthy / receives traffic once it can route; the chart's `livenessProbe` stays
on `/healthz` so a slow catalogue load never gets the pod killed. `GET /metrics` serves the Prometheus exposition
(`osr_route_decisions_total{target}`, `osr_route_latency_ms_bucket`, `osr_route_policy_rejections_total`,
`osr_route_errors_total`, `osr_outcome_*`, `osr_decision_cache_*`, a `targets` gauge, plus per-stage
`osr_span_duration_ms`), `GET /trace/{request_id}` the full trace of one
request (each response carries `X-OSR-Trace-Id`; send `traceparent` to join your own trace) and
`GET /events?name=route.*` the recent buffer. Tune with `OSR_OBSERVABILITY_MEMORY_EVENTS` (buffer size, `0`
off), `OSR_OBSERVABILITY_LOG_EVENTS=true` (JSON lines on stdout), `OSR_OBSERVABILITY_EVENTS_FILE`
(JSONL under the state volume), `OSR_OBSERVABILITY_OTEL=true` (OpenTelemetry, needs the `otel` extra and
the usual `OTEL_EXPORTER_*` variables) and `OSR_OBSERVABILITY_SAMPLE_RATE` under heavy load. The hosted
platform image (platform/README.md (`platform/README.md`)) reads the same variables and adds the
workspace-scoped `GET /api/v1/trace/{request_id}`, `GET /api/v1/events` and the public `GET /api/v1/status`.

## Helm

The chart is published with every release (`https://opensmartroute.ai/download/helm`, the `helm package` tarball
listed with its SHA-256 on the [downloads page](https://opensmartroute.ai/downloads)):

```sh
curl -fsSLO https://opensmartroute.ai/download/helm            # opensmartroute-<version>.tgz
helm install router opensmartroute-*.tgz \
  --set-file config.targets=targets.yaml \
  --set-file config.rules=rules.yaml
```

Key values (see values.yaml (`deploy/helm/opensmartroute/values.yaml`) for all):

The chart refuses two installs that look fine and are not, naming what to set:

- **no catalogue.** `config.targets` is empty by default and there is no illustrative one behind it, because a
  router installed with invented targets and invented prices would price and choose real traffic against them.
  Pass your own (`--set-file config.targets=targets.yaml`) or point `config.existingConfigMap` at one.
- **an unauthenticated API on a hostname.** `ingress.enabled` without `auth.tokens` or `auth.existingSecret`
  publishes an open router. Mint a token with `osr token generate`, or set `auth.allowAnonymousIngress=true`
  if that is genuinely what you want.

- `config.targets` / `config.rules` / `config.models` -> rendered into a ConfigMap at `/config`;
  the pod restarts on change (config checksum annotation).
- `autopilot.enabled` / `autopilot.intervalSeconds` / `autopilot.args` -> self-operation as above
  (needs `persistence.enabled`; `config.slm` is an optional warm start, otherwise the loop trains a fresh
  model into `/state/autopilot/slm.json` on its first cycle). The autopilot promotes models per pod: with
  `replicaCount` above 1 each replica trains and serves its own champion, so run one replica (or
  one autopilot pod behind a shared `StateStore`) when the loop is on. `autopilot.feeds` -> article feeds the
  loop reads (`OSR_SLM_FEEDS`).
- `marketplace.refreshSeconds` (+ `url`, `kinds`, `query`, `tag`, `limit`, `slugs`) -> follow the marketplace
  (`OSR_MARKETPLACE_*`): new and re-versioned skills and tools land in the served catalogue on that schedule.
- `auth.tokens` / `auth.existingSecret` / `auth.required` / `auth.allowAnonymousIngress` -> access tokens for the
  HTTP API (a chart-managed Secret, or your own with a `tokens` key), whether the pod may start without one, and
  the opt-out that lets an ingress publish an open router on purpose.
- `hardening.guard` / `learnedGuard` / `redactPii` / `outputGuard` (`off|flag|redact|block`) / `health` / `auditPath` /
  `originPolicy` / `resourceLimits` -> the production controls above (`OSR_SERVER_*`), all off by default; put
  `auditPath` under `/state` with `persistence.enabled` so the chain survives restarts.
- `persistence.enabled` -> keep bandit posteriors and the feedback log on a PVC (single replica or
  RWX class). For several replicas leave it off and use the Redis/SQL `StateStore`.
- `autoscaling`, `podDisruptionBudget`, `networkPolicy`, `ingress` -> standard knobs, all off or
  conservative by default.
- `livenessProbe` (`/healthz`) / `readinessProbe` (`/readyz`) / `startupProbe` (`/healthz`, five minutes of grace
  so a large catalogue, slow persistent storage or an autopilot cold start is not killed mid-initialisation) ->
  probe paths and timings; CI lints the chart,
  renders both the default and the fully-enabled value sets through `kubeconform`, and installs the chart with
  the freshly built image on a `kind` cluster (`helm install --wait`, `/readyz` through the Service, the chart's
  token gating `/targets` and `/route`).
- Security: `runAsNonRoot`, `readOnlyRootFilesystem`, all capabilities dropped, `automountServiceAccountToken: false`.

Secrets (MCP manifest signing key `OSR_MCP_KEY`, encrypted-state key) are injected with `env`/`envFrom`
`secretKeyRef`s - never in `values.yaml`.

Endpoints: `GET /healthz`, `GET /readyz`, `GET /metrics`, `GET /whoami`, `GET /targets`, `GET /stats`,
`POST /route`, `POST /feedback`, `GET /v1/models`, `POST /v1/chat/completions` (OpenAI-compatible proxy,
`model: "osr/auto"`, streaming and tool calling), `POST /v1/embeddings`, `POST /mcp` (Model Context Protocol).