<!-- OpenSmartRoute: Architecture. https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/ARCHITECTURE -->
# Architecture

OpenSmartRoute is a **decision layer**, not a gateway. Given a request and a catalogue of targets it
returns *which target should handle this, how confident we are, and why* — and it learns from what
happened next. Executing the target is optional and pluggable.

```mermaid
flowchart LR
    subgraph In
        REQ[RouteRequest]
    end
    subgraph Middleware
        G[Guard] --> T[Tenant] --> C[Cache]
    end
    subgraph Core
        S[Signals] --> P[Policy] --> ST[Strategies] --> E[Ensemble] --> D[Decision + Trace]
    end
    subgraph Learn
        O[Outcome] --> AL[AutoLearner] --> ST
        O --> H[Health] --> P
    end
    REQ --> G
    C --> S
    D --> X[Executor / handler]
    X --> O
    CAT[(TargetRegistry)] --> P
```

## Layers

| Layer | Package | Responsibility | Depends on |
|---|---|---|---|
| **Core model** | `core` | `RouteTarget`, `RouteRequest`, `RouteDecision`, `RouteTrace`, `Outcome`, `TargetRegistry` | stdlib |
| **Signals** | `signals` | Deterministic features of a request: complexity, domain/action, language, modality, PII, jailbreak, token estimate; opt-in verbalised difficulty, draft-response and response-uncertainty features (`signals.uncertainty`) | core |
| **Policy** | `policy` | Hard constraints → admissible set + rejection reasons. Never traded off. | core, signals |
| **Strategies** | `strategies` | Soft scorers ∈ [0,1] with rationale: rules, capability fit, similarity, Thompson bandit, LLM judge, cascade executor; opt-in hidden-state probe, token budgets, edge/cloud tiers, auctions; post-answer controls (self-escalation, mixture-of-agents, protocol selection) | core, signals |
| **Learning** | `learning`, `math` | Strategies that update from outcomes (IRT, Bradley–Terry, LinUCB, Markov/MDP, multi-turn history embeddings, per-user shrinkage) and the pure-math they rest on (bandits, calibration, Dirichlet probe, energy model, mixture-cure hand-off); `AutoLearner` fan-out, drift, atomic persistence; the routing SLM (`RouterSLM`, `SLMStrategy`, `distill_router`) and its champion/challenger `SelfImprover` | strategies |
| **Discovery** | `discovery`, `retrieval` | Retrieve-then-rank narrowing, schema-aware tool matching, cache-preserving tool order, skill graphs and submodular skill sets for large catalogues | core, strategies |
| **Router** | `router` | Orchestrates signals → policy → strategies → confidence-weighted ensemble → decision; optional judge escalation; optional plan (persona → skill → model) | all above |
| **Real-time** | `realtime` | Circuit breaker, token bucket, rolling budget, `HealthRegistry`; `HealthPolicy` (hard) and `HealthStrategy` (soft) | policy, strategies, math |
| **Enterprise** | `enterprise` | `RouterBuilder`, `EnterpriseRouter`, middleware chain, telemetry/state/audit ports and in-process adapters | router, realtime, learning |
| **Security** | `security` | `InputGuard` (size, normalisation, confounder gadgets), `Redactor`, `GuardMiddleware`, `OutputGuard` (response-side PII / secrets / injection: flag, redact or block), `moderate()` behind `/v1/moderations`, `OriginPolicy` (tool-parameter provenance), `ResourceLimits` (steps, tool calls, tokens, cost per task), prompt sanitiser, secret loading, the safety red-team suite | enterprise |
| **Adapters** | `adapters` | OpenAI-compatible HTTP client + glue (judge, embedder, handler); agent harnesses; MCP clients (stdio and Streamable HTTP: tools, resources, prompts as targets) and server recommendation, A2A cards (task polling), LangGraph and Microsoft Agent Framework glue, SKILL.md and persona loaders, semantic-router import; marketplace client and `MarketplaceSync`; live model catalogue (OpenRouter prices, Hugging Face cards) and web-search knowledge (https-only, byte-capped, risk-scored); optional sentence-transformers and OpenTelemetry | strategies, enterprise |
| **Config** | `config` | JSON/YAML loaders for targets and rules with validation errors | core, strategies |
| **Protocols** | `responses`, `anthropic`, `media`, `execution`, `compress`, `estimate` | One routed chat call behind several client dialects: OpenAI Chat Completions and Responses, Anthropic Messages, speech / transcription / image generation; the plan executor with tool rounds and fallbacks; prompt compaction; the pre-call cost estimate | router, adapters |
| **Surface** | `aio`, `server`, `mcp_server`, `cli`, `stack`, `ocm`, `eval` | Async façade; FastAPI app (`/route`, `/feedback`, `/v1/*`, `/mcp`, tokens, guard / health / audit / state-store flags); the MCP server (stdio and Streamable HTTP); the `osr` CLI; declarative stacks; Open Capability Manifests; RouterBench-style evaluation, off-policy evaluation and dataset collection from the Hugging Face Hub (`eval.collect`) | router, enterprise |

Dependency direction is strictly downward in this table. `math` depends on nothing but stdlib and
is usable on its own.

## The decision pipeline, precisely

1. **Signals** — `extract_signals(request)`; ~0.1 ms; no I/O.
2. **Candidate pool** — `registry.all(primary_only=True)`, optionally filtered by `kinds`.
3. **Policy** — `Policy.filter(pool)` → `(admissible, rejections)`. `HealthPolicy` adds breaker/rate/budget
   checks (non-consuming). Empty admissible set → `NoRouteError` with the reasons.
4. **Pinned rules** — a matching `Rule(pin=True)` narrows the admissible set (Arch-Router semantics).
5. **Scoring** — every strategy returns `{target_id: StrategyScore(score, rationale, confidence)}`.
6. **Ensemble** — quality estimate $\hat q = \sum_k w_k\kappa_k s_k / \sum_k w_k\kappa_k$ where $\kappa$ is
   the strategy's self-confidence (learners report $\kappa=\min(1,n/n_0)$, so they are silent until they
   have evidence). Utility $U=\,w_q\hat q - w_c\,\tilde c - w_\ell\,\tilde\ell$ with log-min-max normalised
   cost/latency and a hard quality floor.
7. **Confidence** — softmax margin over utilities ($\tau=0.1$). If below `escalate_llm_judge_below` and a
   judge is configured, re-score once with the judge included.
8. **Decision** — top target, ranked alternatives, full trace. Optionally a `RoutePlan` where each slot
   (persona, skill, llm, tool) is its own sub-routing over that kind; a slot is left empty when its best
   candidate's quality estimate is below `slot_quality_floor` (`tool_quality_floor` for tools), so irrelevant
   skills and tools are never attached. A model or agent primary gets up to `tool_slots` tool slots - the
   functions it may call.
9. **Execute** (optional, `execution.py`) — `Router.run()` walks the plan: the `skill` slot's handler runs as a
   pre-processor (may rewrite the request or attach `skill_output`), persona/skill/primary `instructions` are
   composed into `context["system"]`, tool slots become function-calling specs in `context["openai"]["tools"]`,
   the prompt is compacted (`compress.py`: whitespace, repeated lines, history budget; code fences untouched),
   then the primary handler is called (`chat_handler`, a skill function, or an agent harness via
   `CallableHarness`/`HTTPHarness`/`SubprocessHarness`). Tool calls the model makes to offered tools are run and
   fed back for up to `tool_rounds` rounds. One `Outcome` per participant (`role=None|"persona"|"skill"|"tool"`,
   shared `task_id`) is recorded — success, latency, cost from tokens × unit cost.
10. **Learn** — `Outcome` → every strategy's `update()`, drift detectors, health registry, feedback store.

## Extension points

| To add… | Implement | Register via |
|---|---|---|
| a target | `RouteTarget(...)` or a YAML entry | `registry.add()` / `load_targets()` |
| a signal | `SignalExtractor.extract()` | `Router(extractors=[...])` |
| a hard rule | `Policy.check()` → reason or `None` | `RouterBuilder.with_policy()` |
| a scorer | `Strategy.score()` (+ `update()` if it learns) | `with_strategy(s, weight)` |
| pre/post processing | `Middleware.__call__(request, next_)` | `with_middleware()` |
| metrics/tracing | `Telemetry.on_decision/on_outcome/on_error` | `with_telemetry()` |
| span/event capture | `EventSink.emit(event)` (+ `span_start/span_end` for live bridges) | `with_tracing(sink)` / `configure_tracing(sink)` |
| persistence | `StateStore.get/put/delete` | pass to learners / caches |
| a provider | `RouteTarget.handler = chat_handler(client, model)` | `adapters.OpenAICompatClient` |

## Concurrency model

- `Router.route()` is pure with respect to router state (no writes) and safe to call from many threads.
- Learners, health registry, caches and stores take their own locks on write.
- `AsyncRouter` runs routing in the default executor and awaits coroutine handlers.
- Multi-replica: learner updates are commutative sums (Beta counts, IRT gradients are small and
  order-insensitive in practice, BT strengths, LinUCB `A`/`b`, Markov counts), so replicas can be folded
  together with `merge()` / `learning.merge_learners()`, or readers can adopt a writer's snapshot through a
  `StateStore` with `AutoLearner.refresh()`. Corrupt snapshots are quarantined, never fatal
  (`AutoLearner.quarantined`).

## Performance envelope

Measured with the reproducible routing benchmark, 300 requests per configuration (CPython 3.11, x86-64 Windows laptop, pure
Python, no C extensions; 2026-10-09). Numbers are wall-clock per `route()` including signal extraction, policy,
every strategy and the ensemble; re-run the benchmark on your own hardware before sizing. The continuous build
runs the same benchmark and fails when the full configuration's p95 passes 50 ms, so the overhead cannot drift
up unnoticed.

**27-target example catalogue with its rules** (`examples/targets.yaml`, `examples/rules.yaml`, `examples/eval_dataset.jsonl`):

| Configuration | p50 | p95 | p99 |
|---|---|---|---|
| rules + capability + similarity + bandit | 4.6 ms | 6.6 ms | 8.3 ms |
| + IRT + preference + LinUCB + Markov + health | 7.4 ms | 9.1 ms | 14.1 ms |

**Scaling with catalogue size** (synthetic LLM catalogue, defaults + auto-learning). Cost is linear in
admissible targets x strategies until retrieval narrowing (BM25 + hashed dense, reciprocal-rank fusion)
caps the candidate pool; the default `Settings.routing` only narrows above 500 admissible targets (to 50),
`RouterBuilder.with_retrieval(narrow_above=32, narrow_to=24)` turns it on earlier:

| Targets | Narrowing | p50 | p95 | p99 |
|---|---|---|---|---|
| 16 | default (none) | 9.2 ms | 9.5 ms | 10.0 ms |
| 16 | top-24 above 32 (none triggered) | 9.2 ms | 9.6 ms | 10.5 ms |
| 64 | default (none) | 35.7 ms | 36.6 ms | 37.1 ms |
| 64 | top-24 above 32 | 14.5 ms | 15.1 ms | 15.6 ms |
| 256 | default (none) | 142 ms | 145 ms | 158 ms |
| 256 | top-24 above 32 | 16.7 ms | 17.6 ms | 18.5 ms |
| 1024 | default (auto top-50) | 38.7 ms | 43.5 ms | 49.1 ms |
| 1024 | top-24 above 32 | 24.1 ms | 28.2 ms | 32.9 ms |

Rule of thumb: keep the scored pool at or below ~50 targets and routing stays under 25 ms p99 in pure
Python regardless of catalogue size; the retrieval stage itself is ~10 ms at 1k targets and grows
roughly linearly in the catalogue. The LLM judge, when triggered, adds one provider round-trip; keep
`escalate_llm_judge_below` low (<= 0.5) so it fires only on genuinely ambiguous requests.

## Deployment shapes

1. **Library** — import in-process. Lowest latency. State on local disk or in-memory.
2. **Sidecar** — `osr serve` per application (container image from `deploy/Dockerfile`); shared
   `StateStore` for global learning.
3. **Control plane** — one HA deployment (the published Helm chart); a gateway
   (LiteLLM, Envoy, aisix) calls `/route`, executes the chosen target, and posts `/feedback`.
4. **Hosted platform** — the same router behind accounts, API keys, plans and quotas, BYOK providers,
   tenants, SSO, billing and the dashboard ([PLATFORM.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/PLATFORM.md)),
   reached over REST, the `/v1` dialects, MCP, the Python CLI (`osr login`) or the TypeScript client
   (`@opensmartroute/sdk`, which also targets an `osr serve`).

Multi-replica learning: run one *writer* replica that learns and saves through a `StateStore`, and have
readers call `AutoLearner.refresh()` on a timer; or let every replica learn independently and fold them
together periodically with `learning.merge_learners()`. See [deploy/README.md](https://osr-web.kindrock-caaab1a2.centralindia.azurecontainerapps.io/docs/deploy.md).

## Non-goals

Model serving, prompt versioning, chat UI, and provider-to-provider protocol translation as a product:
the `responses`, `anthropic` and `media` modules exist so that clients speaking the OpenAI or Anthropic
dialects can reach the router with one base-URL change - each becomes one routed chat call, not a
general translation layer.