Understanding Model Routing Policy in OpenSmartRoute

A model routing policy determines how requests are directed to different models within OpenSmartRoute. It helps manage model selection, enforce constraints, and optimize performance.

This guide explains the concept of model routing policies, their role in the routing process, and how to configure them effectively.

What is a Model Routing Policy?

A model routing policy is a set of rules and constraints that guide the routing layer in selecting appropriate models for each request. It ensures requests are handled according to predefined criteria such as quality, cost, latency, or safety.

OpenSmartRoute does not automatically authenticate callers or encrypt state at rest. Instead, it relies on policies to control routing behaviors, including safety and resource management.

Components of a Routing Policy

Routing policies in OpenSmartRoute are composed of several elements:

• Constraints: Hard rules that define admissible models based on request features.
• Objectives: The goal of routing, such as maximizing quality or minimizing cost.
• Rules: YAML or decorator-based rules that specify conditions and target models.
• Decision logic: The process that evaluates constraints and objectives to select a model.

Configuring a Model Routing Policy

Routing policies are often defined in a rules file (rules.yaml) or through decorators (@rule). Here is an example configuration snippet:

rules:
  - name: default_policy
    constraints:
      - model_type: safe
    objectives:
      - quality: 1
      - cost: 0.6
    default_target: model_A

This configuration sets constraints and objectives, guiding the router to prefer models that meet safety and quality criteria.

Comparing Routing Strategies