OpenAI Compatible Proxy for AI Routing
An openai compatible proxy allows engineers to route AI requests through a customizable, open-source layer that integrates with the official OpenAI SDKs. This proxy acts as a drop-in replacement for the OpenAI API endpoint, enabling flexible routing, fallback, and management of AI requests.
What is an OpenAI Compatible Proxy?
An openai compatible proxy is a server that mimics the OpenAI API interface, allowing clients to send requests as if they were communicating directly with OpenAI. The proxy intercepts these requests, routes them to appropriate backend models or services, and returns responses in the expected format.
How to Set Up the Proxy with OpenSmartRoute
The osr serve command creates a proxy server that accepts requests at a base URL, such as http://router:8000/v1. Clients configured with the official OpenAI SDK can point to this URL, maintaining compatibility while gaining routing flexibility.
Example Configuration
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="osr_local_...")
client.chat.completions.create(model="osr/auto", messages=[{"role": "user", "content": "..."}])
This setup allows seamless integration with existing OpenAI SDKs, with the proxy handling the routing logic behind the scenes.
Features of the OpenAI Compatible Proxy
The proxy supports various features that enhance routing and fallback capabilities:
- Routing with fallback: Uses
execution.execute_with_fallbackto reroute requests if the primary route fails. - Streaming support: Handles OpenAI chunk events via
execution.ChatStreamandcompletion_chunks. - Request and response handling: Manages body conversions with
server.proxy_request,proxy_response, andembeddings_response. - Compatibility with real clients: Tested against official clients for models, chat, streaming, tool calls, embeddings, and token rejection scenarios.
Using the Proxy with Different Providers
The SDK includes a bundled client that supports OpenAI, Azure OpenAI, vLLM, Ollama, LiteLLM, OpenRouter, Groq, and Mistral, all via the same interface. This allows routing requests to various backend providers transparently.
Comparing OpenAI Compatible Proxy Solutions
| Feature | OpenSmartRoute Proxy | Other Solutions |
|---|---|---|
| Compatibility | Fully compatible with OpenAI SDKs | Varies |
| Routing flexibility | Supports fallback and custom routing | Limited |
| Open source | Yes | No |
| Support for multiple providers | Yes | Limited |
Frequently Asked Questions
What is needed to deploy the proxy?
You need to run osr serve with a configured router. The router defines how requests are routed and handled.
Can I use the proxy with existing OpenAI SDKs?
Yes, the proxy is designed as a drop-in replacement, requiring only the base URL change.
Is the proxy suitable for production?
Yes, with proper hardening and configuration, it can be used in production environments.
Next Steps
Learn more about deploying and hardening your proxy in the Enterprise guide.



