Key Features:

• Unified Endpoint: Provides a single OpenAI- and Anthropic-compatible endpoint for over 30 LLM providers, simplifying integration and reducing the need to manage multiple APIs.

• Failover and Routing: Implements robust routing with retries, circuit breakers, and model fallbacks to ensure reliability and high availability.

• Caching: Offers both exact-match and semantic caching to reduce latency and costs by serving repeated requests without hitting the LLM.

• Budgeting and Rate Limiting: Allows setting spending caps and rate limits per user, path, provider, or model to control costs and manage usage.

• Security: Keeps provider API keys secure within the gateway, with applications authenticating using gateway keys that can be rotated or revoked.

• Guardrails and Auditing: Enables the application of request and response guardrails and maintains an audit log of every model call for compliance and monitoring.

• MCP Gateway: Aggregates multiple LLM providers behind a single authenticated endpoint with namespaced tools for efficient management.

• Usage Analytics: Provides detailed insights into token usage and estimated costs across different providers and models.

• Open Source: MIT-licensed, allowing for self-hosting with Docker or a single binary, and the full source code is available on GitHub.

Supported Providers:

Includes a wide range of providers such as OpenAI, Anthropic, Google Gemini, Google Vertex AI, Azure OpenAI, Amazon Bedrock, xAI Grok, DeepSeek, Groq, Cerebras, Fireworks AI, Cohere, OpenRouter, Hugging Face, Cloudflare Workers AI, Oracle GenAI, Alibaba Cloud Model Studio, MiniMax, Z.ai, Xiaomi MiMo, Kimi Code, Meta Llama, ElevenLabs, Chutes, Kilo, OpenCode Go, Hetzner, and ChatGPT subscriptions. Also supports self-hosted models via Ollama, vLLM, llama.cpp, SGLang, and llm-d.

Compatibility:

Works seamlessly with official OpenAI and Anthropic SDKs by simply changing the base URL and using a gateway key. Compatible with various coding agents and chat UIs, including Claude Code, Cursor, Open WebUI, LibreChat, and n8n.

Deployment Options:

Can be deployed with a one-click option to NEXUS AI, self-hosted using Docker, or built from source. Offers a Free plan with SQLite storage, with an option to attach PostgreSQL for persistent data.