Multi‑model routing implements decision‑making logic between application clients and multiple large‑model services. It distributes incoming requests to appropriate models and providers based on cost, latency, capability, quota and failure status. It also performs automatic fail‑over when errors or timeouts occur. As of September 2026, available solutions fall into four distinct layers: local tool‑side routing, self‑hosted gateways, managed aggregation platforms, and intelligent routing algorithms.
The tool‑side routing layer runs on developer local machines, represented by claude‑code‑router (37 000 GitHub stars) and cc‑switch (130 000 GitHub stars). The self‑hosted gateway layer includes LiteLLM (57 790 stars), Kong AI Gateway, Bifrost, Higress, New‑API and TensorZero, supporting more than 100 model back‑ends and six routing strategies. Managed aggregation platforms are third‑party cloud services, including OpenRouter (5.5 % commission), Vercel AI Gateway (zero commission), Cloudflare AI Gateway, and 4sapi. Users can subscribe to 4‑16 model vendors under one single API key. The intelligent routing algorithm layer includes RouteLLM (claimed 85 % accuracy, saving 95 % inference cost), Not Diamond, OpenRouter Auto Router and LiteLLM Auto Router.
Selection should focus not merely on benchmark numbers, but on deciding which layer executes routing logic, how fail‑over triggers, where raw data persists, and maintenance responsibilities. This article compares official documentation and community datasets, delivers comparison matrices, scenario‑based recommendations and a ten‑item evaluation checklist.
What Is Multi‑Model Routing and Why Layered Architecture Matters
Multi‑model routing sits between end‑user applications and LLM service providers. It assigns each request to proper models and vendors, switches targets for failures or timeouts, and unifies authentication logic. It addresses four core pain‑points for mixed‑model production:
- Cost: Pricing gaps between different models can reach 20‑fold. Invoking flagship models for trivial tasks creates unnecessary expenditure.
- Availability: Business continuity cannot rely on a single provider, given rate‑limits, outages or sudden price adjustments.
- Capability matching: Coding, long‑context processing, vision comprehension and reasoning each have their best‑fit models.
- Governance: Centralized consumption tracking, quota management and audit logs are required for multi‑team, multi‑key and multi‑vendor deployments.
Routing logic can run on completely different infrastructure locations, hence the layered design:
| Layer | Execution Location | Target Audience | Representative Projects |
|---|---|---|---|
| Tool‑side local routing | Developer local workstation | Coding‑agent users | claude‑code‑router, cc‑switch |
| Self‑hosted gateway | Private servers / Kubernetes | Company‑wide production workloads | LiteLLM, Kong, Higress, New‑API, Bifrost, TensorZero |
| Managed aggregation | Third‑party cloud platform | Teams avoiding self‑maintenance | OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, 4sapi |
| Intelligent routing algorithm | Embedded inside upper‑layer routing components | Teams expecting complex automatic model selection | RouteLLM, Not Diamond, OpenRouter Auto Router, LiteLLM Auto Router |
Practical stacks commonly combine two layers: tool‑side routing for local developer workstations and self‑hosted or managed aggregation gateways for production traffic.
Layer 1: Tool‑Side Local Routing
Tool‑side local routing works as a local proxy gateway for coding‑agent applications. Requests from Claude Code, Codex and similar agents hit local endpoints, then the proxy forwards traffic toward selected LLM providers. It solves the pain‑point “each agent maintains its own isolated provider configuration”.
| Project | GitHub Stars | Positioning | Routing Capabilities | Supported Agents |
|---|---|---|---|---|
| claude‑code‑router (CCR) | 37 028 | Local model gateway / control plane, default listen address 127.0.0.1:3456 | Route by request headers and conditions; rewrite prompts; retry; fall‑back; token counting; orderly back‑off | Claude Code, Claude Design, Codex, Grok CLI, Kimi CLI, Kilo Code, OpenCode, PI, ZCode, WorkBuddy |
| cc‑switch | 130 629 | Desktop‑side unified configuration manager | Does not proxy traffic itself; one‑click toggle of provider settings; centrally manage API keys and skill sets | Claude Code, Claude Desktop, Codex, Gemini CLI, Grok Build, OpenCode, OpenClaw, Hermes |
Key differences: CCR acts as runtime proxy; all agent traffic flows through it. cc‑switch edits local configuration files of each agent directly; agents send requests to providers without intermediate proxy hops.
CCR supports OpenAI Chat Completions, Anthropic Messages, Gemini Generate Content, OpenRouter, DeepSeek, SilconFlow, Moonshot, Kimi Code, Mistral, ZAI, Bailian plus custom compatible endpoints. Additional features include Fusion for vision‑incapable model adaptation, ToolHub, request logging, provider resolution, latency statistics, token counting and cost estimation. It can be installed as desktop application or via npm install @musistudio/claude‑code‑router. 22 providers are supported as of current release.
Agent Harness also implements routing within this layer. The dsh‑llm‑pi‑ai component in DeepSeek Harness defines multiple routing entries inside a providers dictionary. Each entry configures protocol, endpoint, API key whitelisting and model lists. Configuration updates apply on next request without extra proxy processes. This native‑agent routing serves only one specific agent instance.
Best‑fit scenarios: Individual developers and small teams. Ideal for sharing provider configurations across multiple coding‑agent tools and automatic switching when certain vendors throttle traffic. Not suitable as production‑facing ingress gateway.
Layer 2: Self‑Hosted Gateway
Self‑hosted gateways deploy inside private infrastructure. All model‑access traffic goes through this gateway. Routing, authentication, quota control and logging stay within internal networks. Multiple categories of self‑hosted solutions exist.
Specialized LLM‑Focused Gateways
| Project | Stars | License | Back‑end Support | Routing Strategies | Highlight Features |
|---|---|---|---|---|---|
| LiteLLM | 57 790 | Custom license | 100+ LLMs, unified OpenAI‑compatible API | simple‑shuffle(default), weighted‑latency, usage‑based‑routing, v2 adaptive routing, least‑busy, cost‑based‑routing, custom rules | Built‑in circuit‑breaker(allowed_fails default 3), cooldown_time default 5 seconds, 429 status cool‑down, retry policies mapped to error codes; Key‑level team quota management; Beta Auto Router |
| Bifrost | 7 745 | Apache‑2.0 | 23+ providers, 1000+ model variants | Automatic back‑off, cross‑key & cross‑provider load‑balance, adaptive load‑balancing for enterprise tier | 5 k RPS under 100 ms latency; reported 50‑times faster than LiteLLM; npm deployment; session caching; MCP gateway capability |
| Portkey Gateway(open‑source) | 12 875 | MIT | 1 600+ LLMs, 50+ guardrails | Retry, cross‑key load‑balance, conditional routing, auto‑fallback, circuit‑breaker, request‑timeout handling | Local runtime support; integrated PRISMA AIRS observability; repository last update May‑2026 |
| TensorZero | 11 716 | Apache‑2.0 | Major mainstream providers | Built‑in A/B testing, routing, retry, back‑off | Gateway latency below 1 ms; tracing & observability; evaluation pipelines integrated with private databases |
| New‑API | 47 050 | AGPL‑3.0 | Domestic & international LLMs, compatible OpenAI‑style API | Weighted traffic splitting, automatic fail‑over, user‑level rate‑limit | Format conversion between OpenAI and Claude‑Messages schema; token‑group billing; multi‑login user management |
| One‑API | 36 686 | MIT | Domestic & international LLMs | Channel management and key distribution | Single‑file executable; repository effectively inactive since Jan‑2026 |
Three practical evaluation points:
- LiteLLM delivers the most comprehensive feature set. It implements six routing algorithms, error‑based retry & cool‑down, routing priority, context‑window pre‑check and geographic filtering. Official documentation warns high routing complexity may degrade throughput;
simple‑shuffleremains default strategy. - Bifrost and TensorZero prioritize raw performance. Both implemented in Rust / Go, delivering microsecond‑to‑millisecond overhead, suitable for high‑QPS workloads. Adaptive load‑balancing and cluster‑mode features are locked within enterprise plans.
- New‑API represents widely‑adopted self‑hosted solution in domestic developer communities. Strengths include comprehensive domestic‑model coverage, format conversion and token‑granularity consumption statistics. Routing strategy options are relatively limited (weighted distribution plus retry). Its README explicitly reminds operators to complete filing, real‑name registration and log‑retention compliance requirements for public‑facing deployments.
AI Extensions Built On Cloud‑Native API Gateway Infrastructure
| Project | Stars | Origin | AI Routing Features | Highlights |
|---|---|---|---|---|
| Kong AI Gateway | 44 076 | Kong API Gateway | 7 load‑balancing algorithms: round‑robin, weighted‑hash, least‑connections, lowest‑latency(EWMA), lowest‑usage(cost‑aware), semantic routing(prompt‑model similarity), priority‑based fail‑over | Cross‑provider fail‑over across incompatible‑schema back‑ends; circuit‑breaker max_fails / fail_timeout; configurable error‑redirection trigger; failover_criteria for http‑429; semantic cache, prompt‑guard, PII masking; advanced enterprise‑only modules |
| Higress | 9 270 | Alibaba open‑source, CNCF project, built upon Istio + Envoy | AI routing management: domain matching, weight distribution, downgrade configuration, consumer‑side throttling, per‑provider downgrade rules, health‑check recovery | Pre‑integrated with Alibaba‑cloud, DeepSeek, Azure OpenAI; semantic cache, RAG, prompt templates; consumer‑quota control; real‑time token consumption metrics; remote MCP‑server support; one‑command Docker deployment |
| Envoy AI Gateway | 1 984 | Envoy Gateway sub‑project | Two‑tier gateway pattern: authentication layer and routing layer; global flow‑control; self‑hosted model‑cluster endpoint selector | OpenAI, Azure OpenAI, Gemini, Vertex, Bedrock, Mistral, Cohere, Grok, DeepSeek mixed‑model traffic support |
| Apache APISIX | 17 070 | Apache top‑level project | AI‑gateway plug‑in suite | Re‑use existing APISIX deployment dashboard |
Domestic platforms differ significantly from overseas aggregation services. Overseas aggregators focus on pure routing capabilities. Domestic platforms act more like unified‑entry marketplaces with billing built‑in; routing strategies are mostly closed‑source. Fail‑over logic depends on upper‑layer self‑hosted or tool‑side routing. Three critical selection criteria for domestic platforms: model‑coverage matching requirements, discount & quota‑curve alignment, and protocol compatibility with OpenAI / Anthropic formats (determines direct integration with Claude‑Code‑class agents).
Layer3: Managed Aggregation Platforms
Managed aggregation platforms are fully‑managed third‑party services. Developers submit requests to one single API entry‑point; the platform handles multi‑provider traffic forwarding, quota splitting and fail‑over. Representative services: OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway, 4sapi. Users subscribe to multiple vendors under one key.
Advantages: Zero operation overhead; ready‑built multi‑vendor pool. Drawbacks: data passes through third‑party infrastructure; custom routing logic may be limited by platform capabilities; extra commission fees may apply. Teams with strict data‑sovereignty constraints need to review data‑residency clauses carefully.
Layer4: Intelligent‑Routing Algorithm Layer
Intelligent‑routing introduces classification or statistical models to decide target model selection. Simple requests get cheaper lightweight models; complex reasoning tasks go to flagship models, instead of purely static manual rules.
| Solution | Form | Decision Input | Public Evaluation Metrics | Status |
|---|---|---|---|---|
| RouteLLM | Open‑source framework, Python SDK & OpenAI‑compatible service | Prompt complexity classification; configurable cost‑vs‑accuracy thresholds | MT‑Bench evaluation: matching flagship‑model quality with 85 % cost reduction; 95 % quality retention | Repository last updated Aug‑2024; research prototype |
| Not Diamond | Commercial API, oriented for programming‑agent scenarios | Predict optimal target model for given input, executed inside gateway or Harness | 20 %‑55 % inference‑cost reduction; accuracy improvement reported | SOC2‑ISO‑27001 certified; commercial paid service |
| OpenRouter Auto Router | Built‑in OpenRouter feature, no extra‑cost | Light‑weight classifier categorizes prompts into ~30 task types; selects optimal model by cost/quality trade‑off | No independent public accuracy figures | Beta feature |
| LiteLLM Auto Router | Embedded inside LiteLLM self‑hosted gateway, Beta | Multi‑dimensional classifier: prompt metadata, token statistics, task‑type tags; four preset complexity grades(SIMPLE / MEDIUM / COMPLEX / REASONING) | No independent public accuracy metrics; documentation warns accuracy may evolve | Beta feature |
Practical Deployment Scenarios
- SaaS multi‑tenant scenario: Use managed aggregation such as 4sapi as upstream. Combine self‑hosted gateway for fine‑grained per‑tenant quota splitting. Connect two‑to‑three official‑model APIs plus one aggregation‑provider for fail‑over. Apply weighted‑shuffle routing plus error‑based cool‑down. For heavy domestic‑model workloads, treat aggregation service as one upstream entry‑point and steer low‑priority traffic toward it.
- Cost‑first public‑facing applications: Adopt managed‑aggregation providers, set routing sort key to
costorttft. Enable model‑level automatic fail‑over. - Domestic enterprise compliance requirements (no cross‑border data outflow, audit requirements): Self‑host gateway as front‑end; only connect domestic‑model APIs and domestic aggregation endpoints. Apply Higress token‑throttling and consumer‑quota management to handle failure and cost control. New‑API works well for pure domestic‑model internal distribution.
- Highly uneven prompt‑complexity distribution: Activate intelligent‑router mode. Run LiteLLM Auto‑Router in observation mode for two weeks, collect
cause=tag distribution inside logs, then decide whether to enable LLM‑based classification or stick to static rule‑based routing. Do not enable intelligent routing directly in production without observation cycles.
Ten‑Item Evaluation Checklist
Before adopting any multi‑model‑routing solution, validate each checklist item:
- Protocol compatibility: Support OpenAI Chat Completions and Anthropic Messages simultaneously? Determines compatibility with Claude‑Code / Codex‑style tool‑chains.
- Fail‑over trigger conditions: Which error codes trigger provider switching? Whether connection time‑outs count as failures. LiteLLM implements special cool‑down logic for status‑429 responses.
- Routing granularity: Whether you can allocate traffic according to user groups, model types, or request‑tags.
- Provider‑key mapping: One‑model supports multiple rotating API‑keys? Can traffic‑volume split ratio be defined per‑key.
- Cache capability: Prompt‑cache transmission support (Anthropic‑style marker headers); semantic‑cache capability. Cache can deliver up to 4‑fold cost‑reduction for workloads such as GLM‑5.3.
- Fee‑structure breakdown: Model input/output rates, recharge commission, subscription discounts, BYOK costs. OpenRouter charges 5.5 % commission; Cloudflare collects 5 %; Vercel offers zero‑commission tiers; 4sapi offers preferential pricing.
- Quota‑rule details: Whether subscription‑quota resets monthly; support for downgrading subscriptions.
- Data‑residency: Zero‑data‑retention options? OpenRouter
zdr=trueflag; Vercel no‑persistence toggle. - Observability: Log retention period, log‑storage quota limits. Can logs expose resolved‑actual‑provider and real‑model identifiers after routing resolution.
- Maintenance vitality: Repository latest‑push timestamp. One‑API and RouteLLM show obvious stagnation; Portkey open‑source repository saw no commits for 5‑months at time‑of‑writing. Always verify activity before selection.
Frequently‑Asked‑Questions
Q: Are multi‑model‑routing and LLM gateway the same concept?
A: Not exactly. Gateway describes deployment form (unified entry‑point). Multi‑model‑routing is one capability inside gateways (model‑selection and provider‑switching). Tool‑side‑router and Agent‑Harness built‑in provider‑routing implement routing functions without being full‑featured gateways. Managed‑aggregation platforms act as both gateway and routing‑engine.
Q: How does weighted‑load‑balancing work in OpenRouter‑style implementations?
A: Probabilistic weighted sampling. Example: $1‑per‑million‑token provider gets selected nine‑times more frequently than $3‑per‑million‑token counterpart. order parameter switches from probabilistic sampling to strict sequential attempt order.
Q: Can self‑hosted gateways achieve high‑availability?
A: LiteLLM itself does not deliver native HA. Bifrost enterprise‑cluster‑mode, Kong and Higress offer mature multi‑replica deployment patterns. These are cloud‑native gateway‑extension capabilities, not core‑routing‑engine functions.
Q: Can domestic‑aggregation‑platforms serve as complete gateways?
A: Managed‑aggregation‑platforms mostly expose closed‑source routing logic without user‑configurable strategies. Use them as upstream back‑ends behind self‑hosted or tool‑side‑routing layers. Their value lies in discount procurement rather than flexible routing capability.
Q: Is intelligent‑routing worth deploying?
A: It brings value when over 70 % of total traffic consists of simple prompts and you can tolerate minor quality fluctuations. RouteLLM reports 85 %‑cost‑saving on MT‑Bench test‑set, but cannot guarantee identical effect for your real‑world prompt distribution. Always run observation‑mode first.
Q: One‑API vs New‑API?
A: New‑API inherits core One‑API logic and keeps active updates in 2026, adding format‑translation, cache‑billing and user‑rate‑limiting functions. One‑API repository stopped meaningful iteration after Jan‑2026.
Conclusion
Multi‑model‑routing selection should follow layer‑by‑layer logic: Developers use claude‑code‑router locally to manage coding‑agent switching and fallback. Production‑traffic for companies deploys self‑hosted gateways such as LiteLLM or Kong AI Gateway. Teams unwilling to operate infrastructure adopt managed‑aggregation platforms like OpenRouter, Vercel AI Gateway or 4sapi.
For self‑hosted gateways: Kong AI Gateway offers most comprehensive feature coverage. Higress and New‑API deliver best domestic‑model compatibility. For managed‑aggregation platforms: OpenRouter implements most complete load‑balance algorithms. 4sapi presents competitive pricing plans for domestic‑model discounted procurement.
If you adopt LiteLLM, pay attention: its default simple‑shuffle policy distributes traffic equally by provider count, not by token‑volume. Traffic proportion may deviate from expectation when model‑token consumption differs drastically. Always audit real‑world traffic metrics after launch.
International access: https://4sapi.com
Domestic access: https://4sapi.cn
