The Multi-Provider Routing Playbook

I. Five decisions, one model name

A team that has connected two or three AI providers directly — an OpenAI key here, an Anthropic key there, maybe a Bedrock role for a compliance-sensitive workload — usually solves routing with an if/else block in application code: this feature calls GPT-4o, that one calls Claude, and nobody touches it again until a model is deprecated or a bill looks wrong. That works until the routing question stops being static. Which provider should a call go to today, given what's actually up right now? Which specific model is worth its price for this particular request, not every request of this type? Which of several equivalent deployments has spare capacity this second? Which of several API keys for the same provider hasn't hit its rate limit in the last minute? A hardcoded if/else answers none of these, because all four questions are about real-time conditions an application shouldn't need to poll for itself. Cognocient answers each one as a separate, composable layer sitting behind a single model name or endpoint your code already calls.

II. Which provider: ten provider types behind one base_url

The first and broadest layer is provider selection itself. Cognocient proxies ten provider types — OpenAI, Anthropic, Google Gemini, Mistral, Groq, Together AI, Azure OpenAI, AWS Bedrock, Google Vertex AI, and any OpenAI-compatible server registered as a custom endpoint — through a single base_url. The provider is detected automatically from the model name: a vendor-namespaced ID like anthropic.claude-3-5-sonnet-... or amazon.nova-pro-v1:0 routes to Bedrock, a vertex/ prefix disambiguates Vertex AI's Gemini from a direct Gemini key, and a custom/ prefix selects a self-hosted server such as Ollama or vLLM. Switching a feature from GPT-4o to Claude, or from a direct Gemini key to the same model served through Vertex, is a one-string change to the model name — the base_url, the proxy key, and the rest of the calling code stay identical. Unlimited provider connections are available starting on the Base plan.

Self-hosted models are a distinct case worth calling out here: because they have no public list price, calls to a custom endpoint are recorded at $0 unless per-1M-token rates are entered manually, and waste or model-mismatch detection simply doesn't apply to them, since there is no price tier to compare against. Everything else — volume, latency, feature and department attribution, budget counts — still works the same as for a metered provider.

III. Which tier: the Auto Router's deterministic classifier

Picking a provider says nothing about picking the right model within it. A support-ticket summarizer and a multi-step code refactor are both “just a chat completion,” but sending both to the same frontier model means paying frontier prices for the easy one. The Auto Router solves this by letting a team define one router model name — Simple, Medium, Complex, and an optional Reasoning tier — and assign a real model to each. A request sent to the router name is classified by a deterministic heuristic running inside the proxy itself: signals like context length, the presence of code blocks or stack tracebacks, big-task wording (“refactor,” “architect,” “security review”), a multi-part request, or explicit reasoning cues (“prove,” “derive,” a differential equation) push the score toward a higher tier, while short factual phrasing pulls it back down. Because the classifier makes no model call of its own, it adds no provider cost and effectively no latency, and the same request always classifies the same way — a property a learned classifier can't promise.

Every routed response carries the tier chosen, the real model used, and the specific signals that decided it, so a misclassification is explainable rather than a black box — and a keyword override (“contains ‘contract’ → Reasoning”) or an X-Cost-Priority: high header can force a specific tier regardless of what the classifier would have picked. Budgets and semantic caching evaluate against the real model that actually serves a routed request, not the router's own name, so cost enforcement and cache keys stay accurate underneath the abstraction. The Auto Router is available on the Growth plan and above.

IV. Which deployment: load balancing by weight, rotation, or cost

Once a provider and a tier are settled, a third question remains whenever more than one deployment can equally serve a request: how should traffic split across them. A load-balanced group is a model name that fans out to 2 to 10 real models — which can themselves span providers, Azure deployments, Bedrock and Vertex model IDs, and custom endpoints — under one of three strategies. Weighted balancing splits traffic randomly in proportion to configured weights, useful for a gradual migration from a 90/10 split toward 50/50. Round robin ignores weights and rotates strictly across healthy deployments, for spreading load evenly across models a team considers interchangeable. Lowest cost picks whichever deployment is cheapest for a given request, computed from list price and estimated tokens, for teams that want cost to be the deciding factor whenever quality is comparable.

Health is measured from real traffic, not synthetic probes: a deployment that returns a rate-limit, timeout, or 5xx error three times within 60 seconds is skipped for 30 seconds, and if every deployment in a group is failing at once, all of them stay eligible — the caller gets the provider's real error rather than a manufactured one. An Auto Router tier can itself point at a load-balanced group, so a “complex” request is classified once and then balanced across several complex-tier deployments. Load balancing is available on the Growth plan and above.

V. Which key: balancing traffic across multiple keys for one provider

A provider rate limit is enforced per API key, which means a load-balanced group spread across several deployments of the same provider can still bottleneck on a single key long before the provider itself is under strain. Multi-key balancing addresses this one level below deployment selection: extra API keys — from different projects or organizations — are added next to a provider's primary key, each with an optional weight, and Cognocient spreads that provider's requests across all of them. A key with weight 3 receives three times the traffic of one at weight 1, and pausing or removing a key takes effect within about 15 seconds.

The health model mirrors load balancing at a stricter threshold: two failures within 60 seconds — rather than three — mark a key as rate-limited and skip it for 45 seconds, since a key-level 429 is a narrower, faster-moving signal than a deployment-level outage. The two mechanisms compose rather than compete: a load-balanced group's deployments each draw from their own provider's key pool, so a group can be spreading traffic across models and keys simultaneously. Multi-key balancing is available on the Growth plan and above, for plain-API-key providers; Azure, Bedrock, Vertex AI, and custom endpoints carry endpoint or region settings a bare key can't substitute for.

VI. When a provider is down: opt-in cross-provider failover

The first four layers all resolve a routing decision before a request is sent. Provider Failover is different: it is a response to what happens after a request has already gone out and the provider it reached is timing out, rate-limiting, or returning a 5xx. When enabled, Cognocient retries the same call exactly once against a different, already-connected provider — never against the same provider again, and never more than once. If the fallback also fails, the caller gets the original error back, indistinguishable from failover being off entirely, rather than a confusing second error or a cascading retry chain. A 401 never triggers it, deliberately: a bad key is a configuration problem, and silently failing over would hide the real fix rather than surface it.

It is off by default, because some teams operate under data-residency or vendor constraints that make an automatic hop to a different provider's infrastructure unacceptable without an explicit decision to allow it. It is also easy to confuse with Degrade mode, but the two solve different problems on different triggers: Degrade responds to a budget threshold by moving to a cheaper model on the same provider, a cost response; Failover responds to an outage by moving to an equivalent-tier model on a different provider, an availability response. The two can be active simultaneously without conflict, since nothing about one condition implies the other.

VII. The stack, together

None of these five layers is optional in the sense that skipping it is free — a team that never configures the Auto Router just pays frontier prices for every request regardless of complexity; a team that never load-balances is one deployment's outage away from a full feature going down; a team that never pools keys has a single API key's rate limit as its real ceiling no matter how much quota the provider grants on paper; a team that never enables failover accepts a provider outage as its own outage. What makes the stack tractable is that each layer is independently optional, independently configurable, and resolves entirely behind one model name or endpoint — application code calls the same client, with the same base_url and the same proxy key, on day one with none of this configured and on day two hundred with all five layers active. The routing decision moves from a hardcoded if/else in application code to configuration that changes without a deploy, which is the entire point of putting a proxy in front of every provider in the first place.