What is a load-balanced group?

A model name you define (for example gpt-pool) that maps to 2 to 10 real models. When a request uses that name, Cognocient picks one of the models for that request and forwards it there. Your code never changes per call.

What strategies are available?

Weighted (random in proportion to weights, for example an 80/20 split), round robin (strict rotation across healthy deployments) and lowest cost (the cheapest deployment for each request, from list price and estimated tokens).

What happens when a deployment is failing?

If a deployment returns rate-limit, timeout or 5xx errors 3 times within 60 seconds, it is skipped for 30 seconds. If every deployment in the group is failing, all stay eligible, so you get the provider's real error rather than an artificial one.

Does it retry a failed request on another model?

No. Load balancing steers future requests away from a failing deployment. It does not replay the same request, which would double-count rate limits and budget. For retry on failure, enable Provider Failover.

Which plans include it?

Load balancing is available on Growth and above.

How does Cognocient load-balance across models and providers?

Define one model name that spreads requests across several models or providers by weight, in rotation, or to the cheapest option, and skips deployments that are failing.

A load-balanced group is one model name that fans out to several real models. Use it to split traffic between providers, rotate across equivalent models, or always take the cheapest option. Configure groups in Load Balancer (sidebar → Control).

Set it up

Open Load Balancer and click New group.
Choose the name callers will send, a strategy, and 2 to 10 deployments (each a model name, with a weight for the weighted strategy).
Send the group name as model:

from openai import OpenAI
 
client = OpenAI(base_url="https://api.cognocient.com/v1", api_key="<your proxy key>")
 
resp = client.chat.completions.create(
    model="gpt-pool",
    messages=[{"role": "user", "content": "Summarise this ticket."}],
)

Deployments can mix providers, including Azure deployments, Bedrock and Vertex model IDs, and custom/<name> endpoints, as long as each model has a provider key in Settings. The group works on /v1/chat/completions and /v1/responses.

Strategies

Strategy	Behaviour	Good for
Weighted	Random, in proportion to each weight	Splitting traffic, gradual migrations (90/10 → 50/50)
Round robin	Strict rotation across healthy deployments; weights ignored	Spreading load evenly across equivalent models
Lowest cost	Cheapest deployment for this request, from list price × estimated tokens	Cost-first routing between comparable models

Health and cooldown

Cognocient watches the real outcome of real traffic; there are no synthetic probes. A deployment that returns 429, 408, 500, 502, 503 or 504 three times within 60 seconds is skipped for 30 seconds, then tried again. The Load Balancer page shows a cooling down badge while it is skipped. If every deployment is failing, none are skipped.

State is shared across proxy workers. If it is ever unreachable, load balancing keeps working and treats every deployment as healthy.

What you can see

Every balanced response carries headers:

Header	Meaning
`x-cog-lb-group`	The group name you sent
`x-cog-lb-model`	The deployment that served the request
`x-cog-lb-strategy`	The strategy used
`x-cog-lb-reason`	Why it was chosen, including any deployments skipped

The Load Balancer page shows calls, cost, average latency and errors per deployment over the last 24 hours. Those numbers count all calls to each model, including any made directly rather than through the group.

Combining with other features

Auto Router: an Auto Router tier can point at a group, so "complex" requests are themselves balanced.
Budgets, guardrails, caching, the Emergency Freeze: apply as normal, evaluated against the chosen model.
Provider Failover: independent and complementary. Failover retries one request on a backup provider; load balancing steers later requests away from a sick one.

Limits

2 to 10 deployments per group; a group cannot contain another group.
A group name must not collide with a real model name or your Auto Router name.
Groups span models and providers, not multiple API keys for the same provider (one key per provider per account today).
Weights are integers from 1 to 1000.