Ollama, vLLM and LM Studio behind an OpenAI-compatible LLM gateway
Put self-hosted models (Ollama, vLLM, LM Studio, llama.cpp, TGI) behind the same gateway as OpenAI and Claude. One API key, the custom/ model prefix, optional per-token rates for GPU cost, and budgets by call volume.
Self-hosted models usually live outside your cost and access controls: a separate URL, no keys, no budget. Routing them through Cognocient AI puts them in the same dashboard, under the same proxy keys and limits, as your hosted providers, and lets you compare a local model's quality-adjusted cost with GPT or Claude on real traffic.
Setup
In Settings → AI Providers → Custom:
| Field | Example | Notes |
|---|---|---|
| Base URL | https://llm.example.com/v1 | /chat/completions is appended unless the URL already ends with it |
| API key | my-server-key | Optional. Leave blank and no Authorization header is sent |
| $ / 1M input, output | 0.20, 0.60 | Optional. Your effective cost per token |
Server-specific base URLs:
| Server | Typical base URL path |
|---|---|
| Ollama | https://<host>/v1 (Ollama's OpenAI-compatible API) |
| vLLM | https://<host>/v1 (vllm serve exposes it by default) |
| LM Studio | https://<host>/v1 |
| llama.cpp server | https://<host>/v1 |
| Text Generation Inference | https://<host>/v1 |
Then call it with custom/ in front of the model name your server knows:
Streaming works. The gateway asks for a final usage chunk (stream_options.include_usage); servers that don't send one are tracked without token counts.
Public HTTPS only
To prevent server-side request forgery, URLs that resolve to localhost, private networks (10.x, 172.16–31.x, 192.168.x), link-local or cloud-metadata addresses, or that embed credentials are rejected, both when you save and on every request. For a model on your laptop or private network, use Cloudflare Tunnel, ngrok, Tailscale Funnel or a TLS reverse proxy, and set an API key on it.
What applies to self-hosted calls
| Feature | Self-hosted |
|---|---|
| Attribution headers, Live Calls, latency | ✅ |
| Budgets | ✅ in dollars if you set rates; otherwise they see $0 |
| Velocity circuit breaker, Emergency Freeze | ✅ every plan |
| Content guardrails (PII, prompt injection) | ✅ Base and above |
| Waste and model-mismatch detection | ❌ (no price tiers to compare) |
| Load balancing with hosted models | ✅ Growth and above, e.g. 80% to custom/... and 20% to gpt-4o-mini |
Self-hosted as a fallback, or the other way round
A common pattern is a local model for the bulk of cheap calls and a hosted model for the hard ones. On Growth and above, the Auto Router can pick between them per request by complexity, and load balancing skips a deployment that is failing, so an overloaded GPU box doesn't take the feature down.
Related articles
Supported Providers
OpenAI, Anthropic, Gemini, Mistral, Groq, Together, Azure, AWS Bedrock, Vertex AI, and custom OpenAI-compatible endpoints.
OpenAI
OpenAI SDK through the gateway: chat, Responses, embeddings, images, audio and batches with per-feature cost and budgets.
Anthropic (Claude)
Claude through the OpenAI SDK (translated) or the native Anthropic SDK (passthrough), with prompt-cache-aware cost.