How does an LLM gateway for cost control work? Architecture overview
How the Cognocient AI gateway works: the proxy request path, where budgets and guardrails are enforced, what is logged, and what each dashboard section is for.
Cognocient AI sits transparently between your application and your AI provider. Every request passes through with budget checking, caching, attribution tagging, and metadata logging happening in-path, adding about 2–8ms at the median at low concurrency (benchmark).
This page covers the proxy specifically
The proxy is the flagship path — the only one with pre-call budget enforcement — but not the only way to get started. See the Python async wrapper or the CSV/OTel importer for lower-commitment alternatives with their own honest tradeoffs.
The core idea
Cognocient AI sits between your application and the AI provider. One URL change is all it takes:
Every request passes through with a few milliseconds of overhead (benchmark). Cognocient AI records metadata (model, tokens, cost, latency, your attribution headers), enforces any budget rules, and forwards the request. Your application code sees the same response format and streaming behaviour, plus the gateway's own errors (such as a 429 when a budget blocks a call; see the error reference).
What Cognocient AI does on each request
In order, for a chat call (the same checks run on embeddings, images, audio and the Responses API):
- Authenticate and check the Emergency Freeze — a frozen account is refused before anything else runs.
- Rate limits — your key's limit, plus the Free plan's per-minute cap.
- Read attribution headers —
X-Cost-*tags are recorded and stripped before forwarding. - Content guardrails (Base+) — PII and secret redaction, prompt-injection screening.
- Resolve the model (Growth+) — an Auto Router alias or a load-balanced group becomes a real model.
- Cache lookup (Growth+, opt-in per request) — an exact or similar earlier answer is returned without calling the provider.
- Budgets — the call's worst-case cost is reserved atomically against every matching budget and any run budget; over the limit it is blocked, degraded to a cheaper model, or alerted on.
- Velocity and loop breakers — token and dollar per-minute circuit breakers, then the Failure Loop Breaker for repeated tool calls.
- Routing rules (Growth+) and
max_tokensclamping so a stream can't outrun its reservation. - Forward to the provider with your provider key, decrypted in memory; failover retries another provider on an outage (Growth+).
- Settle and log — the reservation is corrected to the real cost, and model, tokens, cost, latency and tags are logged. Never prompt content.
Dashboard sections
The app sidebar has five sections. Dashboard Walkthrough covers every page, with the plan each needs.
| Section | What it's for | Start with |
|---|---|---|
| Observe | Where spend goes: by feature, team, workstream, customer, coding tool, environment | Overview |
| Monitor | What is happening now: every call, anomalies, waste, leaked keys, looping agents | Live Calls |
| Control | Stopping and steering spend: budgets, routing, load balancing, keys, the kill switch | Budgets |
| Intelligence | What to change: recommendations, feature ROI, forecasts, reports | Recommendations |
| Executive | The finance view: spend vs budget, ROI, cost per outcome | Executive View |
The 2-minute morning check
The most effective teams scan five things each morning before standup:
- KPI strip — Waste % up from yesterday? Budget health below 30%? Any KPI in red is worth 30 seconds of investigation.
- Open anomalies — Zero means you're clear. Any anomaly has a root-cause hypothesis already prepared — read it and dismiss or escalate.
- Recommendations — Apply anything with >80% confidence. Each applied recommendation creates a routing rule automatically — no code change.
- Spend trend — Is the 30-day line flat or declining? An unexpected uptick that didn't trigger an anomaly alert is still worth a quick Live Calls drill-down.
- Budget health — Any budget below 20% remaining needs attention before it hits enforcement mode.
What you don't need to do
Cognocient AI does not require you to:
- Change your logging pipeline
- Modify your data warehouse
- Wrap individual SDK calls with metadata
- Migrate historical logs
- Install any new SDKs
The proxy handles all of this automatically. Your only code change is the base_url and api_key. See Quickstart for the 2-minute setup.
Zero-code overhead after setup
Attribution headers are optional add-ons, not requirements. You get cost, model, tokens, and latency tracking on day one — before you add a single header.
What Cognocient AI does not cover
Cognocient AI covers AI API spend — OpenAI, Anthropic, Gemini, and similar providers billed by token. It does not optimize GPU instances, cloud compute, or Kubernetes infrastructure costs. If you need GPU or infrastructure optimization, pair Cognocient AI with your cloud provider's native tools or a cloud FinOps partner — the two work well together since they cover different layers of your stack.
Frequently asked questions
Related articles
Use Cases
Setup guides for Customer Support, Engineering, FinOps, and CFO.
Dashboard Walkthrough
A visual guide to every Cognocient AI dashboard page — what each section shows and where to start.
Python async wrapper
A drop-in wrapper for the OpenAI/Anthropic Python SDKs — live attribution without a proxy, honest tradeoffs included.