Getting Started

How does an LLM gateway for cost control work? Architecture overview

How the Cognocient AI gateway works: the proxy request path, where budgets and guardrails are enforced, what is logged, and what each dashboard section is for.

Cognocient AI sits transparently between your application and your AI provider. Every request passes through with budget checking, caching, attribution tagging, and metadata logging happening in-path, adding about 2–8ms at the median at low concurrency (benchmark).

This page covers the proxy specifically

The proxy is the flagship path — the only one with pre-call budget enforcement — but not the only way to get started. See the Python async wrapper or the CSV/OTel importer for lower-commitment alternatives with their own honest tradeoffs.

The core idea

Cognocient AI sits between your application and the AI provider. One URL change is all it takes:

Your app  →  api.cognocient.com/v1  →  OpenAI / Anthropic / Gemini / ...

Every request passes through with a few milliseconds of overhead (benchmark). Cognocient AI records metadata (model, tokens, cost, latency, your attribution headers), enforces any budget rules, and forwards the request. Your application code sees the same response format and streaming behaviour, plus the gateway's own errors (such as a 429 when a budget blocks a call; see the error reference).

What Cognocient AI does on each request

In order, for a chat call (the same checks run on embeddings, images, audio and the Responses API):

  1. Authenticate and check the Emergency Freeze — a frozen account is refused before anything else runs.
  2. Rate limits — your key's limit, plus the Free plan's per-minute cap.
  3. Read attribution headers — X-Cost-* tags are recorded and stripped before forwarding.
  4. Content guardrails (Base+) — PII and secret redaction, prompt-injection screening.
  5. Resolve the model (Growth+) — an Auto Router alias or a load-balanced group becomes a real model.
  6. Cache lookup (Growth+, opt-in per request) — an exact or similar earlier answer is returned without calling the provider.
  7. Budgets — the call's worst-case cost is reserved atomically against every matching budget and any run budget; over the limit it is blocked, degraded to a cheaper model, or alerted on.
  8. Velocity and loop breakers — token and dollar per-minute circuit breakers, then the Failure Loop Breaker for repeated tool calls.
  9. Routing rules (Growth+) and max_tokens clamping so a stream can't outrun its reservation.
  10. Forward to the provider with your provider key, decrypted in memory; failover retries another provider on an outage (Growth+).
  11. Settle and log — the reservation is corrected to the real cost, and model, tokens, cost, latency and tags are logged. Never prompt content.

Dashboard sections

The app sidebar has five sections. Dashboard Walkthrough covers every page, with the plan each needs.

SectionWhat it's forStart with
ObserveWhere spend goes: by feature, team, workstream, customer, coding tool, environmentOverview
MonitorWhat is happening now: every call, anomalies, waste, leaked keys, looping agentsLive Calls
ControlStopping and steering spend: budgets, routing, load balancing, keys, the kill switchBudgets
IntelligenceWhat to change: recommendations, feature ROI, forecasts, reportsRecommendations
ExecutiveThe finance view: spend vs budget, ROI, cost per outcomeExecutive View

The 2-minute morning check

The most effective teams scan five things each morning before standup:

  1. KPI strip — Waste % up from yesterday? Budget health below 30%? Any KPI in red is worth 30 seconds of investigation.
  2. Open anomalies — Zero means you're clear. Any anomaly has a root-cause hypothesis already prepared — read it and dismiss or escalate.
  3. Recommendations — Apply anything with >80% confidence. Each applied recommendation creates a routing rule automatically — no code change.
  4. Spend trend — Is the 30-day line flat or declining? An unexpected uptick that didn't trigger an anomaly alert is still worth a quick Live Calls drill-down.
  5. Budget health — Any budget below 20% remaining needs attention before it hits enforcement mode.

What you don't need to do

Cognocient AI does not require you to:

  • Change your logging pipeline
  • Modify your data warehouse
  • Wrap individual SDK calls with metadata
  • Migrate historical logs
  • Install any new SDKs

The proxy handles all of this automatically. Your only code change is the base_url and api_key. See Quickstart for the 2-minute setup.

Zero-code overhead after setup

Attribution headers are optional add-ons, not requirements. You get cost, model, tokens, and latency tracking on day one — before you add a single header.

What Cognocient AI does not cover

Cognocient AI covers AI API spend — OpenAI, Anthropic, Gemini, and similar providers billed by token. It does not optimize GPU instances, cloud compute, or Kubernetes infrastructure costs. If you need GPU or infrastructure optimization, pair Cognocient AI with your cloud provider's native tools or a cloud FinOps partner — the two work well together since they cover different layers of your stack.

Frequently asked questions

On this page