What a Governance Proxy Actually Costs You in Latency

I. The question every vendor dodges

Any layer inserted between your application and an AI provider has to answer one question honestly before anything else it does matters: what does it cost you in latency. The typical answer in this category is a qualitative claim — “negligible,” “sub-millisecond,” “imperceptible” — offered without a methodology, a sample size, or a concurrency level attached to it. That kind of claim is unfalsifiable, which is exactly why it's common. This report takes the opposite approach: a real benchmark, run against the actual request-handling code that ships in production, with every methodology decision listed and the result reported even where it isn't flattering.

II. Methodology

The benchmark compares two legs, both real loopback HTTP round trips rather than in-process function calls: a direct HTTP call to a mock AI provider (a fixed OpenAI-shaped JSON response with a small simulated think-time), and the identical call routed through Cognocient's real proxy code path to that same mock provider. Isolating the mock provider on both legs means the difference measured is specifically what crossing Cognocient's layer costs — independent of any real provider's own variable network latency, which would otherwise swamp the signal.

What's real in that proxy leg: the actual request-handling code runs unmodified — auth cache lookup, attribution tag extraction, an atomic Redis budget reservation exercised against a real block-mode budget so the reservation genuinely executes rather than being skipped, cost estimation, and response construction.

What's faked, and why it's disclosed rather than buried: Postgres is never connected — auth and provider-key lookups are pre-seeded into the same in-memory caches a real deployment warms after its first request, so this measures steady-state, cache-hit latency, not a cold first request. Redis is an in-memory fake rather than a real hosted instance, which means the measured number does not include that instance's own network round trip — additive on top of this in production, typically low single-digit milliseconds same-region. That is the single biggest caveat on the low-concurrency number, and it is repeated here rather than left as fine print. Two hundred requests were run per concurrency level per leg, with five warm-up requests excluded from each. The script that produced these numbers ships in the repository and requires no real database, Redis, or provider credentials to run — anyone can reproduce it.

III. The results at normal concurrency

At one and ten simultaneous requests against a single instance — the concurrency band most individual production workloads actually run at — Cognocient adds roughly 2-8ms at the median: authentication, attribution tag parsing, the atomic Redis budget reservation, a routing-rule lookup, and response construction. At the tail, p99 overhead at concurrency 1 was +7.5ms.

The concurrency-10 result showing a negative p50 — the proxy leg measuring faster than the direct leg — is noise at this sample size, not a real effect. Anything within roughly ±10ms at this concurrency should be read as noise, not signal.

IV. The results at high concurrency, and the honest caveat

At fifty and one hundred simultaneous requests against a single instance, overhead is not single-digit milliseconds anymore — it is hundreds. At concurrency 50, median overhead was +131.8ms (p95 +395.4ms, p99 +687.6ms). At concurrency 100, median overhead was +311.9ms. This is disclosed here in full, not only reported as the flattering low-concurrency headline number.

The honest reason, not a hedge: Cognocient's backend currently runs as a single uvicorn worker process — verified from the actual deploy command, which sets no --workers flag. Python's asyncio model runs one request's CPU-bound work — parsing, cost calculation, JSON encode/decode — at a time on that single process. At high concurrency against one instance, that work queues rather than running in parallel, and total latency grows faster than linearly.

This is a genuine, current characteristic of today's deployment, not a benchmark artifact invented to pad a number. It carries a real capacity-planning takeaway: a single Cognocient instance is not yet tuned for very high single-instance concurrency, and workloads that fan out many simultaneous subagent calls against one instance at once should factor that in until multi-worker or horizontal scaling is in place. It does not describe what a single call pays — that is the low-concurrency number above — but it is the honest picture of what happens under sustained high concurrent load today, and it is reported here rather than left for a customer to discover in production.

V. What bounds the downside when Cognocient itself is the problem

A per-request overhead number only tells half the reliability story. The other half is what happens when the budget-check layer isn't just slow but genuinely unreachable. The verified behavior, read directly from the request-handling code rather than described aspirationally: on a Redis error, the check logs a warning and returns “allowed” rather than raising and blocking the request. Calls pass through to your AI provider unaffected, within a fraction of a second — not hung, not queued behind a retry.

What's actually lost during that window is narrow: attribution data for calls made during the outage may be incomplete or delayed, and a block-mode budget will not newly block calls it hasn't already confirmed are over limit. One exception is preserved even during an outage — if a block-mode budget's last known spend was already over its limit before the outage started, Cognocient keeps blocking that budget's calls using that last-known value. Fail-open means not newly blocking on missing data, not ignoring data it already has.

This default is deliberately per-budget rather than account-wide, because not every workload has the same risk tolerance: a budget can be switched from fail-open to fail-closed individually, trading availability for certainty on the specific traffic where an unmetered call would be worse than a blocked one — a workload touching financial transactions, for instance — without forcing that same trade-off onto every other budget on the account.

VI. What bounds the downside when your provider is the problem

The fail-open guarantee covers Cognocient's own backend. A separate, independent failure mode is the AI provider itself timing out, rate-limiting, or returning a 5xx — and for that, Cognocient offers opt-in Provider Failover: on a timeout, an HTTP 429, or a 5xx from the primary provider, retry the same call once against a different, already-connected provider instead of just failing. A 401 never triggers it — that's a misconfigured key, not an outage, and failing over would hide the real fix rather than surface it.

It is off by default, because some teams have data-residency or vendor constraints that make a silent cross-provider retry unacceptable — a call that was supposed to stay within one provider's infrastructure could otherwise land somewhere else without anyone noticing. It only ever activates with at least two providers connected, retries exactly once, and if the fallback also fails, you get the original error back — never a cascading retry chain. Response headers on a failed-over call name the original and fallback provider and model, so your application can tell when a response came from somewhere other than where it asked.

VII. The honest latency budget

For the concurrency band most individual production workloads run at against a single instance, adopting Cognocient costs single-digit milliseconds at the median to gain spend attribution, pre-call budget enforcement, and the other guardrails documented across this site. At sustained high concurrency against one instance, that cost is materially higher today — hundreds of milliseconds — for a disclosed, verified reason: a single-worker deployment that has not yet been tuned for very high single-instance concurrency. And in the two failure modes that matter most in production — Cognocient's own backend going down, or your provider going down — the defaults are fail-open and opt-in single-hop failover, respectively, so neither failure mode becomes an application outage by default. A latency number without those two guarantees attached would be an incomplete answer to the question this report opened with.