How much latency does Cognocient's proxy add?
A real, measured benchmark of Cognocient's added latency — methodology, raw numbers by concurrency level, and an honest caveat about what changes at high concurrency.
Most latency claims in this space are estimates. This one is a real, reproducible benchmark, run at four concurrency levels, with every methodology decision and limitation disclosed below — including the one result that isn't flattering.
Results
200 requests per concurrency level, 5 warm-up requests excluded, run on 2026-07-19.
| Concurrency | Leg | p50 | p95 | p99 |
|---|---|---|---|---|
| 1 | Direct | 34.8ms | 40.0ms | 41.3ms |
| 1 | Proxy | 37.1ms | 42.8ms | 48.8ms |
| 1 | Overhead | +2.3ms | +2.9ms | +7.5ms |
| 10 | Direct | 54.4ms | 65.9ms | 67.7ms |
| 10 | Proxy | 46.6ms | 72.6ms | 90.3ms |
| 10 | Overhead | -7.8ms | +6.8ms | +22.7ms |
| 50 | Direct | 91.8ms | 116.6ms | 120.2ms |
| 50 | Proxy | 223.6ms | 512.0ms | 807.8ms |
| 50 | Overhead | +131.8ms | +395.4ms | +687.6ms |
| 100 | Direct | 157.4ms | 191.2ms | 564.5ms |
| 100 | Proxy | 469.3ms | 528.1ms | 882.4ms |
| 100 | Overhead | +311.9ms | +336.9ms | +317.9ms |
The high-concurrency numbers are worse than the headline figure, and we're not hiding that
At 50-100 simultaneous requests to a single instance, added latency is hundreds of milliseconds, not single-digit. This is disclosed in full below rather than only reporting the flattering low-concurrency number.
What "overhead" means here, and why it isn't flat
At concurrency 1-10 — one or a few requests in flight against a single instance at a time — Cognocient adds roughly 2-8ms at the median: authentication, attribution tag parsing, an atomic Redis budget reservation, a routing-rule lookup, and response construction. The concurrency=10 result showing a negative p50 (proxy measuring faster than direct) is noise at that sample size, not a real effect — treat anything within about ±10ms at this concurrency as noise, not signal.
At concurrency 50-100, overhead grows to the hundreds of milliseconds. The honest reason: Cognocient's backend currently runs as a single uvicorn worker process (verified from railway.json's actual deploy command — no --workers flag is set). Python's asyncio model runs one request's CPU-bound work — parsing, cost calculation, JSON encode/decode — at a time on a single process; at high concurrency to one instance, that work queues rather than running in parallel, and total latency grows faster than linearly. This is a genuine, current characteristic of today's deployment, not a benchmark artifact invented to pad the number — it directly informs a real capacity-planning takeaway: a single Cognocient instance is not yet tuned for very high single-instance concurrency, and workloads with many simultaneous subagent calls in flight against one instance should factor that in until multi-worker/horizontal scaling is in place.
Methodology
- What's compared: a direct HTTP call to a mock AI provider (fixed OpenAI-shaped JSON response, small simulated think-time) versus the identical call routed through the real Cognocient proxy code path to that same mock provider. Both legs are real loopback HTTP round trips — not in-process function calls — so the difference isolates what crossing Cognocient's layer costs, independent of any specific real provider's own variable network latency.
- What's real: the actual
app/proxy.pyrequest-handling code runs unmodified — auth cache lookup, attribution tag extraction, the real atomic Redis Lua budget reservation (atomic_reserve_budget, exercised against a real "block"-mode budget so the reservation genuinely executes rather than being skipped), cost estimation, and response construction. - What's faked, and why: Postgres is never connected — auth and provider-key lookups are pre-seeded into the same in-memory caches a real deployment warms after its first request, so this measures steady-state (cache-hit) latency, not a cold first request. Redis is
fakeredis(in-memory, verified to support the exact LuaEVALscript this codebase uses) rather than a real Upstash instance — this means the measured number does not include Upstash's own network round trip, which is additive on top of this in production (typically low single-digit milliseconds same-region). This is the single biggest caveat on the low-concurrency number and is repeated here rather than left as fine print. Routing-rule lookup and the velocity circuit breaker are stubbed to their default (no rules configured / not tripped) — optional features a given customer may not have enabled, not the path every call takes. - Sample size: 200 requests per concurrency level per leg, 5 warm-up requests excluded per leg.
- Reproduce it yourself:
backend/scripts/benchmark_proxy_overhead.pyin the repository. No real database, Redis, or provider credentials required to run it.
What this number is not
This is not a claim about total system throughput under sustained heavy concurrent load — that depends on worker/instance count, which is a scaling configuration question, not a per-request overhead question. It is a measurement of what one request pays, in addition to reaching an AI provider directly, to get spend attribution, pre-call budget enforcement, and the other guardrails documented in Guardrails & Rate Limits — at the concurrency level your traffic actually runs at against a single instance.
Frequently asked questions
Related articles