How do I rate-limit LLM calls and stop runaway agents with velocity limits?
Pre-call budget enforcement, max_tokens clamping, hierarchical budget chains, and velocity circuit breakers. Stop runaway spend before it reaches your provider bill.
Guardrails add a second enforcement layer on top of budgets — per-minute and per-hour call velocity limits that catch runaway usage before it drains your monthly budget. Combined with max_tokens clamping, they prevent overspend structurally rather than alerting after the fact.
Most cost tools read billing exports that are 24 hours late. By the time an alert fires, a weekend agent loop has already run. Cognocient AI checks and reserves budget before the call reaches the provider — making it impossible to exceed your limits, not just possible to be notified after the fact.
The check is a single atomic Redis operation: reserve the estimated cost, allow if within budget, reject if over. 50 concurrent agent calls against the same $0.10 budget will result in exactly 10 succeeding — every time.
Budget enforcement modes
Three modes, same atomic check:
| Mode | What happens | Best for |
|---|---|---|
| Block | HTTP 429 returned. Provider never sees the request. No charge possible. | Experiments, dev/test, agent sandboxes |
| Degrade | Request auto-switched to a cheaper model and continues. | Production services with SLA requirements |
| Alert | Request proceeds normally. An alert is sent at each threshold: email (every plan) and Slack (Base+). | Baseline measurement before committing to limits |
How budget enforcement works
When a request arrives:
- Cognocient AI estimates the cost from prompt tokens (inferred from message length) and
max_tokens - An atomic Lua script reserves that amount in Redis — or rejects if the reservation would exceed the limit
- The call proceeds (or returns 429)
- After the call, the reservation is reconciled to the actual cost
Because reservation and check happen in a single Redis operation, there is no race condition. Concurrent calls cannot collectively exceed the budget.
HTTP 429 response:
max_tokens clamping
When a budget reservation is made in block mode, Cognocient AI also clamps the forwarded max_tokens to the value the remaining budget can actually afford.
Why this matters: without clamping, a caller requests max_tokens=4096. Budget has $0.10 remaining. The proxy reserves $0.10 (enough for ~167 tokens at gpt-4o rates) but forwards max_tokens=4096. The stream runs 4096 tokens. Actual spend: $0.40. Reservation: $0.10. Budget overshot 4x — the reservation was meaningless.
With clamping, the forwarded request gets max_tokens=167. The model physically cannot generate beyond what was reserved.
Response headers when clamping occurs:
Your application can read these headers to understand when clamping occurred. The clamped value is always at least 16 (minimum meaningful response).
Clamping is applied to chat completion calls only — embeddings have no max_tokens concept.
Hierarchical budget enforcement
Budgets form a chain: run → feature → department → org. Every matching level is checked atomically before a call proceeds. If any level is exhausted, the call is blocked — even if child budgets still have room.
Example: A per-run limit of $0.50 looks fine individually. 50 runs × $0.49 = $24.50 against a $20 department budget. Without hierarchy, every individual run passes while the department budget is blown. With hierarchy, the department ceiling wins.
See Budget Enforcement for the full hierarchy explanation and scope labeling.
Velocity circuit breaker
An independent limit on tokens per minute (TPM), separate from budget enforcement. Activates automatically on runaway usage spikes — a single agent loop generating tokens 10x faster than the normal baseline is blocked, even if the budget has room.
The circuit breaker uses a sliding 60-second window. Auto-detection trips at a fixed 10× your key's own rolling baseline — this multiplier isn't user-configurable. If a key needs a tighter or more predictable ceiling than the auto-baseline gives it, set a manual tokens-per-minute cap instead, which overrides auto-detection entirely for that key.
A key with no baseline yet (its first three days) has fixed ceilings instead: 500,000 tokens and $25 per minute. The token ceiling is set for real agent traffic, since a single coding-agent turn can send a 25,000-token prompt that is mostly cheap cache reads; the dollar ceiling is the money guard.
Both breakers run on chat completions, embeddings and the Anthropic /v1/messages endpoint (Claude Code); image generation and audio are checked on dollars per minute.
Set a manual limit in Settings → Safety & Limits → Velocity Limit. An alert is sent on trip to every channel with Anomalies & security switched on (email on every plan, Slack on Base+) — see Alerts.
Dollar-denominated velocity
A flood of image or video generation calls has a tiny token footprint — a short prompt, no completion tokens — but a large dollar one, since those calls are billed per-image or per-video, not per-token. The TPM breaker above can't see that shape of spike at all. This is why the velocity check runs twice: once on tokens/minute, once on dollars/minute.
The same 10×-baseline auto-detection as the TPM breaker, keyed by each call's worst-case estimated cost (from max_tokens) instead of token count, and counted before the call is forwarded. The window is a true sliding 60 seconds, built from 1-second buckets, so a burst that straddles a clock-minute boundary counts in full. Either check tripping is sufficient to block the call. Set a manual $/min cap alongside the TPM one in Settings → Safety & Limits → Velocity Limit. The two limits are independent, so you can set one, both, or neither.
The response carries Retry-After (seconds until the fuse releases) and X-Cognocient-Breaker: spend-fuse. reason says which check tripped: dollar_velocity_limit_exceeded (your $/min limit), dollar_velocity_spike_detected (10× baseline), dollar_cold_start_limit ($25/min with no baseline yet), burst_dollar_velocity_limit_exceeded (burst limit), run_dollar_velocity_limit_exceeded (per-run limit, with run_id), or one of the two tier_ reasons below.
A dollar-velocity trip is eligible for automatic freeze if you've opted in — unlike the TPM breaker, this is exactly the pattern behind the incident that motivated Emergency Freeze in the first place, so it's treated as freeze-eligible rather than alert-only.
Per-tier dollar windows
One overall $/min number has a blind spot: a steady flood of cheap calls sets a high baseline, and an expensive model can then multiply its own spend without moving the total anywhere near 10×. So every chat, /v1/messages and embeddings call is also counted in a window for its model's price tier, based on the model's output price per 1M tokens:
| Tier | Output price per 1M tokens | Typical models |
|---|---|---|
| Economy | under $3 | mini, nano, flash class |
| Standard | $3 to under $15 | GPT-4o, o3, Gemini Pro, Haiku 4.5 class; unknown models land here |
| Premium | $15 and up | Sonnet, Opus class |
Each tier trips at 10× its own 7-day baseline (once it has 3 days of traffic) or at a manual $/min limit you set per tier in Settings → Safety & Limits → Velocity Limit. A tier trip blocks only calls to that tier; other tiers keep running. The overall checks above still run first, and there is no per-tier cold-start ceiling (the overall $25/min covers a new key). The 429 has the same type: "dollar_velocity_limit_exceeded", plus "model_tier" naming the tier. Image generation and audio stay on the overall window only.
Spend fuse, burst limit and early warning
Three settings in Settings → Safety & Limits → Velocity Limit → Spend fuse control what happens around a dollar trip. They apply to every active proxy key on the account.
| Setting | Default | What it does |
|---|---|---|
| Fuse duration | 5 minutes | When any dollar check trips, the fuse latches: every call on the key is refused for this long, even if the rate drops. A tier trip latches for that tier only. 0 turns the fuse off, so calls are blocked only while the window is over the limit. |
| Burst limit | none | Dollars allowed within the burst window. Catches a loop that does its damage in seconds, such as $20 in 10 seconds. |
| Burst window | 10 seconds | Length of the burst window, 5 to 59 seconds. |
| Per-run limit | none | Dollars per minute each run may spend, counted separately per run. See Per-run limit below. |
| Warn at % of limit | 50% | One alert, before anything is blocked, when a window reaches this share of the point where it trips. At most one per key every 10 minutes. 0 turns warnings off. |
Why the fuse exists. Without it, a looping agent is refused only until the window drops back under the limit, then runs again until it trips again, all day. With the fuse it stays stopped until the fuse runs out or you reset it. Calls refused while the fuse holds are not counted as spend, and only the call that tripped it sends an alert (and triggers automatic freeze, if you've opted in).
Resetting. Once you've stopped whatever was looping, click Reset fuse in the same section (or POST /api/velocity/fuse/reset, which needs the manage_routing permission). This clears the fuse and the spend windows on every key of the account. If the loop is still running, it trips again.
What it can't stop. Every check runs before a call is forwarded, using that call's worst-case cost. Calls already in flight when the fuse trips still complete and are billed by the provider.
Per-run limit
When several agents share one key, a key-level trip stops all of them. The per-run limit gives each agent run its own sliding 60-second window instead. A run is identified by the X-Cost-Run-ID header (the same one run budgets use), or by X-Cost-Session when a call has no run id. Calls with neither are only covered by the key-level checks.
A run that spends more than the limit in any 60 seconds latches a fuse for that run only: its calls are refused for the fuse duration, the other runs on the key keep going, and the 429 has reason: "run_dollar_velocity_limit_exceeded" and run_id. A per-run trip alerts like any other trip but never triggers automatic freeze, since stopping one agent is the point. Settings lists the stopped runs with a Reset button for each; the API takes POST /api/velocity/fuse/reset with {"run_id": "..."}.
The key-level checks still apply on top: if all runs together go over the key's $/min limit, the whole key is fused as before. The per-run limit is a rate, not a total; to cap what one run may spend in all, use a run budget.
Concurrency limits
Rate limits and the spend fuse count per minute, so an agent that fires 200 parallel calls in the same second gets through before either reacts. A concurrency limit caps how many calls can be in flight at once:
- Per key: across everything on the key.
- Per run: within one run (
X-Cost-Run-ID, elseX-Cost-Session; Claude Code's session id counts), so one agent can't take every slot.
Set them in Settings → Safety & Limits → Velocity Limit → Concurrency limits, or with POST /api/velocity/configure and {"max_concurrent": 50, "run_max_concurrent": 8} (null clears a limit). Both are off by default. They apply to every proxy endpoint, on every plan.
The call over the limit is refused at once, before any budget is reserved:
The status is 429 with Retry-After: 1. A slot is freed as soon as a call's response has finished, streams included. If a server stops mid-call, its slot expires after 15 minutes. If Redis can't be reached, calls are not limited.
Failure Loop Breaker — repeated tool calls & consecutive errors
The velocity circuit breaker above catches loops by how fast they run. But a stuck agent doesn't have to be fast to be expensive — it can sit well within your budget and TPM baseline while quietly repeating the same mistake for hours. The Failure Loop Breaker is a second, independent signal that catches that case: it looks at what the agent is actually doing, not how much it's spending or how quickly.
A public agent-failure trace study found that 58% of tokens burned in failed runs were spent after the first clear warning sign appeared — a tool error, or a repeated identical call. The model had enough evidence to stop and kept going. Budget and velocity limits don't catch this, because a failing run can be nowhere near either limit while still burning money in a useless loop.
The failure signals
- Repeated identical tool call — the same tool called with the same arguments, 3 times in a row within a session (or that tool's own threshold, see below). Works for streaming calls too: for those it's read from the conversation you send (the assistant's latest tool call), so it trips one call later than for non-streaming ones. Example: an agent calling
search_docs({"query": "refund policy"})three turns straight because it isn't incorporating the result into its next decision. A retry that happens because of a transient failure (see below) doesn't count against this threshold — the same call showing up again after a rate limit or timeout is expected, not evidence of being stuck. - No progress — read from the conversation you send, which holds every tool call and the result it got back. Trips when the same call (same tool, same arguments) gets the same result 3 times in a row (or that tool's own threshold), which also covers the same write failing with the same error. It also trips on ping-pong: two different calls alternating A, B, A, B while both keep returning the same results (for example: write fails, read the state, write fails again). A changed result resets it, so a retry after someone fixes the underlying problem isn't blocked. Argument formatting doesn't matter: JSON arguments are compared with keys sorted.
- Consecutive errors — the model is fed an error result from a tool, or this proxy's own call to the provider itself fails (401/429/5xx/timeout), several times in a row. How many times "counts" depends on why it's failing — see below.
Per-tool thresholds. Some tools are meant to repeat: a job-status poll legitimately returns "still running" several times. On Monitor → Loop Breaker, give any tool name its own threshold from 2 to 50. Raise it for polling tools, or lower it to 2 for tools where a repeat is never expected (sending an email, creating a record). Tools you don't list keep 3. Each detected loop on the dashboard shows the tool that tripped it, with a shortcut to set its threshold. Per-tool thresholds change the repeated-call and same-result signals; the consecutive-error thresholds below are the same for every tool.
All signals run on chat completions, the Responses API and the Anthropic Messages endpoint (/v1/messages, where Claude Code's session id counts as the session and only tool results marked is_error count as failures). They are evaluated per X-Cost-Session — the same header Debugging Runaway Agent Loops already tells you to use for spotting loops in Live Calls.
Retryable vs. deterministic errors
Not every failure means the same thing. A timeout, rate limit, or 5xx from the provider is a retryable failure — retrying with backoff is the correct, expected response, and a run legitimately working through a few of these isn't stuck. A validation error, bad argument, or 4xx caused by the request itself is a deterministic failure — it will fail identically no matter how many times it's retried, so repeating it even twice is a strong signal the run needs to stop. Failures that don't clearly indicate either are classified unknown and treated conservatively, with a threshold between the other two.
| Classification | Consecutive threshold | Why |
|---|---|---|
| Deterministic | 2 | Fails identically every time — no point letting it repeat |
| Unknown | 3 | Can't tell if a retry would help — conservative middle ground |
| Retryable | 5 | Expected backoff/retry behavior gets real room before this counts as stuck |
Enforcement modes: off, alert, kill
| Mode | What happens |
|---|---|
| Off | Detection disabled entirely. |
| Alert | The run continues; a failure_loop_events entry is logged so you can review the pattern before deciding to enforce. This is the default for every account. |
| Kill | Once a signal crosses its threshold, the next call returns HTTP 429 with type: "failure_loop_detected" instead of reaching the provider. |
Start in alert mode and review the Failure Loop Breaker dashboard for a few days before switching to kill — this confirms the detection isn't catching a legitimate pattern (like a deliberate polling loop) specific to your traffic before it starts stopping calls.
HTTP 429 response (kill mode) — the message differs depending on which signal tripped, since a deterministic failure and a retryable one call for different next steps:
reason is one of: repeated_identical_tool_call, repeated_call_same_result, tool_ping_pong, consecutive_deterministic_errors, consecutive_unclassified_errors, or consecutive_retryable_errors_exceeded_backoff_budget.
Tool-call detection currently only works for non-streaming responses. Anthropic tool-calling now works correctly for non-streaming requests — streaming Anthropic requests with tools are not yet supported, so the tools field isn't forwarded on those calls (this avoids silently dropping a tool call mid-stream) and the repeated-call signal won't fire there. Consecutive-error detection is unaffected either way, since it reads the client's own request history rather than the provider response.
Configure enforcement mode from Monitor → Loop Breaker.
Provider spend-cap circuit — one shared breaker per key
A provider refusing a call because a key's spend limit or credit balance is used up is a different failure from a rate limit. A rate limit clears in seconds, so Cognocient AI passes the provider's 429 and its Retry-After straight through. A spend cap doesn't clear until the billing period resets or someone raises the limit, so retrying it only burns time, and every script sharing that key fails, retries and alerts in the same minute.
When a provider answers with a billing error (OpenAI's insufficient_quota, Anthropic's "credit balance is too low" or "API usage limits", any HTTP 402), Cognocient AI:
- Holds calls on that key for 10 minutes. They get an immediate HTTP 402 with
type: "provider_quota_exhausted"instead of reaching the provider. SDKs don't retry a 402 the way they retry a 429, so every caller sharing the key stops at once. The circuit is per key (and per pooled key), not per script, because the key is what's shared. - Lets one call through every 10 minutes to check. Once you raise the limit or add credits, the next check succeeds and calls carry on. Nothing to reset.
- Routes a key pool around the exhausted key for the same 10 minutes, rather than retrying it.
- Sends one alert per key per day with your budget alerts (email by default, plus Slack if connected), with the provider's own message.
It covers chat completions, the Responses API, /v1/messages (Claude Code), embeddings, image generation, audio and rerank; all of them share the circuit for a given key. Rate limits never open this circuit. If provider failover is on, calls aren't held: they go through so the billing error can fail over to your other provider. Your provider's own spend cap is still worth setting as the last backstop, because it is the only limit that holds if your own code is what's broken.
Retry-tree / handoff depth limit
parent_run_id and X-Cost-Agent-Handoff (see MCP / A2A Attribution) let a run track which subagent handed off to which — but nothing capped how deep that chain could go until now. A pathological chain of subagent handoffs can recurse indefinitely while each individual hop stays well within budget, since per-call cost alone doesn't reveal a runaway chain.
Every call that carries X-Cost-Run-Id now has its position in the handoff tree checked against a max_retry_depth ceiling — independent of and in addition to budget. A call at depth 10 with a limit of 10 is blocked even if the run has spent $0.
Depth 0 is a run's first call. Each subagent handoff to a new run_id that declares the prior run as its X-Cost-Parent-Run-Id adds one level. The default ceiling is 10, configurable per-budget (so a workload that's expected to chain deeper, or shouldn't chain at all, can set its own limit) via the Max Retry/Handoff Depth field when creating a budget, or per-customer as a fallback for calls with no matching budget.
Idempotency keys — client-side retry protection
Distinct from the Failure Loop Breaker above: that system governs Cognocient AI's own decision about whether a failing pattern should keep running. This is about your client sending what is logically the same request twice — your own retry after a timeout, a queue redelivery, a double-click — which would otherwise be reserved and billed as two separate calls.
Send an X-Idempotency-Key header with any value unique to that logical request. If Cognocient AI has already processed that exact key for your account within the last 24 hours, the original response is returned as-is — no new budget reservation, no second call to your provider — with an x-cog-idempotent-replay: true response header so you can tell a replay from a fresh call.
Replay isn't supported for streaming requests — a replayed JSON body can't be reassembled into an SSE stream, the same constraint Cognocient AI's response caches already have.
In-flight protection covers streaming and non-streaming calls alike: a second request with the same key never opens a second budget reservation while the first is running. That's the case where a client retries after its own timeout while the original call is still going.
- Non-streaming retries wait for the original. The retry is held (up to 30 seconds) until the original finishes, then gets the original's result with
x-cog-idempotent-replay: trueandx-cog-idempotent-waited: true. No second reservation, no second provider call, and nothing for your client to retry. - If the original ends without a result (it timed out or failed), the held retry runs as a new call.
- If the original is still running after 30 seconds, or the call is streaming (a stream can't be replayed), the retry gets
409(type: idempotent_request_in_flight, withRetry-After).
Agentic enforcement — write vs. read
An exhausted Block budget returns 429 for every call, whatever its X-Cost-Workload tag. A tag your client sets can't lift a limit.
The tag matters in one place: a Block budget's optional fallback allowance, a capped amount you choose in advance for continuing on a cheaper model after the limit.
| Workload | Block budget exhausted, no allowance | Block budget exhausted, allowance left |
|---|---|---|
agentic-write | Hard stop (429) | Hard stop (429). Write ops change external state, and a weaker model may choose the wrong action. |
agentic-read | Hard stop (429) | Runs on the cheaper model until the allowance is spent, then 429 |
| (not set) | Hard stop (429) | Inferred from tool names: a tool with a write verb (create, update, delete, send, write...) counts as write; anything else uses the allowance |
Defense in depth — orchestration-layer check
For multi-step agent workflows, add a second protection layer by querying remaining budget before making the next tool call. This lets your agent wrap up gracefully instead of being hard-stopped mid-execution.
See Budget Enforcement → Defense in depth for the full LangGraph and CrewAI examples.
Frequently asked questions
Related articles