How do I set velocity limits and guardrails on AI calls?

Guardrails add a second enforcement layer on top of budgets — per-minute and per-hour call velocity limits that catch runaway usage before it drains your monthly budget. Combined with max_tokens clamping, they prevent overspend structurally rather than alerting after the fact.

Budget enforcement modes

Three modes, same atomic check:

How budget enforcement works

When a request arrives:

1. Cognocient estimates the cost from prompt tokens (inferred from message length) and max_tokens
2. An atomic Lua script reserves that amount in Redis — or rejects if the reservation would exceed the limit
3. The call proceeds (or returns 429)
4. After the call, the reservation is reconciled to the actual cost

Because reservation and check happen in a single Redis operation, there is no race condition. Concurrent calls cannot collectively exceed the budget.

HTTP 429 response:

{
  "error": {
    "message": "Budget limit exceeded: customer-success",
    "type": "budget_error",
    "scope": "department"
  }
}

max_tokens clamping

When a budget reservation is made in block mode, Cognocient also clamps the forwarded max_tokens to the value the remaining budget can actually afford.

Without clamping (broken):
  Caller sends:  max_tokens=4096
  Budget has:    $0.10 remaining (~167 tokens worth)
  Proxy reserves $0.10 but forwards max_tokens=4096
  Model generates 4096 tokens → ACTUAL SPEND: $0.40
  Budget overshot 4× — reservation was meaningless ✗

With clamping (Cognocient):
  Caller sends:  max_tokens=4096
  Budget has:    $0.10 remaining
  Proxy computes: $0.10 ÷ $0.0006/token = 167 tokens
  Proxy forwards: max_tokens=167
  Model physically cannot exceed reservation ✓
  Response headers: x-cog-max-tokens-clamped: 167
                    x-cog-max-tokens-original: 4096

Why this matters: without clamping, a caller requests max_tokens=4096. Budget has $0.10 remaining. The proxy reserves $0.10 (enough for ~167 tokens at gpt-4o rates) but forwards max_tokens=4096. The stream runs 4096 tokens. Actual spend: $0.40. Reservation: $0.10. Budget overshot 4x — the reservation was meaningless.

With clamping, the forwarded request gets max_tokens=167. The model physically cannot generate beyond what was reserved.

Response headers when clamping occurs:

x-cog-max-tokens-clamped: 167
x-cog-max-tokens-original: 4096

Your application can read these headers to understand when clamping occurred. The clamped value is always at least 16 (minimum meaningful response).

Clamping is applied to chat completion calls only — embeddings have no max_tokens concept.

Hierarchical budget enforcement

Budgets form a chain: run → feature → department → org. Every matching level is checked atomically before a call proceeds. If any level is exhausted, the call is blocked — even if child budgets still have room.

Example: A per-run limit of $0.50 looks fine individually. 50 runs × $0.49 = $24.50 against a $20 department budget. Without hierarchy, every individual run passes while the department budget is blown. With hierarchy, the department ceiling wins.

See Budget Enforcement for the full hierarchy explanation and scope labeling.

Velocity circuit breaker

An independent limit on tokens per minute (TPM), separate from budget enforcement. Activates automatically on runaway usage spikes — a single agent loop generating tokens 10x faster than the normal baseline is blocked, even if the budget has room.

The circuit breaker uses a sliding 60-second window. Auto-detection trips at a fixed 10× your key's own rolling baseline — this multiplier isn't user-configurable. If a key needs a tighter or more predictable ceiling than the auto-baseline gives it, set a manual tokens-per-minute cap instead, which overrides auto-detection entirely for that key.

Set a manual limit in Settings → Proxy Keys → Velocity Limit. A Slack alert fires on trip if you've routed Budget Alerts to a channel — see Slack Alerts.

Failure Loop Breaker — repeated tool calls & consecutive errors

The velocity circuit breaker above catches loops by how fast they run. But a stuck agent doesn't have to be fast to be expensive — it can sit well within your budget and TPM baseline while quietly repeating the same mistake for hours. The Failure Loop Breaker is a second, independent signal that catches that case: it looks at what the agent is actually doing, not how much it's spending or how quickly.

The two failure signals

• Repeated identical tool call — the same tool called with the same arguments, 3 times in a row within a session. Example: an agent calling search_docs({"query": "refund policy"}) three turns straight because it isn't incorporating the result into its next decision. A retry that happens because of a transient failure (see below) doesn't count against this threshold — the same call showing up again after a rate limit or timeout is expected, not evidence of being stuck.
• Consecutive errors — the model is fed an error result from a tool, or this proxy's own call to the provider itself fails (401/429/5xx/timeout), several times in a row. How many times "counts" depends on why it's failing — see below.

Both signals are evaluated per X-Cost-Session — the same header Debugging Runaway Agent Loops already tells you to use for spotting loops in Live Calls.

Retryable vs. deterministic errors

Not every failure means the same thing. A timeout, rate limit, or 5xx from the provider is a retryable failure — retrying with backoff is the correct, expected response, and a run legitimately working through a few of these isn't stuck. A validation error, bad argument, or 4xx caused by the request itself is a deterministic failure — it will fail identically no matter how many times it's retried, so repeating it even twice is a strong signal the run needs to stop. Failures that don't clearly indicate either are classified unknown and treated conservatively, with a threshold between the other two.

Enforcement modes: off, alert, kill

Start in alert mode and review the Failure Loop Breaker dashboard for a few days before switching to kill — this confirms the detection isn't catching a legitimate pattern (like a deliberate polling loop) specific to your traffic before it starts stopping calls.

HTTP 429 response (kill mode) — the message differs depending on which signal tripped, since a deterministic failure and a retryable one call for different next steps:

{
  "error": {
    "message": "This run was stopped after 2 consecutive failures that appear to be caused by the request itself (invalid input, bad arguments, or similar), not a temporary provider issue. Retrying this call again is unlikely to succeed without changing the request.",
    "type": "failure_loop_detected",
    "reason": "consecutive_deterministic_errors"
  }
}

reason is one of: repeated_identical_tool_call, consecutive_deterministic_errors, consecutive_unclassified_errors, or consecutive_retryable_errors_exceeded_backoff_budget.

Configure enforcement mode from Dashboard → Loop Breaker.

Retry-tree / handoff depth limit

parent_run_id and X-Cost-Agent-Handoff (see MCP / A2A Attribution) let a run track which subagent handed off to which — but nothing capped how deep that chain could go until now. A pathological chain of subagent handoffs can recurse indefinitely while each individual hop stays well within budget, since per-call cost alone doesn't reveal a runaway chain.

Every call that carries X-Cost-Run-Id now has its position in the handoff tree checked against a max_retry_depth ceiling — independent of and in addition to budget. A call at depth 10 with a limit of 10 is blocked even if the run has spent $0.

{
  "error": "max_retry_depth_exceeded",
  "depth": 10,
  "max_depth": 10,
  "message": "This call would exceed the maximum retry/handoff depth of 10 for this run. This is a safety limit independent of budget, intended to catch runaway recursive handoff chains."
}

Depth 0 is a run's first call. Each subagent handoff to a new run_id that declares the prior run as its X-Cost-Parent-Run-Id adds one level. The default ceiling is 10, configurable per-budget (so a workload that's expected to chain deeper, or shouldn't chain at all, can set its own limit) via the Max Retry/Handoff Depth field when creating a budget, or per-customer as a fallback for calls with no matching budget.

Idempotency keys — client-side retry protection

Distinct from the Failure Loop Breaker above: that system governs Cognocient's own decision about whether a failing pattern should keep running. This is about your client sending what is logically the same request twice — your own retry after a timeout, a queue redelivery, a double-click — which would otherwise be reserved and billed as two separate calls.

Send an X-Idempotency-Key header with any value unique to that logical request. If Cognocient has already processed that exact key for your account within the last 24 hours, the original response is returned as-is — no new budget reservation, no second call to your provider — with an x-cog-idempotent-replay: true response header so you can tell a replay from a fresh call.

X-Idempotency-Key: order-48291-confirmation-email

Not supported for streaming requests — a replayed JSON body can't be reassembled into an SSE stream, the same constraint Cognocient's response caches already have.

Agentic enforcement — write vs. read

The X-Cost-Workload header controls what happens when a budget limit is reached during an agentic workflow:

Defense in depth — orchestration-layer check

For multi-step agent workflows, add a second protection layer by querying remaining budget before making the next tool call. This lets your agent wrap up gracefully instead of being hard-stopped mid-execution.

# Before each agent step — check remaining budget
import httpx
 
async def check_budget(feature: str, run_id: str) -> bool:
    resp = await httpx.get(
        "https://api.cognocient.com/api/budgets/status",
        headers={"Authorization": f"Bearer {COGNOCIENT_KEY}"},
        params={"feature": feature, "run_id": run_id},
    )
    status = resp.json()
    if not status["can_proceed"]:
        return False          # Budget exhausted — wrap up
    if status["remaining_usd"] < 0.05:
        return False          # Less than $0.05 left — wrap up
    return True

See Budget Enforcement → Defense in depth for the full LangGraph and CrewAI examples.

Frequently asked questions