How do I debug and stop runaway agent loops?

Goal: Identify which agent is looping, stop the active spend, and set up enforcement so it can't recur.

Prerequisites: Agent calls going through the Cognocient proxy (so they're visible in the dashboard).

Step 1 — Spot the loop in Live Calls

Go to Dashboard → Live Calls. Sort by timestamp (newest first). A runaway loop shows up as a dense cluster of calls from the same session ID, hitting the same model repeatedly, each within a few seconds of the last.

Signs of a loop:

• Same session_id appearing 50+ times in under 5 minutes
• Each call has very similar token counts (the prompt hasn't changed)
• Latency is low (the model is responding fine — it's your code that keeps re-calling)

Click any call in the cluster to see the full metadata including the X-Cost-Feature and X-Cost-Session headers. This tells you exactly which feature and which specific run is looping.

Step 2 — Check if a circuit breaker fired

Two independent detectors can catch a loop before you spot it manually:

• Velocity circuit breaker — Go to Dashboard → Engineering Dashboard and check the Circuit Breaker metric in the System Health bar. A trip count > 0 means the token-per-minute limit already caught this loop and started blocking calls.
• Failure Loop Breaker — Go to Dashboard → Loop Breaker. This one catches loops the velocity limit misses: an agent repeating the same tool call, or hitting the same tool error back to back, without necessarily generating tokens fast enough to trip a TPM limit. See Failure Loop Breaker for the two signals it checks.

If neither fired, the loop may be under both detectors' thresholds. Continue to Step 3 to tighten them.

Step 3 — Stop the active loop immediately

If the loop is still running:

Option A — Block the specific session (fastest): In Live Calls, click the session ID → Block session. All further calls from this session ID return 429 immediately.

Option B — Cut the feature budget: In Dashboard → Budgets, find the budget for this feature and temporarily set it to $0. All calls tagged with that feature are blocked instantly until you raise the limit.

Option C — Revoke the proxy key: In Settings → Proxy Keys, revoke the key being used. All calls using that key stop immediately and the action can't be undone. Use this for emergencies — it affects all features on that key.

Step 4 — Set a velocity limit to prevent recurrence

Velocity protection applies per proxy key, not per feature. Auto-detection trips automatically at 10× that key's own rolling baseline — this multiplier is fixed, not user-configurable. If the affected key legitimately needs tighter protection than the auto-baseline gives it, go to Settings → Proxy Keys → Velocity Limit and set a manual tokens-per-minute cap instead — a hard number, not a multiplier.

The circuit breaker uses a sliding 60-second window regardless of which mode is active. Once tripped, calls on that key are blocked until the rate normalises. See The Token-Velocity Circuit Breaker for the full mechanism.

Step 5 — Add a budget check in your agent loop code

The most robust protection is a pre-flight budget check before each agent step. Add this to your agent's step-execution function:

import httpx
 
def budget_ok(feature: str, session_id: str) -> bool:
    try:
        resp = httpx.get(
            "https://api.cognocient.com/api/budgets/status",
            headers={
                "Authorization": f"Bearer {COG_API_KEY}",
                "X-Cost-Feature": feature,
                "X-Cost-Session": session_id,
            },
            timeout=0.5,  # don't let the budget check slow your agent
        )
        return resp.json().get("can_proceed", True)
    except Exception:
        return True  # fail open if status check itself fails
 
# In your agent loop:
for step in planned_steps:
    if not budget_ok("document-processor", run_id):
        logger.warning(f"Budget limit reached after {len(completed_steps)} steps")
        break
    result = execute_step(step)
    completed_steps.append(result)

async function budgetOk(feature: string, sessionId: string): Promise<boolean> {
  try {
    const resp = await fetch("https://api.cognocient.com/api/budgets/status", {
      headers: {
        Authorization: `Bearer ${COG_API_KEY}`,
        "X-Cost-Feature": feature,
        "X-Cost-Session": sessionId,
      },
      signal: AbortSignal.timeout(500),
    });
    const data = await resp.json();
    return data.can_proceed ?? true;
  } catch {
    return true; // fail open
  }
}

Step 6 — Review in the dashboard after

Once the loop is stopped, go to Dashboard → Feature Intelligence and filter to the affected feature. You'll see the exact spike in the cost-over-time chart, with the loop visible as a vertical cost cliff. This view also shows your average cost per call before and during the loop — useful for estimating the total impact.

Preventing loops from the start

For any new agent workflow, apply these defaults before it goes to production:

1. Per-session budget — Cap the cost of a single run (e.g., $0.50 per document processing job).
2. Feature-level Block budget — Hard limit for the feature per month, not just Alert.
3. Budget pre-check in code — The budget_ok() function above in every agent loop.
4. Velocity circuit breaker — Set at 3× baseline, action = Block.

See hierarchical budgets for how to set up all four levels together.