Anomaly detection for AI spend: a root cause, not just a spike alert
“Spend is up” is the easy half of anomaly detection. Any threshold-based alert can tell you a number moved. The half that actually saves an engineer time is “spend is up, and here's the most likely reason” — the difference between a notification that starts a research project and one that starts a five-minute confirmation.
Getting to a useful root-cause hypothesis requires two things most naive alerting skips: a baseline that accounts for normal time-of-day and day-of-week variation, so a Monday morning spike isn't compared against Sunday-night quiet and falsely flagged, and a set of recognizable failure patterns the system can match the anomaly's shape against.
Why a flat threshold produces too much noise to be useful
A fixed threshold — alert if spend exceeds $X per hour — has to be set somewhere, and wherever it's set, it's wrong for some legitimate part of the week. Set low enough to catch a real anomaly on a quiet Sunday and it fires constantly during normal Monday morning traffic. Set high enough to stay quiet during normal peaks and it misses a genuine anomaly that happens during a quiet period, when the same absolute spend level is actually unusual.
The fix isn't a smarter fixed number — it's comparing against a baseline that already accounts for the pattern. Cognocient builds a rolling baseline per feature from the past 14 days, segmented by time-of-day and day-of-week, so a Monday 9am reading is compared against other Monday 9am readings specifically, not against a global average that flattens away the very pattern that makes a threshold noisy in the first place. An anomaly triggers when a metric deviates more than 2.5 standard deviations from that specific-time-slot baseline — no manual threshold to set, and no noisy false positives from comparing across mismatched time windows.
The five anomaly types, and what each usually means
Spend spike (most common)
A feature's hourly or daily cost is significantly above its baseline. Usual causes: an eval harness accidentally left running, a rate-limit backoff loop, or a new deployment with higher token usage than the previous version.
Call frequency spike
Call volume rose sharply without a matching rise in users. Usual causes: an infinite retry loop, a polling pattern, or a background job now triggering an AI call on every execution where it previously didn't.
New untagged feature
A meaningful volume of calls with no X-Cost-Feature header has appeared — a code path reached production without attribution headers attached.
Model change detected
A feature that consistently used one model has switched to a more expensive one — often a code change that isn't noticed as a cost decision until the invoice reflects it.
Budget approaching limit
Current spend rate projects to exhaust a configured budget before the billing cycle ends — flagged well ahead of the actual limit so there's time to react.
The investigation workflow, not just the alert
Every anomaly ships with a plain-English root-cause hypothesis — “likely eval harness left running” or “new code deployment detected at 14:23 UTC” — as the starting point, not the final word. From there, a call-frequency graph over the preceding 48 hours with the anomaly window highlighted distinguishes a one-time event (a sharp vertical line — a deployment or manual trigger) from a gradual rise (a growing backlog or genuine usage growth). Drilling into the live call feed filtered to that feature and time window shows whether the anomaly is concentrated in one user, one session, or spread randomly — the detail that tells you whether it's a code bug, a runaway agent, or legitimate demand. See the full anomaly detection reference for the complete five-step workflow and notification channel setup.
If the root cause turns out to be a runaway process, a budget cap can be applied to the affected feature immediately from the same anomaly detail view — closing the loop from detection to mitigation without switching tools.
What this looks like in practice
Consider a hypothetical ticket-resolver feature with a stable Monday-morning baseline around 180 calls per hour and $4.20 per hour in spend. One Monday, the same hour shows 4,200 calls and $98 — a deviation of well over 2,000% against that specific baseline, not against a flat daily average that might have masked how unusual this particular hour actually is. The anomaly fires with a root-cause hypothesis attached: an eval harness likely left running, given the pattern of uniform, rapid-fire calls concentrated in that one feature.
Confirming the hypothesis via the live call feed takes a couple of minutes rather than a multi-hour investigation starting from “spend is unusually high, no idea why.” This is illustrative of how detection and root-cause hypotheses behave, not a specific customer's reported outcome.
Anomaly detection catches an unusual pattern after it has already started. Catching a runaway cost rate within the same minute it begins is a faster, complementary mechanism — see the token-velocity circuit breaker guide for how the two work on different timescales.