Anomaly detection for AI spend: a root cause, not just a spike alert

“Spend is up” is the easy half of anomaly detection. Any threshold-based alert can tell you a number moved. The half that actually saves an engineer time is “spend is up, and here's the most likely reason” — the difference between a notification that starts a research project and one that starts a five-minute confirmation.

Getting to a useful root-cause hypothesis requires two things most naive alerting skips: a rolling baseline instead of a fixed number, and a set of recognizable failure patterns the system can match the anomaly's shape against.

Why a flat, fixed threshold produces too much noise to be useful

A hard threshold — alert if spend exceeds $X per hour — has to be picked once and then goes stale as a feature's normal traffic level drifts. Set it for today's baseline and it either fires constantly a month later once usage has organically grown, or misses a real anomaly if usage has since dropped.

The fix isn't a smarter fixed number — it's comparing against a baseline that moves with you. Cognocient builds a rolling baseline per feature from the mean and standard deviation of hourly spend over the past 14 days, and flags an anomaly when a given hour's spend deviates more than 2.5 standard deviations from that baseline — no manual threshold to set, and no re-tuning as normal usage grows or shrinks. (The baseline is a flat 14-day average, not segmented by time-of-day or day-of-week — a Monday morning reading is compared against the same rolling average as every other hour.)

The two anomaly types, and what each usually means

The root-cause hypothesis, not just the alert

Every anomaly ships with a root-cause hypothesis ranked by confidence, generated from a small set of structural signals around the anomaly window: did average tokens per call jump (likely a prompt template change), did a new proxy key appear around that time (likely a new deployment), or did the dominant model for that feature change (likely a routing or code change)? If none of those signals fire, it's reported as an unexplained traffic increase. This is a starting point for investigation, not the final word — confirm it against the API Call Log filtered to that feature and time window. See the full anomaly detection reference for the complete mechanism and notification setup.

There's no one-click action to cap a budget directly from the anomaly card today — if the root cause turns out to be a runaway process, set or tighten a budget for that feature from the Budgets page.

What this looks like in practice

Consider a hypothetical ticket-resolver feature with a stable 14-day hourly baseline around 180 calls and $4.20 in spend. One hour shows 4,200 calls and $98 — a deviation of well over 2,000% against that baseline. The anomaly fires, and the root-cause check finds no new proxy key and no model change, but a call-volume pattern consistent with a retry loop, reported as a call frequency spike with an unexplained-traffic-increase hypothesis.

Confirming the hypothesis via the API Call Log takes a couple of minutes rather than a multi-hour investigation starting from “spend is unusually high, no idea why.” This is illustrative of how detection and root-cause hypotheses behave, not a specific customer's reported outcome.

Anomaly detection catches an unusual pattern after it has already started. Catching a runaway cost rate within the same minute it begins is a faster, complementary mechanism — see the token-velocity circuit breaker guide for how the two work on different timescales.