Architecting Zero-Waste AI Systems

I. The anatomy of token waste

Two structural patterns account for most avoidable AI spend, and neither shows up as a bug report. Token maxing is defaulting to the most capable, most expensive model for every task regardless of whether the task needs that capability — a classification or extraction call routed to a frontier model because that was the safest choice at ship time, never revisited as pricing tiers diverged further apart underneath it. Pricing spreads between model tiers within the same provider family can exceed 4,000x for comparable output quality on narrow, well-defined tasks.

Context tax is the recurring cost of resending a large, mostly static system prompt on every call — a 10,000-token preamble of product documentation and tone guidelines, billed at full price on every single turn, when 80–90% of it hasn't changed since the last call. Neither pattern is visible from the provider invoice, which reports model and token count with no signal about whether the model choice or the prompt structure was ever revisited. Both are detectable from the same source: the actual shape of a feature's call history, not its code.

II. The discounts most teams leave unclaimed

Beyond fixing waste, two provider-native discounts exist for the taking and go unused for the same underlying reason waste goes undetected: nobody has looked at real call patterns to see which features qualify. Prompt caching discounts the repeated static portion of a prompt — up to roughly 90% off on cached reads with some providers — directly addressing context tax rather than just measuring it. Batch routing discounts non-real-time workloads by roughly 50% in exchange for turnaround measured in hours instead of seconds — free money for report generation, bulk document analysis, and any pipeline where nobody is watching a live response stream in.

Neither discount requires a model change or a quality tradeoff. Both require knowing which specific features actually match the qualifying pattern — a question about real traffic, not something answerable by re-reading a provider's pricing page.

III. The architecture that determines what's possible

Detecting waste and flagging discounts is a data problem, solvable by reading billing history. Stopping waste before it's spent — blocking a call, auto-downgrading a model under budget pressure, halting a runaway agent loop mid-run — is a different kind of problem entirely, and it has a hard architectural precondition: the system doing it has to sit inside the request path, between the application and the provider, before the call is billed. No amount of engineering sophistication recovers pre-call enforcement from a system that only ever reads billing data after the fact — by the time that data is available, the provider already has the money.

The honest cost of this position is real: a pre-call budget check adds small but genuine latency to every call, and it raises a question that has to be answered directly rather than glossed over — what happens to production traffic if the proxy itself has a problem. A fail-open design answers it: if the enforcement layer becomes unreachable, calls pass through to the provider unaffected, trading temporary loss of cost visibility for continued availability, rather than making an infrastructure hiccup on the cost tool a worse outage than the one it's meant to prevent.

IV. A zero-waste request path

Put together, the four pieces above describe one coherent request path rather than four separate tools: every call is tagged for attribution as it passes through, checked against a budget before it reaches the provider, evaluated against known waste and discount patterns using its actual token shape, and matched against a cache before a new provider call is made at all. None of it requires a second SDK per provider or an after-the-fact reconciliation step — it requires one proxy in the request path, and application code that changes one URL to reach it.