How do I split AI spend by environment and compare prompt versions on cost?

Environments

Send X-Cost-Environment on every call. Use any names you like; common ones are production, staging, ci, eval and dev.

import os
from openai import OpenAI
 
client = OpenAI(
    base_url="https://api.cognocient.com/v1",
    api_key=os.environ["COGNOCIENT_KEY"],
    default_headers={"X-Cost-Environment": os.environ.get("APP_ENV", "production")},
)

With the Python wrapper, pass cognocient_environment="staging" instead.

Observe → Environments shows spend, share, calls, tokens and error rate for each environment, a daily stacked chart, and how much of your spend is outside production. Calls without the header are grouped as (untagged).

Budget an environment

Create a budget with scope Environment and the environment name (for example ci). It behaves like any other budget: alert, degrade or block, but only for calls that carry that environment. A common setup is a hard block budget on ci and eval so a runaway test loop cannot eat into production's headroom.

Variants (Experiments)

Tag each call with the version it ran under:

variant = "prompt-v2" if bucket(user_id) else "prompt-v1"   # your own split
client.chat.completions.create(
    model="gpt-4o-mini",
    messages=build_messages(variant),
    extra_headers={"X-Cost-Variant": variant, "X-Cost-Feature": "support-reply"},
)

With the Python wrapper, pass cognocient_variant="prompt-v2".

Intelligence → Experiments (Growth) compares the variants side by side:

Pick a baseline and every other column shows its change against it. Filter by feature, environment or outcome. Variants with fewer than 30 calls are flagged as a small sample.

You can also compare by model (did the cheaper model, or the Auto Router, really save money on this feature?) or by environment.

The same comparison is available to AI assistants through the MCP server (compare_variants).