How do I enable semantic caching to eliminate duplicate API calls?

Add two headers to enable exact and semantic caching. Semantically equivalent requests return at $0.00. A meaningful cache hit rate can cut your AI bill significantly for FAQ-style workloads.

Goal: Eliminate redundant API calls — both exact duplicates and semantically equivalent prompts — so you pay $0.00 whenever the same question has effectively been asked before.

Time: 5 minutes to enable. Cache hits appear immediately.

Best for: FAQ chatbots, help centre assistants, support bots, product Q&A — any workload where users ask similar questions repeatedly.

Step 1 — Add the cache headers to eligible requests

Caching requires two headers: X-Cog-Cache: true opts the call into caching at all (and enables exact-match caching), and X-Cog-Similarity-Cache: true additionally enables semantic (near-match) caching on top of it. Sending only X-Cog-Similarity-Cache without X-Cog-Cache does nothing — semantic caching is layered on top of the base cache opt-in.

Set it once at the client level

To enable caching on every call from a client — not per-request — set both headers in default_headers:

client = OpenAI(
    api_key="sk-cog-YOUR-PROXY-KEY",
    base_url="https://api.cognocient.com/v1",
    default_headers={
        "X-Cost-Feature":         "support-bot",
        "X-Cog-Cache":            "true",
        "X-Cog-Similarity-Cache": "true",
    }
)
# Every call from this client now checks the cache

Step 2 — Verify a cache hit

Make the same request twice, then a semantically similar one. Inspect the response headers:

Header	Value on a cache hit
`x-cog-cache-hit`	`true`
`x-cog-cache-type`	`semantic` (present on similarity hits only)
`x-cog-similarity-score`	e.g. `0.97` — how similar the cached prompt was (similarity hits only)
`x-cog-cost`	`0.00` (similarity hits only)

An exact cache hit (identical prompt, X-Cog-Cache only) is logged as a $0.00 call and shows up in the API Call Log. A semantic cache hit (X-Cog-Similarity-Cache, near-match) currently returns the cached response directly without being logged — it will not appear in the API Call Log or count toward feature spend totals. Don't rely on the call log to measure your semantic cache hit rate; use the response headers on the client side instead.

Step 3 — Tune the similarity threshold (optional)

The default threshold is 0.95. Adjust with X-Cog-Similarity-Threshold:

Threshold	Behaviour	Use when
`0.98`	Near-exact matches only	Factual data, technical specifications
`0.95`	Default — good balance	General FAQ, help centre content
`0.90`	More aggressive matching	Policy questions, canonical answers

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "How do returns work?"}],
    extra_headers={
        "X-Cog-Cache":                "true",
        "X-Cog-Similarity-Cache":     "true",
        "X-Cog-Similarity-Threshold": "0.90",  # more aggressive for FAQ
    }
)

Do not enable caching for prompts that include user-specific content, real-time data, or anything that should vary per user. Cache entries are shared across your account.

Cache lifetime

Cached responses expire after a fixed 24 hours — there is currently no header to override the TTL per request.

What to expect

Check Dashboard → Feature Intelligence for your exact + semantic cache hit rates and total savings. Typical results:

Workload	Expected hit rate	Bill reduction
FAQ / help centre	40–70%	35–60%
Support bot (varied users)	20–40%	15–30%
Product Q&A	30–50%	25–40%
Classification (fixed inputs)	60–90%	50–80%

These are illustrative ranges, not a guaranteed result — your actual hit rate depends on how repetitive your real traffic is.