How does Cognocient's semantic similarity caching work?

Cognocient caches semantically equivalent prompts — not just exact duplicates. When a new prompt is close enough to a cached one, Cognocient returns the cached response without calling the provider. A 30% cache hit rate typically reduces your AI bill by 25–35%.

Exact match caching

The simplest form of caching. When the exact same prompt is sent again (byte-for-byte identical), Cognocient returns the cached response immediately without forwarding to the provider.

Enable with a single header:

from openai import OpenAI
 
client = OpenAI(
    api_key="sk-cog-YOUR-PROXY-KEY",
    base_url="https://api.cognocient.com/v1",
)
 
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is our refund policy?"}],
    extra_headers={
        "x-cog-cache": "true",  # enable caching for this request
    }
)

Cache hit indicators:

Exact-cache hits are logged as $0.00 calls and appear in the API Call Log — this is the one caching path that shows up there.

Semantic similarity caching

Exact match only catches identical prompts. Semantic caching catches prompts that mean the same thing. "What is your return policy?" and "How do I return an item?" are different strings but semantically equivalent — semantic caching serves both from cache.

How it works under the hood:

Incoming prompt
       │
       ▼
OpenAI text-embedding-3-small
       │  generates 1536-dim vector
       ▼
pgvector HNSW index
       │  cosine similarity search
       │  threshold: 0.95 (configurable)
       ▼
Cache hit?  ──YES──▶  Return cached response (<10ms, $0.00)
       │
       NO
       │
       ▼
Forward to AI provider (normal flow)
       │
       ▼
Store embedding + response in cache

New headers for semantic caching:

X-Cog-Similarity-Cache: true

Enable semantic (vector) caching for this request. When set to true, Cognocient embeds the prompt and searches the HNSW index before forwarding to the provider.

X-Cog-Similarity-Threshold: 0.95

Cosine similarity threshold for a cache hit. Range: 0.0–1.0. Default: 0.95. Higher values (0.98+) only match near-identical prompts. Lower values (0.90) are more aggressive and may occasionally return slightly mismatched responses.

Semantic caching also requires X-Cog-Cache: true to be set alongside X-Cog-Similarity-Cache: true — similarity matching is layered on top of the base cache opt-in, not a standalone switch.

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is your return policy?"}],
    extra_headers={
        "X-Cog-Cache":                "true",  # required
        "X-Cog-Similarity-Cache":     "true",
        "X-Cog-Similarity-Threshold": "0.90",  # optional, default 0.95
    }
)
 
# Check the response header (when using raw fetch) — the response body
# itself is the original cached provider response, unmodified.
# x-cog-cache-hit: true
# x-cog-cache-type: semantic
# x-cog-similarity-score: 0.973

const response = await openai.chat.completions.create(
  {
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: 'How do I get a refund?' }],
  },
  {
    headers: {
      'X-Cog-Cache':                'true',
      'X-Cog-Similarity-Cache':     'true',
      'X-Cog-Similarity-Threshold': '0.90',
    },
  }
);
 
// "What is your return policy?" was cached with similarity 0.973
// → same response returned, $0.00 billed

Cache TTL (time-to-live)

Cached responses expire after a fixed 24 hours. There is currently no header to override this per request.

Fail-safe behaviour

Caching is designed to be transparent and non-disruptive. If the cache lookup fails for any reason (index unavailable, timeout, error), the request automatically falls through to the provider. Your application never sees a cache error.

When to use (and when not to)

Good candidates

• FAQ responses and help documentation
• Product descriptions and policy questions
• Classification prompts with fixed inputs
• Any prompt where inputs are drawn from a finite set
• Support bot responses to common issues

Not suitable

• Personalised responses with user-specific content
• Real-time data queries (prices, inventory, news)
• Streaming responses where freshness matters
• Creative generation (should vary each time)
• Prompts containing the current date/time

Frequently asked questions