Most teams using Large Language Models (LLMs) like OpenAI have no idea they are wasting up to 40% of their budget on redundant queries. A $2,000/month OpenAI bill tells you nothing about which features are burning your budget, and teams often realize too late that they are paying for the same query multiple times. Cognocient solves this with semantic caching, a feature that automatically identifies and caches similar queries, so you don't pay for the same query twice. With Cognocient, teams see an average reduction of $800/month in their OpenAI bills, and the cache hit rate is visible in real-time on the dashboard.
What is semantic caching and why it matters
Semantic caching is a technique used to cache queries based on their meaning, rather than exact text matching. This is particularly useful for LLMs, where slight variations in the input query can result in the same output. Cognocient's semantic caching feature uses advanced natural language processing (NLP) algorithms to identify similar queries and cache the results, so that subsequent queries with the same meaning can be served from the cache instead of being sent to the LLM. This not only reduces the cost of using LLMs but also improves the performance of applications that rely on them. For example, a chatbot that uses OpenAI to answer user queries can benefit from semantic caching by reducing the number of duplicate queries sent to the LLM.
The benefits of semantic caching are twofold. Firstly, it reduces the cost of using LLMs by minimizing the number of redundant queries sent to the model. Secondly, it improves the performance of applications that rely on LLMs by serving cached results instead of waiting for the model to respond. Cognocient's semantic caching feature is particularly useful for applications that have a high volume of user queries, such as FAQ bots, search engines, and summarization tools. With Cognocient, teams can see a significant reduction in their LLM costs, with some customers reporting a reduction of up to 40% in their monthly bills.
How semantic caching works
Cognocient's semantic caching feature works by analyzing the input query and identifying its meaning using NLP algorithms. The algorithm then checks if a similar query has been cached before, and if so, serves the cached result instead of sending the query to the LLM. If not, the query is sent to the LLM, and the result is cached for future use. This process happens automatically, without requiring any changes to the application code. The only change required is to point the API calls to Cognocient's endpoint instead of the LLM provider's endpoint.
For example, the following Python code shows how to integrate Cognocient's semantic caching feature into an application:
# Before
import openai
openai.api_key = "YOUR_API_KEY"
client = openai.Completion()
# After — Cognocient intercepts, logs, and tags every call
import requests
api_url = "https://api.cognocient.com/v1/openai/completion"
api_key = "YOUR_API_KEY"
headers = {"Authorization": f"Bearer {api_key}"}
query = "What is the capital of France?"
response = requests.post(api_url, headers=headers, json={"query": query})
In this example, the only change required is to point the API call to Cognocient's endpoint instead of the OpenAI endpoint. Cognocient then takes care of caching the results and serving them from the cache when possible.
The difference between exact and similarity caching
Exact caching is a technique where queries are cached based on exact text matching. This means that if a query has been cached before, the exact same query must be sent again for the cached result to be served. Similarity caching, on the other hand, caches queries based on their meaning, rather than exact text matching. This means that if a query has been cached before, subsequent queries with the same meaning can be served from the cache, even if the text is not identical.
Cognocient's semantic caching feature uses similarity caching, which is more effective for LLMs than exact caching. This is because LLMs are often used to answer complex queries that may have multiple variations, and similarity caching can capture these variations and serve the cached result. Exact caching, on the other hand, would require an exact match of the query text, which may not always be possible.
The following table shows the difference between exact and similarity caching:
| Caching Type | Query Matching | Cache Hit Rate |
|---|---|---|
| Exact Caching | Exact text matching | 10% |
| Similarity Caching | Similar meaning | 40% |
| In this example, similarity caching has a higher cache hit rate than exact caching, indicating that it is more effective at capturing variations in queries and serving cached results. |
Which workloads benefit most: FAQ bots, search, summaries
FAQ bots, search engines, and summarization tools are some of the workloads that benefit most from Cognocient's semantic caching feature. These applications often have a high volume of user queries, and semantic caching can help reduce the number of redundant queries sent to the LLM. For example, a chatbot that uses OpenAI to answer user queries can benefit from semantic caching by reducing the number of duplicate queries sent to the LLM.
Search engines can also benefit from semantic caching, as users often search for similar queries. By caching the results of similar queries, search engines can improve their performance and reduce the cost of using LLMs. Summarization tools can also benefit from semantic caching, as they often require generating summaries of similar documents. By caching the results of similar summaries, summarization tools can improve their performance and reduce the cost of using LLMs.
Cognocient's semantic caching feature is particularly useful for these workloads because it can help reduce the cost of using LLMs and improve the performance of applications that rely on them. With Cognocient, teams can see a significant reduction in their LLM costs, with some customers reporting a reduction of up to 40% in their monthly bills.
Measuring cache hit rates in the dashboard
Cognocient provides a dashboard that allows teams to measure the cache hit rate of their applications. The cache hit rate is the percentage of queries that are served from the cache instead of being sent to the LLM. A higher cache hit rate indicates that the semantic caching feature is effective at capturing variations in queries and serving cached results.
The dashboard provides a detailed breakdown of the cache hit rate, including the number of queries served from the cache, the number of queries sent to the LLM, and the total cost savings. Teams can use this information to optimize their applications and improve the performance of the semantic caching feature.
For example, the following table shows the cache hit rate for a chatbot that uses OpenAI to answer user queries:
| Date | Queries Served from Cache | Queries Sent to LLM | Total Cost Savings |
|---|---|---|---|
| 2023-02-01 | 100 | 200 | $50 |
| 2023-02-02 | 150 | 150 | $75 |
| 2023-02-03 | 200 | 100 | $100 |
| In this example, the cache hit rate is increasing over time, indicating that the semantic caching feature is becoming more effective at capturing variations in queries and serving cached results. |
Combining prompt caching with semantic caching
Prompt caching is a technique where the results of a query are cached based on the prompt used to generate the result. Semantic caching, on the other hand, caches queries based on their meaning, rather than the prompt used to generate the result. Cognocient's semantic caching feature can be combined with prompt caching to improve the performance of applications that rely on LLMs.
By combining prompt caching with semantic caching, teams can reduce the number of redundant queries sent to the LLM and improve the performance of their applications. For example, a chatbot that uses OpenAI to answer user queries can benefit from combining prompt caching with semantic caching by reducing the number of duplicate queries sent to the LLM.
The following code example shows how to combine prompt caching with semantic caching using Cognocient's API:
# Before
import openai
openai.api_key = "YOUR_API_KEY"
client = openai.Completion()
# After — Cognocient intercepts, logs, and tags every call
import requests
api_url = "https://api.cognocient.com/v1/openai/completion"
api_key = "YOUR_API_KEY"
headers = {"Authorization": f"Bearer {api_key}"}
prompt = "What is the capital of France?"
query = "What is the capital of France?"
response = requests.post(api_url, headers=headers, json={"prompt": prompt, "query": query})
In this example, the prompt and query are cached separately, and the results are served from the cache when possible.
Real numbers: before/after with gpt-4o-mini
Cognocient's semantic caching feature can help teams reduce their LLM costs by up to 40%. For example, a team that uses OpenAI's gpt-4o-mini model to answer user queries can see a significant reduction in their costs by using Cognocient's semantic caching feature.
Before using Cognocient, the team may have seen a monthly bill of $2,000 for 100,000 queries. After using Cognocient's semantic caching feature, the team can see a reduction in their costs to $1,200 per month, with a cache hit rate of 40%. This represents a cost savings of $800 per month, or 40% of their original costs.
The following table shows the before/after numbers for the team:
| Month | Queries | Cost | Cache Hit Rate |
|---|---|---|---|
| Before | 100,000 | $2,000 | 0% |
| After | 100,000 | $1,200 | 40% |
| In this example, the team sees a significant reduction in their costs and an improvement in their cache hit rate after using Cognocient's semantic caching feature. |
Key Takeaways
- Semantic caching reduces costs: Cognocient's semantic caching feature can help teams reduce their LLM costs by up to 40%.
- Similarity caching is more effective: Similarity caching is more effective than exact caching for LLMs, as it can capture variations in queries and serve cached results.
- FAQ bots, search, and summaries benefit most: FAQ bots, search engines, and summarization tools are some of the workloads that benefit most from Cognocient's semantic caching feature.
- Cache hit rate is measurable: Cognocient provides a dashboard that allows teams to measure the cache hit rate of their applications.
Try Cognocient Free
Most teams using Large Language Models (LLMs) like OpenAI waste up to 40% of their budget on redundant queries, resulting in an average monthly cost of $800. Cognocient gives teams a semantic caching feature that automatically identifies and caches similar queries, reducing their LLM costs by up to 40% and improving the performance of their applications.
Start your 10-day free trial →
No credit card required · Setup in 2 minutes.