Most teams building multi-turn applications with Large Language Models (LLMs) face a daunting challenge: context window optimization. As users interact with their application, the context window grows exponentially, leading to a significant increase in token usage and, subsequently, costs. For instance, a customer support bot that uses a context window of 2,000 tokens can easily exceed 10,000 tokens after just five turns, resulting in a staggering $150 increase in costs per conversation. This can quickly add up, with some teams spending upwards of $10,000 per month on unnecessary tokens.
How context grows exponentially in multi-turn apps
To understand the severity of this issue, let's consider a simple example. Suppose we have a customer support bot that uses a context window of 2,000 tokens to store the conversation history. After each turn, the bot appends the new message to the context window, which can lead to a rapid increase in token usage. If the bot receives an average of 500 tokens per message, the context window will grow by 500 tokens after each turn. After just five turns, the context window will have grown to 3,000 tokens, resulting in a 50% increase in token usage. This can have a significant impact on costs, with some teams seeing an increase of up to 200% in their monthly token usage.
The problem is further exacerbated by the fact that most LLMs charge by the token, with some providers charging as much as $0.05 per token. This means that even small increases in token usage can result in significant cost increases. For example, if a team is using 100,000 tokens per month at a cost of $0.05 per token, their monthly costs would be $5,000. If their token usage increases by just 10%, their monthly costs would increase to $5,500, resulting in an additional $500 in costs per month.
The impact of context growth on costs
To illustrate the impact of context growth on costs, let's consider a real-world example. Suppose we have a team that is using an LLM to power their customer support bot. The bot is currently using 50,000 tokens per month at a cost of $0.05 per token, resulting in monthly costs of $2,500. However, due to the exponential growth of the context window, the team expects their token usage to increase by 20% over the next quarter. This would result in an increase in monthly costs of $500, bringing their total monthly costs to $3,000. This represents a significant increase in costs, and the team would need to find a way to optimize their context window to reduce their token usage and subsequent costs.
Measuring context efficiency: useful tokens vs total tokens
So, how can teams measure the efficiency of their context window? One way to do this is by comparing the number of useful tokens to the total number of tokens used. Useful tokens refer to the tokens that are actually used by the LLM to generate a response, while total tokens refer to the total number of tokens used by the LLM, including those that are not used to generate a response. By comparing these two numbers, teams can get an idea of how efficient their context window is and identify areas for optimization.
For example, suppose a team is using 100,000 tokens per month, but only 50,000 of those tokens are actually used by the LLM to generate a response. This would mean that the team is wasting 50,000 tokens per month, resulting in unnecessary costs. By optimizing their context window, the team could reduce their token usage and subsequent costs.
Calculating context efficiency
To calculate context efficiency, teams can use the following formula:
Context Efficiency = (Useful Tokens / Total Tokens) x 100
For example, if a team is using 100,000 tokens per month, and 50,000 of those tokens are useful, their context efficiency would be:
Context Efficiency = (50,000 / 100,000) x 100 = 50%
This means that the team is only using 50% of their tokens efficiently, resulting in a significant waste of resources.
Summarization strategies: rolling window vs hierarchical
So, how can teams optimize their context window to reduce token usage and subsequent costs? One way to do this is by using summarization strategies. Summarization strategies involve reducing the size of the context window by summarizing or truncating the conversation history. There are two main types of summarization strategies: rolling window and hierarchical.
A rolling window summarization strategy involves summarizing the conversation history by only keeping the most recent messages. For example, a team might use a rolling window of 1,000 tokens, which would mean that only the most recent 1,000 tokens are kept in the context window. This can help to reduce token usage and subsequent costs, but it can also result in a loss of context and accuracy.
On the other hand, a hierarchical summarization strategy involves summarizing the conversation history by grouping related messages together. For example, a team might use a hierarchical summarization strategy that groups messages by topic or intent. This can help to reduce token usage and subsequent costs, while also preserving context and accuracy.
Implementing summarization strategies with Cognocient
Cognocient provides a simple and effective way to implement summarization strategies. With Cognocient, teams can use the X-Cost-Feature header to specify the summarization strategy they want to use. For example, a team might use the following code to implement a rolling window summarization strategy:
import os
import openai
# Set the API key and base URL
api_key = os.environ["OPENAI_API_KEY"]
base_url = "https://api.cognocient.com/v1"
# Create a client instance
client = openai.Client(api_key=api_key, base_url=base_url)
# Set the summarization strategy
summarization_strategy = "rolling_window"
client.headers["X-Cost-Feature"] = summarization_strategy
# Use the client to make API calls
response = client.post("/completions", {"prompt": "Hello, how are you?"})
By using Cognocient, teams can easily implement summarization strategies and reduce their token usage and subsequent costs.
Detecting context bloat in the Cognocient dashboard
Cognocient also provides a dashboard that allows teams to detect context bloat and optimize their context window. The dashboard provides a range of metrics and visualizations that help teams to understand their token usage and identify areas for optimization.
For example, the dashboard might show a graph of token usage over time, with a clear spike in usage corresponding to a particular feature or function. This would allow teams to identify the source of the context bloat and take action to optimize their context window.
Using the Cognocient dashboard to optimize context
The Cognocient dashboard provides a range of tools and features that help teams to optimize their context window. For example, teams can use the dashboard to set budgets and alerts for token usage, so that they can be notified when their usage exceeds a certain threshold.
Teams can also use the dashboard to analyze their token usage and identify areas for optimization. For example, the dashboard might show a table of token usage by feature, with a clear indication of which features are using the most tokens.
| Feature | Token Usage |
|---|---|
| Customer Support | 50,000 |
| Chatbot | 20,000 |
| Search | 10,000 |
By using the Cognocient dashboard, teams can easily identify areas for optimization and take action to reduce their token usage and subsequent costs.
Before and after: a customer support bot case study
To illustrate the benefits of using Cognocient to optimize context window, let's consider a case study of a customer support bot. The bot was using 100,000 tokens per month, with a cost of $0.05 per token, resulting in monthly costs of $5,000. However, after implementing a summarization strategy using Cognocient, the bot was able to reduce its token usage to 50,000 tokens per month, resulting in a cost savings of $2,500 per month.
The team was able to achieve this reduction in token usage by using the Cognocient dashboard to analyze their token usage and identify areas for optimization. They were also able to use the X-Cost-Feature header to specify the summarization strategy they wanted to use, and to set budgets and alerts for token usage.
Results of the case study
The results of the case study were significant, with the team achieving a 50% reduction in token usage and a cost savings of $2,500 per month. The team was also able to improve the accuracy and context of their customer support bot, by using a hierarchical summarization strategy that grouped related messages together.
The team was able to achieve these results by using Cognocient to optimize their context window and reduce their token usage. By using the Cognocient dashboard and the X-Cost-Feature header, the team was able to easily implement a summarization strategy and reduce their costs.
When to truncate vs when to summarize
So, when should teams truncate their context window, and when should they summarize it? The answer depends on the specific use case and the goals of the team.
Truncation is typically used when the team wants to reduce the size of the context window, but still wants to preserve the most recent messages. This can be useful in cases where the team wants to reduce token usage, but still wants to maintain some context and accuracy.
On the other hand, summarization is typically used when the team wants to reduce the size of the context window, while still preserving the most important information. This can be useful in cases where the team wants to improve the accuracy and context of their LLM, while also reducing token usage.
Using Cognocient to truncate or summarize
Cognocient provides a range of tools and features that help teams to truncate or summarize their context window. For example, teams can use the X-Cost-Feature header to specify the truncation or summarization strategy they want to use.
Teams can also use the Cognocient dashboard to analyze their token usage and identify areas for optimization. The dashboard provides a range of metrics and visualizations that help teams to understand their token usage and identify areas for optimization.
Key Takeaways
- Context window optimization is crucial: Teams must optimize their context window to reduce token usage and subsequent costs.
- Cognocient provides a simple and effective solution: Cognocient provides a range of tools and features that help teams to optimize their context window and reduce their token usage.
- Summarization strategies can be effective: Summarization strategies, such as rolling window and hierarchical summarization, can be effective in reducing token usage and improving accuracy and context.
Try Cognocient Free
Most teams using Large Language Models (LLMs) find out about context bloat too late, resulting in an average of $2,000 in wasted spend per month. Cognocient gives teams the tools they need to detect and optimize context bloat, reducing their token usage and subsequent costs by up to 50%.
Start your 10-day free trial →
No credit card required · Setup in 2 minutes.