The cost profile of autonomous AI agents is shifting. According to McKinsey & Company, “Cost versus value: Managing agentic AI system performance”, about 60 % of agentic AI task costs go to iterative response refinement (checking, correcting, improving) rather than the first inference call. That single number reframes how engineering and finance teams should think about budgeting, performance monitoring, and governance for AI‑driven workflows.
1. Understanding the 60 % Figure and What “Response Refinement” Means
An agentic AI (or AI agent) is a software component that can make decisions, invoke tools, and iterate until it reaches a goal. The agent loop is the sequence of steps the agent repeats:
- Prompt generation – the agent builds a request for a large language model (LLM).
- Inference call – the LLM returns a raw answer (the first inference).
- Response refinement – the agent evaluates the raw answer, decides whether it meets the goal, and, if not, modifies the prompt, adds context, or calls auxiliary services (e.g., a calculator, a search API) before asking the LLM again.
Response refinement therefore encompasses any post‑inference activity that triggers another LLM call or a supporting micro‑service. It includes validation (checking factuality), correction (fixing syntax or logic errors), augmentation (adding missing details), and re‑prompting (changing temperature, system messages, or tool usage). Each refinement step consumes compute cycles, token usage, and often additional API calls, all of which appear on the bill as separate line items.
2. Why the First Inference Call Is Not the Cost Driver
The intuition that the first LLM request is the most expensive stems from two misconceptions:
- Token count alone determines cost. While token usage (the number of words or sub‑words processed) is the primary pricing factor, refinement loops typically double or triple the total tokens because each iteration repeats the original prompt plus added context.
- Model size is static. Many teams deploy a single “best‑in‑class” model for the initial call, assuming the same model will be used for refinements. In practice, agents often switch to cheaper, faster models for intermediate checks, but the cumulative cost of multiple calls still outweighs the single‑call expense.
Because the first inference is a single, bounded operation, its share of the total spend is limited. The refinement stage, however, can involve multiple calls, each adding overhead for authentication, network latency, and per‑request fees. The result is a cost distribution where the bulk of dollars flow to the iterative portion of the loop.
3. Production Reality: One Apparent Call, Many Behind‑the‑Scenes Calls
In a production environment, a “single” agent execution may look like this:
| Step | Action | Typical Cost Contribution |
|---|---|---|
| 1 | Generate initial prompt | 5 % |
| 2 | First LLM inference | 10 % |
| 3 | Validate answer (rule‑based check) | 3 % |
| 4 | Re‑prompt with added context | 12 % |
| 5 | Second LLM inference (often a different model) | 15 % |
| 6 | External tool call (e.g., database query) | 5 % |
| 7 | Final polishing prompt | 10 % |
| 8 | Third LLM inference | 15 % |
| 9 | Logging, telemetry, and compliance tags | 5 % |
| Total | – | ≈ 60 % in refinement steps (4‑9) |
The agent appears to the end user as a single “Ask the AI” button, but under the hood it may issue three to five LLM calls, each with its own token bill. Engineering teams that instrument only the first call will dramatically underestimate spend, while finance teams that allocate budgets by “run count” will miss the hidden multiplier effect of refinement loops.
4. The Diminishing Returns Trade‑off Identified by McKinsey
McKinsey’s analysis shows that each additional refinement step yields incremental quality gains that taper off quickly. The first refinement typically resolves the most obvious errors (e.g., missing data fields), delivering a noticeable jump in accuracy. Subsequent refinements often target edge cases or stylistic improvements, which improve user satisfaction but at a higher marginal cost.
A simplified illustration (hypothetical numbers) clarifies the pattern:
- 0 refinements – baseline accuracy 78 %
- 1 refinement – accuracy 86 % (cost +20 %)
- 2 refinements – accuracy 90 % (cost +30 %)
- 3 refinements – accuracy 92 % (cost +45 %)
The curve is concave: the first step provides the biggest return on investment, while later steps deliver diminishing returns. Teams that stop the loop once a predefined confidence threshold is reached can avoid the bulk of the 60 % spend without sacrificing most of the quality benefit.
5. The Blind Spot of “Cost per Run” Metrics
Many organizations track cost per run (total dollars divided by number of agent executions). This metric is useful for high‑level budgeting but masks the internal composition of each run. When the majority of spend is hidden inside refinement, a “low” cost‑per‑run figure can be misleading because:
- Variability across runs – Some tasks require zero refinements, others need four or five. Averaging smooths the spikes, hiding the risk of budget overruns on complex cases.
- Misallocation of departmental charges – If the finance team tags the whole run to a single department, the true cost drivers (e.g., data‑science team that designed the refinement logic) remain invisible.
- Inefficient alerting – Alert thresholds based on total run cost may trigger only after an expensive refinement cascade has already occurred, reducing the chance to intervene in real time.
A more granular view separates inference cost (first call) from refinement cost (subsequent calls). This split enables engineers to profile the loop, identify bottlenecks, and negotiate model contracts that align pricing with actual usage patterns.
6. Implications for Finance and Engineering Teams
Engineering Perspective
- Instrument each loop iteration. Embed lightweight telemetry that records the number of LLM calls, token counts per call, and any external tool usage.
- Set early‑exit thresholds. Use confidence scores or rule‑based checks to stop the loop once the answer meets a predefined quality bar.
- Model‑selection policy. Deploy cheaper, faster models for intermediate refinements and reserve the most capable model for the final answer.
Finance Perspective
- Allocate budgets by refinement stage. Create separate line items for “initial inference” and “iterative refinement” to surface where the bulk of spend occurs.
- Adopt per‑department tagging. Use request headers (e.g., X‑Cost‑Department) to attribute refinement cost to the team that owns the prompt logic.
- Forecast with scenario analysis. Model spend under different refinement depth assumptions (e.g., 1‑ vs 3‑step loops) to understand budget elasticity.
Both sides benefit from a shared dashboard that visualizes the split, allowing rapid negotiation on where to tighten loops without compromising critical business outcomes.
Key Takeaways
- Response refinement dominates cost. Roughly 60 % of an agentic AI task’s spend is tied to the iterative steps that follow the first inference.
- First inference is a small slice. The initial LLM call typically accounts for less than 15 % of total spend in a typical production loop.
- Quality gains taper. Each additional refinement improves output quality, but the marginal return diminishes sharply after the first or second iteration.
- Cost‑per‑run hides detail. Aggregated metrics mask the internal cost structure; separating inference from refinement reveals true spending patterns.
- Targeted controls reduce waste. Early‑exit criteria, model tiering, and granular tagging can cut refinement spend without major quality loss.
What This Means for Your AI Spend
Understanding that most of the bill comes from refinement loops makes it essential to monitor each iteration separately. Cognocient AI’s AI Efficiency Score provides a single 0‑100 metric that aggregates token usage, loop depth, and quality outcomes for every team, giving finance leaders a clear view of where refinement costs are clustering. By tracking this score over time, you can pinpoint agents that are over‑refining and apply budget caps or redesign prompts before expenses spiral.
Free forever on one provider. No credit card, ever.