How do I use a custom OpenAI base URL in LlamaIndex?

Use OpenAILike from llama-index-llms-openai-like with api_base=https://api.cognocient.com/v1 and your sk-cog- proxy key. The plain OpenAI class validates model names against OpenAI's list, so use OpenAILike for Claude, Gemini or other providers.

How do I track RAG costs in LlamaIndex?

Tag the query LLM and the embedding model with different X-Cost-Feature values (for example rag:answer and rag:embed). Ingestion, retrieval-time embeddings and answer generation then appear as separate lines with real token costs.

LlamaIndex cost tracking: route LlamaIndex LLMs and embeddings through an OpenAI-compatible gateway

Use OpenAILike and OpenAIEmbedding to send LlamaIndex queries, agents and ingestion through one LLM gateway. Track RAG cost per feature and per user, use any provider, and cap spend before the call.

LlamaIndex apps spend money in two places: embedding documents (often in large batch jobs) and generating answers. Routing both through the gateway shows them separately and lets you cap each.

Setup

pip install llama-index-llms-openai-like llama-index-embeddings-openai

from llama_index.core import Settings, VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms.openai_like import OpenAILike
from llama_index.embeddings.openai import OpenAIEmbedding
 
GATEWAY_URL = "https://api.cognocient.com/v1"
KEY = "sk-cog-YOUR-KEY"
 
Settings.llm = OpenAILike(
    model="claude-sonnet-4-6",          # any gateway model name
    api_base=GATEWAY_URL,
    api_key=KEY,
    is_chat_model=True,
    is_function_calling_model=True,     # lets agents use tool calling
    context_window=200000,
    default_headers={"X-Cost-Feature": "docs-qa:answer"},
)
 
Settings.embed_model = OpenAIEmbedding(
    model="text-embedding-3-small",
    api_base=GATEWAY_URL,
    api_key=KEY,
    default_headers={"X-Cost-Feature": "docs-qa:embed"},
)
 
index = VectorStoreIndex.from_documents(SimpleDirectoryReader("./handbook").load_data())
print(index.as_query_engine().query("What is the parental leave policy?"))

Use OpenAILike, not OpenAI, for non-OpenAI models

LlamaIndex's OpenAI class looks up context sizes by OpenAI model name and raises an error for names it doesn't know, such as claude-sonnet-4-6. OpenAILike takes context_window and capability flags from you instead.

Why split embed and answer costs

Re-indexing a large corpus can cost more in one night than a month of queries. With separate feature tags you can give the ingestion job its own budget, see whether answers or embeddings drive the bill, and spot an ingestion job that re-embeds unchanged documents.

Per-user cost

For a multi-tenant RAG app, set X-Cost-User or X-Cost-Customer per request by building the LLM per request (it is a light object) or by keeping one per tenant. Customer Margin (Growth plan) then shows AI cost per customer next to their revenue.

LlamaIndex cost tracking: route LlamaIndex LLMs and embeddings through an OpenAI-compatible gateway

Setup

Why split embed and answer costs

Per-user cost

On this page