Why Over-Retrieving Chunks Drives Up RAG Token Costs

Your RAG app is probably paying to read ten chunks so it can quote one. A top-k of 10 or 20 feels like cheap insurance, but retrieved text is the biggest section of the prompt, and RAG token costs follow it. This piece makes the case for measuring where your input tokens go, cutting what the answer never uses, and proving with a fixed question set that quality held.

Why RAG token costs are mostly retrieved text

Picture a typical prompt: a 400-token system prompt, a 50-token question and ten chunks of about 500 tokens each. Retrieved text is 5,000 of 5,450 input tokens, roughly 92%. Drop to four chunks and the prompt falls to about 2,450 tokens. At 100,000 questions a day, that is 300 million fewer input tokens daily. These numbers are illustrative; yours will differ, which is why you measure.

Prompt caching won't rescue you. Caching rewards a repeated prefix, and retrieved chunks change with almost every question, so the largest section of the prompt is the one you can't reuse.

Extra chunks also add latency and distractors. The Lost in the Middle paper found that models do worse when the relevant passage sits in the middle of a long input.

Measure tokens per answer by section

You can't argue about top-k without numbers. Count the tokens in each section of the prompt on every request, and log them next to the usage figures your provider bills.

import tiktoken

enc = tiktoken.get_encoding('o200k_base')

def count(text: str) -> int:
    return len(enc.encode(text))

def prompt_breakdown(system: str, chunks: list[str], question: str) -> dict:
    parts = {
        'system': count(system),
        'chunks': sum(count(c) for c in chunks),
        'question': count(question),
    }
    total = sum(parts.values())
    return {**parts, 'total': total, 'chunk_share': round(parts['chunks'] / total, 2)}

tiktoken is OpenAI's tokenizer, so counts for other models are approximate. Use it for the share of each section, and trust the provider's usage numbers for the bill.

Then find out which chunks the answer actually used. Ask the model to return the ids of the chunks it relied on, and log sent versus used. If you send 10 and it uses 2 on most requests, you have your case.

Reduce RAG cost: retrieve wide, send narrow

Cutting RAG top-k blindly trades money for misses. Instead, keep the vector search wide, because it is cheap. Then use reranking to cut tokens: a reranker scores every candidate against the question, and you send only the best few that clear a score floor.

import cohere

co = cohere.ClientV2('<your-cohere-api-key>')

def select_chunks(query: str, candidates: list[str], top_n: int = 5, min_score: float = 0.1) -> list[str]:
    results = co.rerank(
        model='rerank-v5.0-pro',
        query=query,
        documents=candidates,
        top_n=top_n,
    )
    # An empty list means: skip the LLM call and reply that nothing relevant was found.
    return [
        candidates[r.index]
        for r in results.results
        if r.relevance_score >= min_score
    ]

This call shape comes from Cohere's reranking quickstart. A self-hosted cross-encoder from sentence-transformers is the other common option.

Don't copy the 0.1 floor. In Cohere's own quickstart the correct document scores about 0.12, and score scales differ between models, so tune the floor on your data. The reranker also adds a call and some latency, so put it in your cost math.

The objection: fewer chunks hurts recall

Fair, and there is evidence for it. Anthropic's contextual retrieval write-up found that passing the top 20 chunks worked better than the top 10 or top 5. The same post reports that reranking cut the top-20 retrieval failure rate from 5.7% to 1.9%, and it warns that more context can distract the model.

Note what they did: they retrieved 150 candidates, reranked them and kept 20. They didn't send 150.

So the lesson isn't to send fewer chunks every time. The right number depends on your corpus and your questions, and a reranker makes any number cheaper, because the right chunk lands at position 2 instead of position 14. If your evals say you need 12 chunks, send 12. Just don't pay for 20 on a hunch.

Check that quality holds on a fixed question set

Build 50 to 100 real questions with expected answers, and include awkward ones from your logs. Run the same set across configurations and compare accuracy with average input tokens.

CONFIGS = [
    {'name': 'baseline', 'retrieve_k': 10, 'rerank': False, 'top_n': 10, 'min_score': 0.0},
    {'name': 'rerank-5', 'retrieve_k': 30, 'rerank': True, 'top_n': 5, 'min_score': 0.1},
    {'name': 'rerank-3', 'retrieve_k': 30, 'rerank': True, 'top_n': 3, 'min_score': 0.1},
]

def run_sweep(questions, retrieve, answer, grade):
    # retrieve(question, k) -> list[str]
    # answer(question, chunks) -> (text, input_tokens)
    # grade(text, expected) -> 0 or 1
    rows = []
    for cfg in CONFIGS:
        correct = tokens_in = 0
        for q in questions:
            candidates = retrieve(q['question'], cfg['retrieve_k'])
            if cfg['rerank']:
                chunks = select_chunks(q['question'], candidates, cfg['top_n'], cfg['min_score'])
            else:
                chunks = candidates[:cfg['top_n']]
            text, used = answer(q['question'], chunks)
            correct += grade(text, q['expected'])
            tokens_in += used
        n = len(questions)
        rows.append({'config': cfg['name'], 'accuracy': correct / n, 'avg_input_tokens': tokens_in / n})
    return rows

Decide the accuracy drop you will tolerate before you look at the results, say one point, then pick the cheapest configuration inside it. If you have labelled the source chunk for each question, also log whether it made it into the sent set. That separates a retrieval miss from a generation miss.

Re-run the set whenever you change chunking, the embedding model or the generator.

Take the first step today

Add the per-section token log to one endpoint and let it run for a day. If retrieved text is over half your input, you have the case, and the sweep above tells you how much of it you can drop. In production, input tokens are the part of the bill that scales with traffic, so every chunk you stop sending is a saving you collect on every request.