Why Chat History Is the Hidden Cost in Your Token Bill

Your chat feature's token bill grows faster than its traffic, and the cause is the part users never see. Chat history token cost is the line item most forecasts miss, because every turn resends the whole conversation. Below: the arithmetic, a trim-or-summarize class, and the first metric to log.

Why chat history token cost grows with the square of the turns

The Messages API is stateless: you always send the full conversational history. Every earlier message is billed as input again on every turn.

Take a modest assistant with a 1,000-token system prompt, 100 tokens per user message and 300 per reply. Turn n sends about 1,100 + 400 × (n - 1) input tokens. Summed over a session:

| Turns | Content in the conversation | Input tokens billed | At $3 per million (illustrative) | | --- | --- | --- | --- | | 10 | ~5,000 | 29,000 | $0.09 | | 20 | ~9,000 | 98,000 | $0.29 | | 40 | ~17,000 | 356,000 | $1.07 | | 80 | ~33,000 | 1,352,000 | $4.06 |

Go from 10 turns to 40 and the bill grows about 12 times. One 80-turn session bills as much input as about 68 eight-turn ones, so a forecast built on average tokens per request misses it.

Agents make it worse: a 3,000-token tool result at turn 5 of 40 rides along on 35 later requests, 105,000 input tokens for one lookup.

Prompt caching cuts the price, not the growth

Prompt caching bills a repeated prefix at a fraction of the input price: reads cost 0.1x on most Claude models, and 5-minute cache writes cost 1.25x. With every turn landing inside the cache window, the 40-turn session falls from 356,000 to about 55,000 token-equivalents. Turn it on.

It does not remove the growth, though:

• The read term is still quadratic. The 80-turn session comes to about 173,000, roughly 3 times the 40-turn figure for twice the turns.
• The default cache lasts five minutes. A user who returns after a coffee forces a rewrite of the whole history at 1.25x, which is worse than the uncached price.

The objection: trimming makes the bot forget

Fair, and naive trimming earns it. Drop the oldest turns and you may delete the constraint the user stated in turn 2 ("we're on the EU plan"). Three habits shrink the risk:

• Pin what must survive. Facts, IDs and stated constraints go into a summary, not whatever scrolls off.
• Cut bulk first. Old tool results and pasted documents are most of the tokens and rarely needed word for word.
• Measure. Replay your longest real sessions with and without compaction and score both with your existing evals.

Context trimming and conversation summarization to reduce input tokens

This class keeps the last six messages verbatim and folds older ones into a rolling summary in the system prompt. It decides when to compact from the usage the API just reported, so it needs no extra token-counting calls.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

MODEL = "<your-model-id>"
SUMMARY_MODEL = "<a-cheaper-model-id>"
BUDGET = 8_000   # compact when the last prompt went over this many input tokens
KEEP_LAST = 6    # recent messages kept verbatim; keep it even so history starts on a user turn


def prompt_tokens(usage) -> int:
    """Total input the API billed for the last call, cached or not."""
    return (
        usage.input_tokens
        + (usage.cache_creation_input_tokens or 0)
        + (usage.cache_read_input_tokens or 0)
    )


def summarize(old: list[dict], previous: str | None) -> str:
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in old)
    resp = client.messages.create(
        model=SUMMARY_MODEL,
        max_tokens=500,
        messages=[{
            "role": "user",
            "content": (
                "Update this running summary of a conversation. Keep names, IDs, "
                "decisions, constraints the user stated, and open questions. "
                "Drop pleasantries.\n\n"
                f"Current summary:\n{previous or '(none)'}\n\n"
                f"New messages:\n{transcript}"
            ),
        }],
    )
    return resp.content[0].text


class Chat:
    def __init__(self, system: str):
        self.system = system
        self.summary: str | None = None
        self.messages: list[dict] = []
        self.last_prompt_tokens = 0

    def send(self, user_text: str) -> str:
        if self.last_prompt_tokens > BUDGET and len(self.messages) > KEEP_LAST:
            old, self.messages = self.messages[:-KEEP_LAST], self.messages[-KEEP_LAST:]
            self.summary = summarize(old, self.summary)

        self.messages.append({"role": "user", "content": user_text})
        system = self.system
        if self.summary:
            system += f"\n\nSummary of the earlier conversation:\n{self.summary}"

        resp = client.messages.create(
            model=MODEL,
            max_tokens=1024,
            system=system,
            messages=self.messages,
        )
        self.last_prompt_tokens = prompt_tokens(resp.usage)
        reply = resp.content[0].text
        self.messages.append({"role": "assistant", "content": reply})
        return reply

It needs Python 3.10 or newer for the type hints. With the example numbers, compaction fires once a prompt passes 8,000 tokens and drops the next one to about 2,800. The 80-turn session lands near 430,000 input tokens instead of 1.35 million, plus a few cheap summary calls.

Gotchas before you ship it

• Compaction rewrites the prefix, so each one costs a cache write. Use a high trigger and a low target; do not compact every turn.
• The class assumes plain-text messages. In a tool-using agent, replace old tool results with a one-line stub rather than deleting messages, because each tool_use needs its matching tool_result.

First step: log input tokens per turn

Before you change anything, log what each call actually billed. With caching on, input_tokens counts only the uncached part, so log all the input fields.

import logging

log = logging.getLogger("llm_usage")


def log_usage(session_id: str, turn: int, usage) -> None:
    log.info(
        "llm_call session=%s turn=%d input=%d cache_read=%d cache_write=%d output=%d",
        session_id,
        turn,
        usage.input_tokens,
        usage.cache_read_input_tokens or 0,
        usage.cache_creation_input_tokens or 0,
        usage.output_tokens,
    )

Call it after every response for a day, sort sessions by total input tokens, and read the top 20. If a few dozen sessions account for a big slice of the bill, the class above is your fix.

A token budget built on average request size is wrong in exactly the sessions where users stay longest, so measure the tail before you forecast.