State & Historical Context

Why the tenth turn costs more than the first

A conversation has no memory on the server. Each request carries the whole history, because that is the only way the model knows what was said.

Inference puts the consequence precisely: turn ten resends turns one through nine, which is why thread cost grows faster than message count. A twelve-turn support chat is not twelve requests worth of input. It is the sum of a growing prefix.

So a long conversation gets more expensive per turn, and at the same time the useful signal in it thins out, for the reasons long context processing covers. Both problems have the same root and the same family of answers.

The transcript is not the state

The instinct is to treat the message list as the thing to preserve, and to look for ways to compress it.

Start somewhere else. Ask what the model needs to answer the next turn, and the answer is usually a handful of settled facts rather than the conversation that produced them.

A booking assistant twenty turns in needs: destination Lisbon, dates fixed, two travellers, window seat requested, budget cabin, payment not yet taken. Six facts. The twenty turns that established them, including the two abandoned date ranges and the flight the customer rejected, are how the facts were reached and not what the model needs to continue.

That reframing is what makes the options below choices rather than compromises. You are not looking for a lossy compression of the transcript. You are looking for the state, and the transcript is one inefficient representation of it.

Three ways to carry a conversation, and what each loses

A rolling window. Keep the last N turns, drop the rest. Simple, and it bounds cost. It loses anything established early, which in the example above is the destination. A customer who says "actually make it three people" at turn twenty-two gets an agent that no longer knows where they are going.

A running summary. Periodically replace older turns with a summary. This is compaction, described in Anthropic's context engineering work as "taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window." It keeps more than a window does, and what it loses is unpredictable: the summariser decides what mattered, before knowing what turn twenty-three will ask.

Extracted state. Keep a structured record outside the prompt and update it as facts settle, then send that record rather than the history. The same source calls the general technique structured note-taking: "the agent regularly writes notes persisted to memory outside of the context window." It keeps exactly what you decided to keep, and the cost is that you have to decide, which means a schema and code that maintains it.

The three are not exclusive. A common shape is extracted state plus a short window of recent turns, so the agent has the settled facts and the immediate conversational context.

The option that saves tokens and costs more

There is a trap in the middle of this, and it comes from caching.

On Anthropic's published rates, checked 2026-09-16, a cache read is billed at 0.1x the base input price. A cached prefix costs a tenth of an uncached one. Prompt caching covers the mechanism.

Caching requires the prefix to be identical between requests. A growing conversation has exactly that property: turns one through nine are unchanged at turn ten, so they can be read from cache.

A rolling window does not. Dropping the oldest turn changes the beginning of the prefix, which invalidates the cache, which means the whole remaining history is billed at full price and rewritten to cache again on every turn.

So the option that sends fewest tokens can produce the larger bill. A fifty-turn conversation kept whole and cached can cost less than the same conversation trimmed to the last ten turns uncached.

The practical shape that follows: keep the prefix stable and append, for as long as quality allows. When you do have to shorten, do it rarely and in large steps rather than on every turn, so you pay the cache rebuild occasionally rather than continuously. Summarising at turn forty and again at turn eighty is cheaper than trimming one turn at a time from turn ten.

Whether that holds for your workload depends on your provider's cache pricing and how long your conversations are, which is a calculation worth doing rather than a rule to adopt.

Further reading

• Anthropic, Effective context engineering for AI agents: compaction and structured note-taking.
• Anthropic, Pricing: the cache multipliers the last section depends on.
• Context isolation: the same techniques applied to separating tasks rather than shortening one conversation.
• Inference: why thread cost grows faster than message count.