Long Context Processing

Fitting in the window is not the same as being read

A model that accepts 200,000 tokens will accept your 200,000 tokens. Whether it uses them well is a different question, and the answer is not encouraging.

Capacity is a hard limit: exceed it and the request fails. Performance is a curve, and it bends long before you reach the limit. A prompt can fit perfectly and still bury the one sentence that answers the question.

What the research found, and on which tasks

Liu and colleagues measured this in Lost in the Middle. Their finding, in the paper's own words:

performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must > access > relevant information in the middle

The shape is a U. Accuracy is strong at both edges and sags in the middle, so the weak position is not the end of a long prompt but its center.

Two qualifications belong with that finding. The tasks measured were multi-document question answering and key-value retrieval, which is a narrower base than "models read long contexts unevenly" implies. And the effect holds "even for explicitly long-context models", which is the sentence that answers the opening question: a larger window does not exempt you.

Effect size varies by model and by task, so no single percentage describes it. A number from one experiment quoted as a general rule would mislead more than it helps.

Where to place the passages you most need read

Two practical rules follow.

Put the most relevant material at the edges. If your retrieval returns eight passages ranked by score, the conventional order puts the best one first and the worst in the middle, which is fine. The order that hurts is ranking by document position or by date when relevance does not correlate with either.

Put instructions where they will be read. A system prompt at the start and the specific question at the end is the layout that puts both at a strong position. Burying the actual task in the middle of a large context is the arrangement most likely to produce an answer to a different question.

Reranking pays for itself here. It changes what sits in the weak positions, and its value is not only that the best passage is first; it is that the worst ones are where they can do least harm.

The alternatives to one enormous prompt

Before filling the window, consider not filling it.

Retrieve less. Fewer, better passages beat many mediocre ones. If your retrieval returns twenty chunks and three are relevant, sending all twenty puts seventeen distractors in the context and pushes the useful three further apart.

Split across calls. Summarize each document separately, then reason over the summaries. Each call gets a short context it reads well, at the cost of more calls and a summarization step that can drop detail.

Summarize the history rather than carrying it. A long conversation does not need to arrive verbatim. Compaction replaces it with a summary and continues, which is one of the practices a context layer implements.

Measure rather than assume. Whether your assembled context holds the answer, and where, is measurable. Context evaluation covers how, including why precision rewards ranking the right passage early.

The common thread: a large window is a budget, not an invitation. Anthropic's framing is that models have an attention budget every token depletes, which is the same conclusion from the other direction.

Further reading

• Liu et al., Lost in the Middle: the study, its tasks, and the claim about long-context models.
• Context evaluation: measuring whether the context held the answer, and where.
• Large language models: the window as a shared input and output budget.