What is a Context Layer?

The symptom: nobody can say what the prompt contained

Someone asks why the assistant gave a customer the wrong refund window. You go looking for what the model was sent, and there is no answer.

The prompt was assembled inline: a template string in the handler, three retrieved chunks appended by a helper, conversation history sliced somewhere else, a system prompt from a constants file that another team edited last week. You can reconstruct roughly what went in. You cannot say what went in.

Two more signs. Changing how retrieval works means editing four call sites, because four features each grew their own assembly. And nobody can answer how much of the window a typical request spends on history rather than on documents, because nothing measures it.

That is the gap a context layer fills.

Defining context engineering against prompt engineering

Anthropic draws the line. Prompt engineering is "writing and organizing LLM instructions for optimal outcomes." Context engineering is "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference."

One is about wording. The other is about what is in the window at all.

A context layer is the component that does the second job. Given a request, it decides what goes into the prompt and returns it. One service, one seam, one place that answers what was sent.

What a context layer is responsible for

Four jobs.

Gather. Pull from every source a request might need: retrieved documents, conversation history, user and account records, tool schemas, the system prompt.

Rank. Order them by what this request needs, and put the most relevant material where the model reads it best.

Budget. Decide what does not go in.

Record. Emit what was assembled and why, so the question you could not answer above becomes a query.

Why the budget is about attention, not capacity

The obvious reason to leave things out is that the window is finite. That is the weaker reason.

Anthropic describes context rot: model performance degrades as token count grows. Models have "an attention budget" that every added token depletes, and the mechanism is architectural, since transformers relate each token to every other one.

So adding relevant context is not free even when it fits. A prompt at half the window can perform worse than the same prompt with half the material removed. The target they give is "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome."

That is what makes budgeting a policy rather than a limit check. Appending until you hit the ceiling is a strategy that degrades quality long before it errors.

Four practices the layer implements

The abstract responsibilities become concrete here. Anthropic names four strategies, and a context layer is where they live.

Compaction. Summarize a long conversation and start again from the summary. The layer decides when a thread is long enough and what the summary must keep.

Structured note-taking. Persist state to files or records outside the window and retrieve them when relevant, so a long-running task keeps working memory without carrying it in every prompt.

Just-in-time retrieval. Load at runtime through tools rather than pre-loading. Rather than pushing every document that might matter into the prompt, give the model a way to ask for one, and fetch it when it does. The window holds what the request turned out to need instead of everything it might have.

Sub-agents. Hand a focused task to a separate agent with a clean window. Only the condensed result comes back, so the detail never reaches the main thread.

Each one is a decision about what occupies the window. Scattered across call sites, they get built four different ways or skipped.

What a single assembly point buys you

One place to change. Switching embedding models, adding a reranker, or changing how much history you carry is one component, not a search across the codebase.

One place that logs. The layer emits what it assembled, so "what did we send" has an answer. Pair that with the request identifier on your trace and a complaint leads to the exact context.

One place to measure. A function that takes a request and returns assembled context can be evaluated on its own, against questions whose answers you know. That is what context evaluation measures, and it needs this seam to exist.

The term is in live use, not a coinage. RAGFlow describes itself as building "a superior context layer for LLMs," which is a reasonable way to read what a retrieval engine is for.

Further reading

• Anthropic, Effective context engineering for AI agents: context rot, the attention budget, and the four practices.
• Chip Huyen, Building a generative AI platform: the order teams tend to add these components in.
• Context evaluation: measuring what this layer produces.