Context Engineering
Five things make up the payload, and each is a decision with its own failure. Tool definitions are billed on every call whether used or not.
From wording an instruction to choosing a payload
Prompt engineering asks how to word an instruction. Context engineering asks what should be in the window at all.
Anthropic draws the line this way: prompt engineering is "writing and organizing LLM instructions for optimal outcomes", while context engineering covers "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference."
The shift happens when you notice that your prompt is mostly not your prompt. By the time a request reaches the model, your carefully worded instruction is a small fraction of what arrived, and the rest got there through code nobody thought of as writing.
What the payload is made of
Five parts, each assembled by a different piece of your system.
The system instruction. Usually static, often the only part anyone edits.
Conversation history. Grows every turn, and is resent in full each time.
Retrieved documents. Whatever your search returned for this request, which varies per request and is the part you understand least at any given moment.
Tool definitions. Names, descriptions and schemas for everything you declared. These ride along on every call whether or not the model uses them.
Tool results. Whatever came back from a tool, which is content you did not write, from a source that may not be yours.
Most teams can name two of the five, which is why the full list is set out here.
Each part is a decision with its own failure
Pair each part with the way it goes wrong, and the discipline stops sounding abstract.
| Part | The decision | The failure |
|---|---|---|
| System instruction | What is worth saying once, every time | Rules that contradict each other as it grows |
| History | How much to carry, and when to summarize | Cost that climbs every turn, older content crowding out newer |
| Retrieved documents | How many, ranked how, placed where | Noise crowding the answer, or the answer never arriving |
| Tool definitions | Which tools this request needs | Paying for thirty tools on every call, and more chances to pick wrong |
| Tool results | What to keep of what came back | A long result consuming the window, and untrusted text reaching the model |
The tool-definition row surprises people. Definitions are input tokens on every request, and enabling tools adds a system prompt of a few hundred tokens on top. A long tool list is a standing cost.
The tool-results row is the security one. Text you did not write reaches the model, which is why prompt injection is a context problem before it is a prompting problem.
Why more context is not better
The obvious reason to leave things out is that the window is finite. That is the weaker reason.
Anthropic describes context rot: performance degrades as token count grows, because models have an attention budget that every added token depletes. So adding relevant material is not free even when it fits, and the goal is "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome."
What is a context layer develops this and the practices that follow from it.
What context engineering covers
Treat the rest of this section as the parts of the payload above.
Assembly. Where the decisions live, as a component rather than scattered through call sites.
Compaction. Summarizing history so a long conversation does not carry verbatim.
Retrieval and ordering. What comes back and where it sits, which matters because position affects recall.
Memory. State that outlives a conversation, held outside the window and fetched when relevant.
Security. Which sources are trusted, and what is done with content that is not.
Measurement. Whether the assembled context contained the answer, which context evaluation covers.
The term is not a coinage of this site. RAGFlow describes itself as building "a superior context layer for LLMs", and Atlan calls itself "the context layer for enterprise AI", arguing that "context doesn't come from a prompt. It comes from a pipeline." Two vendors reaching for the same phrase is a reasonable sign the category is real.
Further reading
- Anthropic, Effective context engineering for AI agents: the definition, context rot, and the practices.
- What is a context layer: the component that does this work.
- Chip Huyen, Building a generative AI platform: where these pieces sit among the rest.
Knowledge check
Question 1 of 3
Sign in to save your progress and pick up where you left off.