Embedding Documents

The text you embed is not the text you show

Once chunking has produced passages, embedding them looks mechanical: call the model, store the vectors. The decision hiding in it is which text goes to the embedding model, and it does not have to be the same text you return to the reader.

A chunk pulled from the middle of a document has lost the thing that made it findable. "The limit was raised to 500" is a sentence about nothing in particular once it is separated from the page titled "Rate limits on the Enterprise plan".

Contextual retrieval, and what it is worth

Anthropic's answer to that is to prepend chunk-specific explanatory context to each chunk before embedding it, typically 50 to 100 tokens describing where the chunk sits in its document.

The measured effect, on their evaluation:

Two things are worth taking from that table. The first row of improvement comes from changing what you embed rather than how you search, which is the cheapest lever available and the one people skip. The second is that the techniques compose: each stage keeps roughly half of what remains.

The generated context costs tokens at index time. Their costing assumes 800 token chunks, 8k token documents, 50 token context instructions and 100 tokens of context per chunk. The context is a meaningful fraction of the chunk, and you pay for it once per chunk rather than once per query.

Index the keywords too

The same evaluation makes a second point plainly: "Embeddings+BM25 is better than embeddings on their own."

So embedding documents is rarely only embedding. Building a keyword index over the same chunks, with the same prepended context, is what lets the system answer the queries that semantic search is bad at: exact identifiers, rare terms, quoted phrases.

If you are going to add it later, add it now. Backfilling a BM25 index means walking the corpus again.

Metadata is what makes retrieval usable

The vector answers "is this relevant". Everything else you need at query time has to be stored beside it:

• Where it came from. Document ID, title, and position. Without these you cannot cite, and a citation that points at a chunk ID means nothing to a reader.
• Who may see it. The tenant or user scope. This one is not a convenience: RAG with dynamic filters covers why it is the difference between a filter and a data breach.
• When it was written, so a stale document can be down-ranked or excluded.
• Which model produced the vector, because you cannot tell by looking, and embedding models explains why a mixed index fails silently.

You cannot recover any of this from the vector later. Store it on ingest or re-ingest.

Making ingest repeatable

Ingest runs more than once. It runs when a document changes, when you fix a chunking bug, and when you switch embedding model.

Key on content, not on filename. Hash the chunk text. A document that re-uploads unchanged should produce no new work and no duplicate vectors.

Make re-ingest idempotent. The most common bug in a young RAG system is the same passage appearing three times in the results because ingest ran three times. It looks like a ranking problem and is not.

Batch the calls. Embedding APIs accept arrays, and one request of 100 texts is not the same cost in time as 100 requests.

Expect to do it all again. Changing embedding model invalidates every vector. Write the ingest job as something you can run over the whole corpus, because one day you will.

Further reading

• Anthropic, Introducing contextual retrieval: the failure rates, the token costing, and the case for combining with BM25.
• Chunking strategies that survive production: what arrives at this stage and how it was decided.
• Storing in a vector DB: where the vectors and this metadata go.
• Embedding models: the choice that decides what re-ingest costs.