The brief
Build an application where someone uploads a handful of documents, asks questions in plain language, and gets answers that cite the passages they came from.
Citations are the requirement that makes this a real project rather than a demo. An answer with no source is a claim; an answer with a source is a claim a reader can check, and the difference shows up in every part of the design.
Requirements
Ingest. Accept at least two formats, one of them PDF. Split each document, embed the pieces, and store them with enough metadata to find their way home: document ID, title, and where in the document the passage sits.
Retrieve. Given a question, return the passages most likely to answer it. Retrieval is scoped to the documents the asker uploaded, and that scope is derived from the session, never from anything the browser sends.
Generate. Answer from the retrieved passages. Every claim carries a citation that resolves to a specific passage, and the interface can show that passage.
Refuse. When the documents do not contain an answer, say so. This is the requirement people cut first and regret last.
Milestones
- Ingest one document and get its chunks back. No embeddings, no model. A route that takes a file and returns the pieces it produced, so you can look at them before anything depends on them.
- Embed and store, then query by hand. A script that takes a question and prints the top five passages with their scores. You are checking that retrieval works before generation can hide its failures.
- Generate an answer with citations. The smallest prompt that takes the passages and returns an answer plus the IDs it used. Do not stream yet.
- Handle the unanswerable question. Add the refusal path, and a question you know the corpus cannot answer, and make it pass.
- Make it multi-document and multi-user. Scope every query to the session. Write the test that queries as one user for another user's content and asserts the result is empty.
- Ship the interface. Upload, ask, read, click a citation and see the passage it came from.
Acceptance criteria
- A question answerable from one passage is answered, with a citation that resolves to that passage.
- A question needing two sections is answered from both, with both cited.
- A question the corpus cannot answer produces a refusal rather than a confident paragraph.
- Every citation in an answer resolves to a passage that exists. No citation points at a document the asker cannot see.
- Querying as one user never returns another user's passages, and there is a test that would fail loudly if that stopped being true.
- Ingesting the same document twice does not double its passages.
- The app reports what a question cost, in tokens or in currency.
Where this usually goes wrong
Citations that point at the chunk instead of the source. A chunk ID means nothing to a reader. Store the document title and the position when you ingest, because you cannot recover them from the vector later.
Scope as a parameter. The filter that limits retrieval to one user's
documents looks like search configuration and sits next to topK, which is
exactly why it gets passed in from the caller. Derive it from the session, and
make the retrieval function impossible to call without one.
Re-ingesting on every deploy. Embedding is the slow, paid part. Make ingest idempotent on a document hash early, or you will pay for the same corpus repeatedly while developing.
Refusal added last. A prompt tuned for months on answerable questions will answer unanswerable ones just as fluently. Add the refusal case at milestone four, not at the end.
Going further
- Add a second retrieval strategy and compare them on the same questions.
- Record which passages were retrieved for every question. When someone reports a bad answer, that record is the difference between debugging and guessing.
- Put a cost per question on the screen. Nothing changes a design faster than watching what it spends.