The task
Take one document, split it three ways, and measure which split answers questions best. The point is not to find the winning number. It is to see that the number depends on the document and the question, which is why the answer cannot be copied from an article.
Starter material
Use a document you already have with real structure: an API reference, a policy handbook, a long README. It needs headings and it needs to be long enough that retrieval has to choose, so aim for 5,000 words or more.
Write down five questions it should be able to answer before you split anything. Two should be answerable from a single paragraph. Two should need information from two different sections. One should have no answer in the document at all.
That last one is not padding. A retrieval system that returns confident passages for a question the corpus cannot answer is the failure you are least likely to notice.
The three splits
Fixed size. 500 tokens, 50 tokens of overlap, ignoring structure entirely.
Recursive. Split on paragraph boundaries, then sentences, falling back to characters, with the same 500-token ceiling.
Structural. One chunk per heading section, whatever size that produces. Record the spread, because you will need it in a moment.
Embed all three sets with the same model. Changing the chunker and the embedding model together tells you nothing about either.
What to check
For each of the five questions, against each of the three splits:
- Does the top result contain the answer?
- Is the answer complete, or cut off at a boundary?
- For the two-section questions, do the top three results cover both sections?
- For the unanswerable question, what came back and how confident did it look?
Record all of it in a table. Sixty cells is enough to see a pattern and few enough to fill in by hand.
Hints
The structural split produced one chunk of 4,000 tokens and several of 80.
That is the expected result and the reason structural splitting is not simply the best option. A chunk far larger than the rest dominates nothing in particular: a single vector over 4,000 tokens covering six subjects is a blunt average, and it tends to rank moderately for everything and strongly for nothing.
The usual fix is a second pass that splits oversized sections by paragraph while keeping the heading as a prefix on each piece. Try it and rerun the five questions.
Every split answers the single-paragraph questions. Nothing separates them.
Then your questions are too easy, which is itself a finding: if your real traffic looks like that, chunking is not where your effort should go.
Make the questions harder in a specific way. Ask something whose answer sits at a boundary — the last sentence of one section and the first of the next. That is the case fixed-size splitting handles badly and structural splitting handles well, and it is where the difference lives.
How do I compare "confidence" on the unanswerable question?
Look at the raw similarity scores rather than the order. A store returns a top result whether or not a good one exists, so the ranking is never empty. The question is whether the best score for the unanswerable question is visibly lower than for the answerable ones.
If it is, you have a threshold worth setting. If it is not, no threshold will save you and the honest answer has to come from the generation step instead.
What you should end up with
A table, and three sentences you can defend:
- Which split you would ship for this document, and why.
- The question type that separated them.
- The number you would have to change if the document doubled in length.
If the third sentence is hard to write, that is the useful discomfort. Chunk size is a decision about the corpus, not a setting with a default.
Going further
- Re-run the structural split with the heading prepended to every chunk, and see whether the two-section questions improve.
- Keep the five questions. They are the beginning of a regression set, and they cost nothing to re-run the next time you change the chunker.