How should we chunk documents for RAG?

Why chunking decides more than it should

A chunk is the unit of retrieval. If the answer spans two chunks, retrieval returns half of it. If a chunk contains five topics, its embedding represents none of them well. Most retrieval quality problems that get blamed on the embedding model are chunking problems.

The strategies, in ascending order of effort

Fixed-size. Split every n characters. Trivial to implement and reliably poor: it cuts sentences, separates clauses from headings, and splits tables down the middle. Use it only as a baseline to beat.

Recursive. Split on paragraph breaks, then sentences, then characters, until chunks fit the target size. A large improvement over fixed-size for very little work, and a reasonable default for undifferentiated prose.

Structure-aware. Parse the document’s actual structure (headings, sections, list items, table boundaries) and chunk along it. This is what I use for most production systems. Section boundaries are semantic boundaries that the author already provided; ignoring them is throwing away free signal.

Semantic. Use embeddings to detect topic shifts and split there. Sometimes better than structure-aware on documents with no usable structure: meeting transcripts, long unformatted text. It costs an embedding pass over the corpus and, in my benchmarks, rarely beats good structure-aware chunking on documents that have real structure. Try it, but measure before adopting it.

Practical parameters

Size: 400–800 tokens for prose. Small enough to be specific, large enough to be self-contained. Question-answer pairs and reference entries work well much smaller. Narrative or legal text often needs the upper end.

Overlap: 10–15%. Enough to catch answers that straddle a boundary. More than about 20% mostly inflates your index and returns near-duplicate results that crowd out the genuinely different sources.

Always prepend context. Every chunk should carry the document title and its heading path, for example Employee Handbook > Leave > Parental leave. This is the highest-return, lowest-effort improvement available: it makes a passage retrievable by terms that appear in its heading rather than its body, and it gives the model orientation when the passage arrives out of context.

Keep structures intact. A table split across chunks is useless in both halves. Extract tables as units; if a table is too large, repeat the header row in each part.

Attach metadata. Source, section, date, document type, and whatever your permission model needs. Metadata filters are often more effective than any tuning of the vector search itself.

Two techniques worth the extra effort

Small-to-big. Embed and search over small, precise chunks, then return the larger parent section to the model. You get the precision of small chunks in retrieval and the completeness of large ones in generation.

Contextual chunk headers. Generate a one-sentence summary of what each chunk covers in the context of its document, and prepend it before embedding. It costs one cheap model call per chunk at index time and measurably improves retrieval on corpora full of ambiguous references.

How to decide

Take your labelled evaluation set. Run three configurations: recursive at 512 tokens, structure-aware at 512, and structure-aware at 1024 with parent retrieval. Then compare recall@5. This takes an afternoon and settles the question with evidence rather than with whatever the last blog post recommended.