---
title: How should we chunk documents for RAG?
source: https://rubradigital.com/answers/rag-chunking-strategy
site: Rubra Digital
description: Chunking strategies for RAG compared: fixed-size, recursive, structure-aware and semantic chunking, with practical guidance on size, overlap and metadata.
updated: 2026-06-15
topic: rag
also_asked: What chunk size is best for RAG? | How much overlap should chunks have? | Is semantic chunking worth it?
---

# How should we chunk documents for RAG?

**Summary:** Chunk along the document structure rather than at a fixed character count: split on headings, sections and natural boundaries so each chunk stays self-contained. For most prose corpora, target 400 to 800 tokens with 10 to 15 percent overlap, and prepend the document title and section heading to every chunk so a retrieved passage carries its own context. Tables and lists should stay intact rather than being split mid-structure. There is no universally best size, so benchmark two or three configurations against a labelled evaluation set on your own corpus. The right answer differs by document type.

## Why chunking decides more than it should

A chunk is the unit of retrieval. If the answer spans two chunks, retrieval
returns half of it. If a chunk contains five topics, its embedding represents
none of them well. Most retrieval quality problems that get blamed on the
embedding model are chunking problems.

## The strategies, in ascending order of effort

**Fixed-size.** Split every _n_ characters. Trivial to implement and reliably
poor: it cuts sentences, separates clauses from headings, and splits tables down
the middle. Use it only as a baseline to beat.

**Recursive.** Split on paragraph breaks, then sentences, then characters, until
chunks fit the target size. A large improvement over fixed-size for very little
work, and a reasonable default for undifferentiated prose.

**Structure-aware.** Parse the document's actual structure (headings, sections,
list items, table boundaries) and chunk along it. This is what I use for most
production systems. Section boundaries are semantic boundaries that the author
already provided; ignoring them is throwing away free signal.

**Semantic.** Use embeddings to detect topic shifts and split there. Sometimes
better than structure-aware on documents with no usable structure: meeting
transcripts, long unformatted text. It costs an embedding pass over the corpus
and, in my benchmarks, rarely beats good structure-aware chunking on documents
that have real structure. Try it, but measure before adopting it.

## Practical parameters

**Size: 400–800 tokens for prose.** Small enough to be specific, large enough to
be self-contained. Question-answer pairs and reference entries work well much
smaller. Narrative or legal text often needs the upper end.

**Overlap: 10–15%.** Enough to catch answers that straddle a boundary. More than
about 20% mostly inflates your index and returns near-duplicate results that
crowd out the genuinely different sources.

**Always prepend context.** Every chunk should carry the document title and its
heading path, for example `Employee Handbook > Leave > Parental leave`. This is
the highest-return, lowest-effort improvement available: it makes a passage
retrievable by terms that appear in its heading rather than its body, and it
gives the model orientation when the passage arrives out of context.

**Keep structures intact.** A table split across chunks is useless in both
halves. Extract tables as units; if a table is too large, repeat the header row
in each part.

**Attach metadata.** Source, section, date, document type, and whatever your
permission model needs. Metadata filters are often more effective than any
tuning of the vector search itself.

## Two techniques worth the extra effort

**Small-to-big.** Embed and search over small, precise chunks, then return the
larger parent section to the model. You get the precision of small chunks in
retrieval and the completeness of large ones in generation.

**Contextual chunk headers.** Generate a one-sentence summary of what each chunk
covers in the context of its document, and prepend it before embedding. It costs
one cheap model call per chunk at index time and measurably improves retrieval
on corpora full of ambiguous references.

## How to decide

Take your labelled evaluation set. Run three configurations: recursive at 512
tokens, structure-aware at 512, and structure-aware at 1024 with parent
retrieval. Then compare recall@5. This takes an afternoon and settles the
question with evidence rather than with whatever the last blog post recommended.

---

Source: https://rubradigital.com/answers/rag-chunking-strategy
Rubra Digital. Independent LLM and RAG consulting for regulated and document-heavy organisations in Europe and North America.
Contact: hello@rubradigital.com