Chunk size for RAG: a starting point and one tuning pass

"What chunk size should I use?" is the most common RAG question, and the honest answer is that no single number wins. What you can have is a sensible starting point and a short, repeatable way to move from it.

Fix the boundaries before the size

On web pages fetched as Markdown, most of the chunking work is already done by the author: sections under ## and ### headings each cover one idea. Split on those first, with a heading-aware splitter such as LangChain's MarkdownHeaderTextSplitter¹ or LlamaIndex's MarkdownNodeParser.² Size only matters for what is left over: very long sections and very short ones.

If you start from fixed-size windows instead, every size you test also moves the boundaries, so you cannot tell whether a better score came from the size or from luckier cuts.

A starting point

For documentation and articles, a reasonable first configuration is:

• split on headings;
• cap sections at roughly 800 to 1,200 tokens, splitting longer ones on paragraph boundaries;
• merge consecutive sections shorter than a few hundred tokens under the same parent heading;
• little or no overlap, because heading-based chunks already start at natural boundaries.

These are defaults to test, not results. We have not benchmarked them across corpora. The chunking guide explains each rule, including tables and code blocks that should never be split.

Count in the right unit

Embedding models truncate by tokens, not characters. Many splitters measure length with a character count unless you give them a tokenizer-based length function, so a "1,000" cap can mean 1,000 characters, roughly 250 tokens of English. Check which one your splitter uses before you compare numbers with anyone else's.

One tuning pass

Chroma's technical report on chunking evaluation measures retrieval at the token level and compares several chunkers and sizes on the same questions; its results differ by chunker, by size and by corpus.³ The practical lesson is the method, not a number:

1. Write about 30 real questions and record the page and section that answers each.
2. Index with the starting configuration and measure recall at k: how often the right section appears in the top results.
3. Read every miss. If the answer was split across two chunks, raise the cap. If the right chunk was found but ranked below longer, vaguer ones, lower it.
4. Change one setting, re-index and measure again.

Two or three passes are usually enough to find a range where results stop changing much. Store the chunker settings on every chunk, so you can re-run a page with new settings and compare.

When size is not the problem

If misses persist at every size, the cause is usually elsewhere: the right page was never fetched, the Markdown kept navigation residue, or the chunk lost its page title and heading path. The retrieval guide shows how to tell these apart.

Footnotes

1. LangChain documentation, "Split Markdown by headers". ↩

2. LlamaIndex documentation, "Node parsers". ↩

3. Chroma, "Evaluating Chunking Strategies for Retrieval" (technical report, 2024). ↩