Chunking Markdown for RAG: structure first, size second

Chunking decides what a retriever can return. A chunk that mixes two topics matches neither well; a chunk that lost its heading matches nothing at all. Web pages fetched as Markdown make this easier than PDFs or raw HTML, because the author has already divided the content into sections. This guide shows how to use that structure, where size limits still matter, and what to do with the parts that do not split cleanly.

Split on headings first

Use the page's own outline as the first cut. Each ## or ### section becomes one chunk. Sections are written to cover one idea, which is exactly what you want a retrieved passage to do.

LangChain's MarkdownHeaderTextSplitter takes a list of heading levels and records the headings above each section as metadata.¹ LlamaIndex's MarkdownNodeParser does the same for nodes.² Either is enough for most docs sites and articles.

from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")],
    strip_headers=False,  # keep the heading inside the chunk text too
)
sections = splitter.split_text(markdown)
# sections[i].metadata -> {"h1": "Configure the client", "h2": "Timeouts"}

Size caps and overlap as starting points

Headings alone do not bound size. A reference page can have a 4,000-word section; a changelog can have hundreds of three-line ones. Add two rules:

• Cap long sections. Split them on paragraph boundaries, then on sentences only if a paragraph is still too long. A cap of roughly 800 to 1,200 tokens is a reasonable place to start for documentation and articles.
• Merge tiny sections. Join consecutive short sections under the same parent heading until they reach a few hundred tokens, so a page of one-line FAQ answers does not become forty near-empty chunks.

Treat these numbers as defaults to test, not findings. We have not benchmarked them across corpora, and the right values depend on your embedding model, your questions and how much text you pass to the model per answer. The retrieval guide shows how to build a small question set to tune them on, and the note on chunk size for RAG walks through one tuning pass.

Overlap matters less with heading-based splits, because chunks start at natural boundaries. Keep it for paragraph-split sections only, and keep it small: a sentence or two. Large overlaps mainly produce near-duplicate results.

Note that many splitters count characters, not tokens, by default. RecursiveCharacterTextSplitter measures length with Python's len unless you pass a tokenizer-based length function.

Heading path and page title on every chunk

A section called "Limits" means nothing on its own. "Billing API › Rate limits › Limits" does. Store the heading path as metadata, and also prepend a short header to the text you embed:

Page: Billing API reference (example.com/docs/billing)
Section: Rate limits > Limits

Each key allows 10 requests per second...

This is the cheapest form of the idea behind contextual chunk headers and Anthropic's contextual retrieval: give each chunk the context it lost when it was cut out of its page. For web pages most of that context already exists in the title and headings, so you get it without an extra model call per chunk. Keep the header short; a long one dilutes the embedding.

Tables, lists and code blocks

Some blocks lose their meaning when they are split:

• Tables. A row without its header row is a list of unlabelled values. Keep a table whole when it fits under your cap. When it does not, split by rows and repeat the header row in every piece.
• Code blocks. Never split inside a fenced block. A half function is worse than no function. If a block is longer than the cap, keep it whole as an oversized chunk or split it at blank lines between functions and repeat the fence language.
• Lists. Keep a list with the sentence that introduces it. "The following settings are required:" followed by nothing is a common failure.

Check these by reading chunks, not by reading code. Pick 20 chunks per source type and look for orphaned rows, broken fences and lead-in sentences without their lists.

When the Markdown is bad

Main-content extraction is good but not perfect. Watch for three problems before chunking:

• Navigation residue. Lines such as "Previous page" or "Edit on GitHub" repeated across every page end up in every chunk. Strip them with a per-source rule list; they are easy to spot because they repeat.
• Flat pages. Some pages have no headings at all. Fall back to paragraph splitting with the cap, and use the page title as the only path.
• Near-empty pages. A page with a title and a cookie notice was probably rendered in the browser. Do not chunk it; log it as a failure and find another source.

Parent documents and small-to-big retrieval

Small chunks match queries precisely; large chunks give the model enough to answer. You can have both. Index small units (paragraphs or short sections) for matching, and when one matches, pass its parent section to the model instead. LangChain's ParentDocumentRetriever and LlamaIndex's hierarchical node parser with auto-merging retrieval implement this pattern.

The Dense X Retrieval paper takes the small side further and indexes propositions: single self-contained factual statements. It reports better retrieval than passage- or sentence-level units on the open-domain QA sets it tested, at the cost of a model call to extract the propositions.³ Late chunking goes the other way: it embeds the whole document with a long-context model, then pools token embeddings per chunk, so each chunk vector carries context from the rest of the page.⁴ Both are worth testing once structure-first chunking works and your question set shows what is still missing.

Tooling

You do not need a framework for any of this; a regular expression that splits before ## lines gets you most of the way. If you use one, LangChain's text splitters and LlamaIndex's node parsers both support Markdown headings, recursive splitting and metadata out of the box. Whatever you use, store a chunker version on every chunk, so a change to the rules can be re-run on all pages and compared.

Where to go next

Clean input makes all of this simpler; see web content ingestion for RAG for the metadata to keep before you split. Then measure the result with the question set in retrieval for web-sourced RAG.

Footnotes

1. LangChain documentation, "Split Markdown by headers" (MarkdownHeaderTextSplitter). ↩

2. LlamaIndex documentation, "Node parsers" (MarkdownNodeParser, hierarchical node parser). ↩

3. Chen et al., "Dense X Retrieval: What Retrieval Granularity Should We Use?", arXiv 2312.06648 (2023). ↩

4. Günther et al., "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models", arXiv 2409.04701 (2024). ↩