Chunking Decides More Than Your Retriever!
Chunking Decides More Than Your Retriever¶
The least glamorous, highest-leverage decision in a RAG system — and why the real answer isn't a chunk size at all, but a decoupling: retrieve at one granularity, generate at another.
The tension a single chunk size cannot resolve¶
Two components in your pipeline want opposite things from the same chunk, and that conflict is the whole subject of this post.
The embedding model wants small chunks. Short, focused text produces a vector that means one thing; similarity matching against it is sharp. The LLM wants large chunks. It synthesises better answers from coherent surrounding context; an isolated sentence is often unanswerable on its own. Pick one size and you've chosen which component suffers. Small chunks retrieve precisely and the LLM gets fragments — "…in accordance with regulatory standards…" — that it cannot synthesise, so it fills gaps from pretraining and hallucinates. Large chunks feed the LLM well and the vector averages across topics, so retrieval precision drops and the right chunk never arrives.
The damage from getting this wrong is concrete and unforgiving: a heading severed from its table, a clause severed from its article, code severed from its comment. Each fragment retrieves plausibly and generates garbage. Garbage in, garbage retrieved — when logical units are split arbitrarily, the retriever either misses the fact or hands the generator pieces that no longer mean anything, and no downstream sophistication recovers it.
The reframe
The naive question is "what chunk size?" The better question is "what gets indexed versus what gets retrieved?" — because those don't have to be the same thing. Everything interesting in modern chunking follows from decoupling the two.
The strategy ladder, from blunt to smart¶
| Strategy | Mechanism | Trade-off | Use when |
|---|---|---|---|
| Fixed-size + overlap | Split every N tokens; consecutive chunks share 10–20% | Simple, predictable; splits mid-thought | The honest baseline; uniform simple content |
| Recursive / structural | Split on a hierarchy of separators (section → paragraph → sentence), then size-cap | Respects natural boundaries; variable sizes, some chunks tiny | The robust default for structured text |
| Structural + min-length floor | Structural split, merge chunks below N words | Kills orphan headings; slightly more logic | Books, wikis, manuals |
| Semantic | Split where embedding similarity between consecutive sentences drops past a threshold | Follows the author's train of thought; encodes every sentence — real compute | Narrative content with shifting topics; experiment, don't default |
| LLM-based | An LLM segments text into self-contained propositions | Most coherent, most expensive, hard to audit | Only after simpler strategies measurably fall short |
| Context-added | Any strategy + an LLM prepends a situating summary to each chunk | Improves retrieval and generation; additive to everything above | The best first upgrade beyond fixed-size |
Anti-pattern
Don't reach for tiny chunks on the theory that more granularity equals more precision. Blind 128-token splits orphan tables from their labels and return sentence fragments the LLM can't use; hallucination rates climb until someone finally audits the chunk boundaries. Granularity helps retrieval only when each chunk remains a complete thought.
Decouple retrieval from generation: sentence-window retrieval¶
The first decoupling pattern is almost embarrassingly simple. Index sentences; deliver windows.
- Parse the document into individual sentences. Each sentence is embedded and indexed — that's the retrieval unit, maximally sharp.
- Store each sentence's surrounding ±N sentences as metadata on the node, not in the index.
- At query time, retrieve top-K sentences by similarity.
- Before generation, a postprocessor swaps each retrieved sentence's text for its stored window — the LLM sees coherent context, the retriever never did.
- Then rerank. Order matters: expand to windows first, rerank the expanded text, keep the best few.
The window size is a real tuning parameter with a shape worth memorising — an inverted U. In my runs comparing windows of 1, 3, and 5 sentences on the same eval set: window 1 scored low on context relevance and groundedness — the LLM, starved of context, filled gaps from pretraining. Window 3 was the sweet spot — context relevance roughly 0.57 → 0.90, groundedness effectively saturating. Window 5 went flat on relevance and groundedness dropped: too much surrounding text and the model is overwhelmed by noise and starts hallucinating again, from the opposite direction.
The inverted U
Too small → the LLM invents to fill gaps. Too large → the LLM drowns and invents anyway. Both failure modes end in hallucination, which is why "just add more context" is not a safe default. There is a per-corpus optimum; find yours by running the same eval questions across window sizes, never by feel.
Hierarchies that assemble themselves: parent-child and auto-merging¶
Sentence windows expand contiguously — they help when the relevant material sits next to the matched sentence. They can't help when two non-adjacent chunks from the same section are both relevant. For that, build a hierarchy.
Parent-child is the static version: split documents into large parents (~1,000–2,000 tokens — a section) and small children (~100–250 tokens — a paragraph). Embed and index only the children; parents live in a document store, fetched by ID. Match on a child, return its parent to the LLM. Precision from the small unit, context from the large one.
Auto-merging makes the hierarchy dynamic with one rule: if a majority of a parent's children are retrieved, discard the children and return the whole parent.
Parent (2048 tokens) ← docstore only, never embedded
├── child (128) ← retrieved ✓ ← only leaves are embedded + indexed
├── child (128) ← retrieved ✓
├── child (128) ← retrieved ✓
└── child (128)
3 of 4 children hit → merge up: send the coherent 512/2048-token parent, not three fragments
Because leaves are small, you over-fetch (top-12 or so) to give the merge logic material, then rerank the merged result down to a handful. Two experimental results worth carrying:
- Deeper hierarchies won. A three-layer [2048, 512, 128] tree beat a two-layer [2048, 512] tree by roughly 20% on context relevance — at about half the cost. Smaller leaves match more precisely, which makes better merge decisions, which sends fewer irrelevant tokens to the LLM. Precision and economy moved together.
- The advanced retrievers were also cheaper than the baseline. Better context means the synthesis prompt needs less padding; in my runs both sentence-window and auto-merging cut per-query cost roughly 4× versus naive top-K. "Quality versus cost" is often a false dichotomy in chunking.
| Scenario | Better choice | Why |
|---|---|---|
| Relevant info clusters near the matched sentence | Sentence window | Contiguous expansion is exactly the right shape |
| Non-adjacent chunks of one section both relevant | Auto-merging | Siblings merge through their shared parent regardless of adjacency |
| One parameter to tune, fast iteration | Sentence window | window_size vs a whole hierarchy spec |
| Lowest hallucination risk in production | Both, combined | They target complementary failure modes |
Document structure is free signal — stop throwing it away¶
Everything above treats documents as flat text. Most documents aren't: papers have sections, contracts have clauses, manuals have headings, and that structure is exactly the parent-child hierarchy you were about to approximate with token counts. Structure-aware parsing (GROBID-class tools for scholarly PDFs; equivalent layout-aware parsers for contracts and forms) emits the document's actual hierarchy — and chunking collapses from heuristics into iteration: each section is a parent, each paragraph inside it is a child. No character-count fudge, no overlap guesswork, and a chunk never straddles two unrelated topics because it happened to be 512 tokens long.
Three habits that pay for themselves immediately:
- Never split across a section boundary. A Methods paragraph and a Results paragraph fused into one chunk is a false-positive generator.
- Tables and figure captions become their own chunks, carrying their section as metadata. The number someone needs usually lives in a table; if the table isn't independently retrievable, it's invisible.
- Propagate metadata onto every chunk — section title, source, position, date, access labels. It powers pre-filtering, scoped search, and citations, and it must be attached at ingestion; you cannot bolt it on later.
What the experiments actually show¶
Two pieces of evidence to anchor all of this, one mine and one published.
Chunk size is a query-type decision. Running four chunking configurations over the same book corpus against two query classes: for a broad conceptual query ("history of git"), paragraph chunks with a minimum-length floor won and 25-word fixed chunks lost — tiny chunks matched semantically but carried no narrative. For a specific factual query ("how to add the URL of a remote repository"), the ranking inverted: small fixed chunks pinpointed the exact command while larger chunks buried it in surrounding prose. Neither configuration is "best." Your traffic's dominant query shape decides — which means you need to know your traffic's dominant query shape.
Simple decoupling beats elaborate algorithms. The ARAGOG benchmark (Eibich et al., 2024) ran a head-to-head of advanced RAG techniques on one corpus and found sentence-window retrieval ranked first for retrieval precision — ahead of HyDE, multi-query, MMR, and both commercial and LLM rerankers. A chunking-layer decoupling outscored every fancier retrieval-layer trick on that corpus. The equally important caveat: that's one corpus, and findings like these are corpus-dependent — treat them as a strong prior for where to invest first, then verify on your own data.
Takeaways¶
- Chunking caps everything downstream. Retrieval quality is bounded by ingestion quality; the best reranker in the world can't reassemble a clause that was severed from its meaning at indexing time.
- Decouple retrieval granularity from generation granularity. Sentence-window, parent-child, and auto-merging are one idea in three shapes: match small, deliver big.
- Tune for the inverted U. Both too-little and too-much context end in hallucination. Sweep window/chunk size against a fixed eval set and find your corpus's optimum empirically.
- Prefer deeper hierarchies before fancier algorithms. Three-layer auto-merging beat two-layer on quality and cost simultaneously — and sentence-window beat far more elaborate techniques in published head-to-heads.
- Use the structure your documents already have. Sections are parents, paragraphs are children, tables are first-class chunks, and metadata is attached at ingestion or never.
- Chunk size is a query-type decision. Factual lookups favour small; conceptual questions favour large. Know your traffic before you commit — and re-measure when your traffic changes.