A Failure-Mode Map of Advanced RAG!
A Failure-Mode Map of Advanced RAG¶
Naive RAG answers roughly 44% of factual questions correctly; the best systems reach about 63%. Every named technique — HyDE, Multi-Query, Step-Back, Decomposition, Contextual Retrieval, RAPTOR, GraphRAG — lives inside that gap. The way to close it is diagnosis, not collection.
The ceiling is measured: 44% → 63% → the wall¶
In 2024 we finally got honest numbers. Meta's CRAG benchmark — 4,409 question-answer pairs across five domains and eight question types — measured what RAG systems actually achieve on general factual questions:
| System | Factual questions answered correctly |
|---|---|
| LLM, no retrieval | ~34% |
| Naive RAG (chunk → embed → top-K → generate) | ~44% |
| State-of-the-art RAG, techniques combined | ~63% |
Read it both ways. The optimistic reading: advanced techniques close nearly half the remaining gap — a 20-point swing is enormous, and it is exactly where this post lives. The sobering reading: even the best systems are wrong on roughly a third of factual questions — false-premise queries, fast-changing facts, multi-hop aggregation are where they break. Two consequences follow: evaluation and graceful failure matter as much as raw accuracy, and anyone claiming 95% on general-purpose RAG is usually describing a weak eval, not a strong system.
Failure modes first, techniques second¶
The defining mistake of advanced-RAG adoption is treating techniques as a menu. Every named method exists because one specific failure mode exists. Diagnose first; the technique selects itself.
| Failure mode | Symptom | Reach for |
|---|---|---|
| Vocabulary mismatch | Short query, long technical docs; embeddings don't align | HyDE, query rewriting |
| Single-perspective retrieval | One reading of an ambiguous query misses relevant chunks | Multi-Query, RAG-Fusion |
| Over-specific query | Detail-heavy question buries the underlying concept | Step-Back prompting |
| Compound question | One embedding can't represent three sub-questions | Query decomposition |
| Chunks lose their document context | Fragment makes sense in situ, not standalone | Contextual retrieval |
| Long-document reasoning | Question spans sections; flat chunks miss the higher structure | RAPTOR |
| Multi-hop relationship queries | Answer requires traversing entities across documents | GraphRAG |
The discipline
Name the failure mode, then the mechanism. "HyDE solves the vocabulary-mismatch problem between short queries and long technical documents — the mechanism is…" beats any recitation of technique names. And pick the stage that matches your diagnosis: GraphRAG will not help if your actual problem is that chunking splits tables in half.
Fixing the query: rewriting, Multi-Query, Step-Back, Decomposition, HyDE¶
Raw user queries are often terrible retrieval inputs — vague, reference-laden ("how do I fix it?"), or multi-part. The cheapest interventions happen before the index is ever touched.
Query rewriting is the universal baseline: a small LLM call that strips conversational noise, resolves pronouns against history, expands abbreviations, and injects domain terms. Almost always worth its latency.
Multi-Query generates N rewrites of the query from different angles, retrieves for each, and merges. RAG-Fusion adds Reciprocal Rank Fusion over the N ranked lists, so documents appearing across variants rise. The published critique is worth knowing: generated variants tend to be paraphrases, not perspectives — "RAG pipeline optimization" and "how to improve RAG performance" probe the same direction, and the recall benefit erodes. The fix is to engineer diversity explicitly: prompt for one variant per axis — latency, accuracy, cost — rather than hoping for it.
Step-Back prompting handles the opposite problem: a query so specific that the conceptual background can't match it. Generate a more general version ("what's the memory impact of HNSW M=32?" → "how do HNSW parameters affect memory?"), retrieve against both, combine contexts. One extra LLM call; demonstrated gains on reasoning-heavy benchmarks (Zheng et al., arXiv:2310.06117).
Query decomposition is for compound questions — "compare X and Y on performance, cost, and operations" collapses into one muddy vector if embedded whole. Decompose into focused sub-queries, retrieve each independently, then synthesise. There's a real architectural choice in the synthesis: merge-all-chunks-then-generate is simpler and cheaper; generate-sub-answers-then-combine structures long-form output better and doubles LLM cost. Conversational output favours the first; report-shaped output the second.
HyDE (Gao et al., arXiv:2212.10496) deserves its own paragraph because the mechanism is genuinely counterintuitive: instead of embedding the user's query, have an LLM write a hypothetical answer document — then embed that and search with it. The hypothesis may be factually wrong, and it doesn't matter: what matters is that it's written in the corpus's vocabulary and shape, so the comparison becomes document↔document instead of the inherently asymmetric question↔document. Reported nDCG@10 of 61.3 versus 44.5 for a dense baseline — approaching fine-tuned retrievers zero-shot. The honest bill: it roughly doubles LLM calls per query. At serious volume, a domain-tuned embedding model is often the cheaper long-term answer to the same mismatch.
Fixing the index: contextual retrieval¶
Some chunks are doomed at indexing time no matter how good the query is. "Section 7.3 mandates that…" — which document? "The above approach yields 35% improvement…" — which approach? Chunking strategies mitigate this; they don't fix the underlying problem, because the chunk's embedding still represents the fragment in isolation.
Contextual retrieval (Anthropic, 2024) fixes it at the embedding: before indexing, an LLM generates a short paragraph situating each chunk in its document — what document, which section, what "the above approach" refers to — and that context is prepended to the chunk before embedding and before BM25 indexing. The reported numbers, stated precisely because precision matters here:
| Configuration | Retrieval failure rate | Reduction |
|---|---|---|
| Baseline | 5.7% | — |
| Contextual embeddings | 3.7% | 35% |
| + contextual BM25 | 2.9% | 49% |
| + reranking | 1.9% | 67% |
Note what these numbers are: reductions in retrieval failure rate — how often the right chunk misses the top-K. Not hallucination rate, not end-to-end accuracy. The cost profile is the attractive part: one LLM call per chunk at index time, amortised over every future query (and prompt caching cuts it further, since the document-level context is shared across all of a document's chunks). For enterprise corpora — policies, contracts, specs written assuming the reader holds the whole document in their head — this belongs near the top of any improvement list. The 67% combined figure is among the largest published gains for any single RAG technique.
Fixing the structure: RAPTOR and GraphRAG¶
The last tier reshapes the index itself. These buy the final accuracy points at real complexity cost, and the guidance from everyone who has benchmarked them is consistent: put hybrid retrieval, sensible chunking, and reranking in place first.
RAPTOR (Sarthi et al., arXiv:2401.18059) attacks long-document reasoning. A 200-page report contains paragraph details, section themes, and an executive-summary level of meaning — but flat chunking only indexes the bottom layer, so "what's the main risk factor in this report?" has nothing to match. RAPTOR builds the missing layers: embed all chunks, cluster them (Gaussian mixture, not k-means — it handles variable density), summarise each cluster with an LLM, then recurse upward until a root summary; retrieval searches all levels simultaneously and takes the best match from any altitude. Reported: +20 points absolute on the QuALITY benchmark with GPT-4 (62.4% vs 60.4% DPR). The bill: tree construction is roughly one LLM call per cluster per level — real money on big corpora (a cheaper model for the summarisation step holds up well) — and the tree must be rebuilt as the corpus changes, so it suits stable corpora of long documents.
GraphRAG (Microsoft Research, arXiv:2404.16130) attacks relationship queries — "which vendors support X and have Y?", comparative analysis, causal chains — where flat retrieval fetches two unrelated piles of chunks and hopes the LLM joins them. Offline: LLM-extract entities and relations into a knowledge graph, run community detection (Leiden), generate natural-language summaries per community. Online: local queries traverse the graph; global "sensemaking" queries hit the community summaries. Reported: substantial improvements on global queries over 1M+ token corpora, using 26–97% fewer tokens than naive alternatives — the community summaries are dense. The production-pragmatic pattern is hybrid: vector search carries the bulk of context, graph traversal expands around the specifically-named entities. Pure graph retrieval is slow at scale and entity extraction is unreliable in loose domains.
The upgrade path, and the stacking trap¶
Where to invest depends on where you are. A rough banding that has matched my experience:
| Current retrieval accuracy | Invest in | Typical gain |
|---|---|---|
| < 60% | Better chunking + reranking | ~+25 pts |
| 60–75% | Hybrid search + query rewriting | ~+15 pts |
| 75–85% | Cross-encoder reranking, tuned | ~+10 pts |
| 85–92% | GraphRAG / entity-aware retrieval | ~+7 pts |
| 92–97% | RAPTOR-class hierarchical retrieval | ~+3 pts |
| > 97% | Stop. Optimise latency and cost instead. | — |
The stacking trap
The tempting failure mode is "we use HyDE + Multi-Query + Step-Back + RAPTOR + GraphRAG + contextual retrieval." Stacks like that usually perform worse than a chosen subset: each technique adds latency and cost, several have overlapping mechanisms, and none of them was selected against a diagnosed bottleneck. Three or four techniques matched to measured failure modes beat seven applied indiscriminately — every time I've seen it tested.
And one benchmark caveat to close the loop, because it cuts against this whole post's confidence: ARAGOG — the first head-to-head comparison of these techniques on a single corpus — found sentence-window retrieval beating nearly everything, HyDE confirmed as a real gain, and a well-known commercial reranker showing no notable advantage over naive RAG on that corpus, while LLM-based reranking did help. The lesson isn't "rerankers don't work." It's that technique rankings flip with the corpus. Published benchmarks tell you which failure modes exist and which technique categories address them; they cannot tell you which configuration wins on your data. The bridge between the two is your own evaluation harness — built once, run on every change.
Takeaways¶
- Calibrate against the measured ceiling. Naive ~44%, state-of-the-art ~63% on general factual benchmarks. The 20-point gap is real and closable; the remaining wall is why graceful failure and evaluation matter.
- Diagnose, then select. Every named technique answers one failure mode: mismatch → HyDE, ambiguity → Multi-Query, over-specificity → Step-Back, compounds → Decomposition, context loss → contextual retrieval, long docs → RAPTOR, relationships → GraphRAG.
- HyDE works by making the comparison symmetric — a wrong-but-well-shaped hypothetical document matches the corpus better than the question ever could. Budget the doubled LLM cost honestly.
- Contextual retrieval is the quiet heavyweight: up to 67% fewer retrieval failures for one indexing-time LLM call per chunk. For context-dependent enterprise documents, start here.
- RAPTOR and GraphRAG are last-mile tools. High offline construction cost, specific query shapes. Foundation first: chunking, hybrid, reranking cover most of the gap at a fraction of the complexity.
- Don't stack — match. Three or four techniques tied to diagnosed bottlenecks beat an indiscriminate pile. And validate on your own corpus: benchmark rankings are directional, not portable.