From Pipeline to Loop: What Agentic RAG Actually Changes!
From Pipeline to Loop: What Agentic RAG Actually Changes¶
Strip the hype and agentic RAG is a control structure over retrieval: decide whether to retrieve, grade what came back, verify what was generated, correct if wrong. Each pattern automates one of those decisions — and each sends you a latency bill.
Where the fixed pipeline measurably fails¶
Traditional RAG is a fixed pipeline: retrieve → augment → generate. Every query takes the same path, whether or not retrieval was needed, whether or not the retrieved chunks were any good, whether or not the generated answer actually used them. For single-hop factual lookups that's fine. Then someone asks a question whose answer lives in several places at once, and the pipeline hits a wall you can put a number on.
The FRAMES benchmark (Krishna et al., arXiv:2409.12941) — 824 questions, each requiring two to fifteen Wikipedia articles to answer, scoring factuality, retrieval, and reasoning jointly:
| Retrieval setup | Accuracy | What it tells you |
|---|---|---|
| Single-step retrieval (standard RAG) | ~40% | Naive pipelines can't do multi-hop |
| Multi-step retrieval | ~66% | Iterative retrieval closes most of the gap — 26 points |
| Oracle (perfect) retrieval | ~73% | Even with perfect chunks, the LLM still misses some integrations |
Three readings, all load-bearing. Single-step retrieval is the wrong tool for multi-hop questions — 40% isn't shippable. Multi-step retrieval is worth its cost — a 26-point lift is enormous by any benchmark's standard, and it's the empirical case for everything in this post. And the seven points between multi-step and oracle are a reasoning gap: better retrieval cannot buy them; only better models or better reasoning scaffolds can. Know which gap you're attacking.
From pipeline to control loop¶
Traditional
Query → Retrieve → Generate → Answer
Agentic
Query → Decide / route → Retrieve · grade it → Generate · verify it → Answer
⟲ if ungrounded: retry with a different strategy
That's the whole shift. The LLM (or a separate evaluator model) makes decisions during retrieval rather than before it: do I need to retrieve at all? are these chunks any good? did my answer actually use them? should I retrieve differently and try again? The underlying pattern is one you may recognise from elsewhere in ML systems — a model that judges another model. The same second-order architecture that puts a QA classifier over a segmentation model, or a safety filter over a generator, applied to the retrieval pipeline itself.
Worth saying plainly, because the discourse gets silly: agentic RAG is RAG. Retrieval is still the grounding mechanism; what changed is the control flow around it. And the ingredients are older than the buzzword — ReAct's reason-act loop is 2022, HyDE is 2022, Self-RAG and FLARE are 2023, corrective RAG is early 2024. "Agentic RAG" is the packaging and the enterprise adoption, not the invention.
Self-RAG: the model grades itself¶
Self-RAG (Asai et al., ICLR 2024 oral, arXiv:2310.11511) fine-tunes the generator to emit special reflection tokens at decision points — turning retrieval and self-assessment into part of generation itself:
| Token | Question it answers | Decision triggered |
|---|---|---|
[Retrieve] |
Does the next step need retrieval? | Retrieve, or generate directly — retrieval becomes on-demand, not always-on |
[IsREL] |
Is this retrieved passage relevant? | Irrelevant passages discarded, per passage |
[IsSUP] |
Is this generated claim supported by the passage? | Unsupported claims flagged as ungrounded |
[IsUSE] |
Is the final answer useful for the query? | Poor grades → retry with a different retrieval strategy |
The headline result is the factuality stat: only about 2% of Self-RAG's correct predictions came from outside the retrieved passages, versus 15–20% for baseline LLMs. Its answers are genuinely grounded in retrieval rather than smuggled in from pretraining — which is exactly the property citation-critical domains need.
The fine-tuning bar
Self-RAG's value depends on the reflection tokens being calibrated, and that requires actual fine-tuning on reflection-annotated data. Prompting an off-the-shelf model to "emit IsREL and IsSUP tokens" produces the theatre of self-grading without the calibration — the grades are noise. If you can't invest in fine-tuning, the next pattern is the pragmatic alternative.
CRAG: grade the retrieval, keep a fallback¶
Corrective RAG (Yan et al., arXiv:2401.15884) gets most of the benefit without touching the generator: insert a lightweight retrieval evaluator — a small fine-tuned model, T5-class in the paper — between retrieval and generation. It grades the retrieved set and routes three ways:
- Correct — retrieval is good. Refine it: decompose documents into sentence-level "knowledge strips," keep the strips that matter, generate from those.
- Ambiguous — partial relevance. Keep the useful strips, supplement with a web search, generate from the merged context.
- Incorrect — retrieval failed. Discard it entirely and fall back to web search rather than generating confidently from garbage.
The architectural insight is that retrieval failures cascade: handed irrelevant chunks, an LLM either hallucinates while appearing grounded, emits a non-answer, or produces something subtly wrong. Grading retrieval before generation interrupts the cascade at its source — and the evaluator is cheap enough to run on every query.
The web fallback is a prompt-injection surface
The fallback path feeds your generator content from arbitrary web pages — which adversaries can craft to manipulate the model. Production CRAG needs domain whitelisting, injection detection, and content sanitisation on everything that comes back from the web. Don't ship the naive version; in regulated no-external-source environments, don't ship the fallback at all.
Adaptive RAG and query routing: right-size every query¶
The third decision to automate is the cheapest and, at volume, the most lucrative: how much pipeline does this query deserve? "What's the capital of France?" needs no retrieval. "Summarise revenue trends across our three business units" needs a multi-step plan. Running the heavyweight path for both wastes money and latency on the easy half of your traffic.
Adaptive RAG (Jeong et al., NAACL 2024, arXiv:2403.14403) trains a small classifier to predict query complexity and route accordingly — no retrieval for simple, single-step for moderate, multi-step iterative for complex — reporting better efficiency and accuracy than either always-iterative or always-single-step baselines. The economics make the case by themselves: roughly $0.001 per no-retrieval query, ~$0.005 single-step, ~$0.02–0.05 multi-step. At a million queries a day, routing half of the traffic down the simple path is real infrastructure money, before you count the latency win.
Query routing generalises the idea from "how much pipeline" to "which backend": in a mature deployment the answer may live in the vector store, a SQL warehouse, a knowledge graph, the web, or the model's own weights. The router can be rule-based (fast, deterministic, brittle), classifier-based (generalises, needs training data), or LLM-based (most flexible, costs a call per query). The production pattern that holds up: rules for the predictable 80% of traffic — known patterns, named-entity triggers — with an LLM fallback router for the long tail. Beyond that, the design choices are cascade versus parallel (try the cheap source first and escalate, or fan out and merge) and hard versus soft routing (pick one source, or weight several).
Layering patterns — and the latency bill¶
These patterns aren't competitors; in mature systems they stack, each guarding a different decision:
| Layer | Decision it automates | Latency cost | Build cost |
|---|---|---|---|
| Query routing | Which data source | Low (rules) – medium (LLM) | Low |
| Adaptive RAG | How much retrieval | Low — one classifier pass | Medium — classifier training |
| CRAG-style evaluator | Is the retrieval any good | Medium — evaluator + possible fallback | Medium — evaluator + security hardening |
| Self-RAG reflection | Is the answer grounded | High — reflection during generation | High — LLM fine-tuning |
Stack all four naively and a 50ms baseline query becomes five-plus seconds — every layer adds inline model calls. The architect's question for each layer is the same: what failure mode is this paying for, and does that failure mode exist in my traffic? Routing is almost always worth it. An evaluator gate often is. Generation-time reflection earns its cost only in the highest-stakes domains with the latency budget to absorb it.
When not to go agentic
If your queries are simple and single-hop, a fixed pipeline with good chunking, hybrid retrieval, and a reranker will beat an agentic stack on every axis that matters — cost, latency, debuggability — while matching it on accuracy. Agentic control earns its keep on multi-hop, comparative, and investigative queries. Let the 26-point FRAMES gap argue for it where it applies, and let its absence argue against it everywhere else.
Takeaways¶
- The motivating number: single-step ~40%, multi-step ~66%, oracle ~73% on multi-hop. Multi-step retrieval is worth 26 points where multi-hop questions exist; the last 7 are a reasoning gap retrieval can't close.
- Agentic RAG is a control structure, not a new paradigm. Four decisions get automated: whether to retrieve, what source to use, whether retrieval succeeded, whether the answer is grounded. A model judging a model, applied to retrieval.
- Self-RAG is the ceiling, with a real entry fee. Calibrated reflection tokens require genuine fine-tuning; the payoff is answers that are measurably grounded (~2% out-of-retrieval vs 15–20% baseline).
- CRAG is the pragmatic gate. A small evaluator grading retrieval before generation interrupts the failure cascade — but harden the web fallback against prompt injection before shipping it.
- Route by complexity and by source. The cost asymmetry between pipeline depths makes a once-trained classifier the highest-leverage component at volume. Rules for the 80%, LLM fallback for the tail.
- Every layer sends a latency bill. Stack only the layers whose diagnosed failure mode exists in your traffic — and keep simple lookups single-shot.