Your bar: transform the raw query before you retrieve, match the
retrieval structure to the shape of the question, and know exactly when to graduate
from a static pipeline to a self-correcting agentic loop. By the end you can answer:
given a RAG system that retrieves the wrong thing, which lever — pre-retrieval, retrieval
structure, or an agentic loop — do you pull, and what does each cost?
Lesson 3 shipped the baseline RAG pipeline. This lesson covers the three changes that turn that
baseline into a system. Each chip is one lever:
the raw query is a bad search query
so transform it first (rewrite · HyDE · decompose)
flat top-k is not the only retrieval structure
and retrieval can be an action in a loop, not a step
Three independent upgrades to the same pipeline. M1 rewrites the input before it hits the
index, M2 changes what "retrieval" returns, and M3 wraps the whole thing in a loop
that checks its own work. You can adopt one, two, or all three.1
1
Query Transformation
The raw user query is often a bad search query: short, vague, full of pronouns, worded
nothing like the documents that answer it. Fix it in a pre-retrieval stage, before it
touches the index.
A stage that didn’t exist in the baseline
One vague query in, several precise queries (or a hypothetical answer) out. The model that
answers is unchanged; you hand the index a better search than the user typed.2
One phrasing misses; generate N paraphrases, retrieve each, union & dedupe → higher recall.
N retrievals + dedupe
HyDE
Questions look nothing like answers; embed a drafted hypothetical answer to bridge the gap.
1 LLM draft + embed
Decomposition
A multi-part question no single chunk answers; split into sub-queries, retrieve each, combine.
1 split + k retrievals
Step-back
Too specific to find grounding; ask a broader question first, then the specific one.
1 extra retrieval
Routing
Wrong datasource entirely; classify the query and send it to docs vs code vs SQL.
1 classify call
HyDE — the query/document vocabulary gap
Questions and answers use different words, so the question's embedding sits far from the
document that answers it. HyDE has the LLM draft a plausible answer and embeds that —
even a partly-wrong draft has the right shape to retrieve the real source.2
Is Pre-retrieval is any transform applied to the query before it hits the
retriever: rewrite, multi-query, HyDE, decomposition, step-back, or routing. The index and
model are untouched.
Why it exists Recall lost at retrieval cannot be recovered downstream — no reranker,
no bigger model can return a document that was never retrieved. The cheapest place to fix
recall is the query.
Like (world) A good librarian doesn’t search your exact words. They restate “why is
it slow?” as “checkout latency, connection pool” — the terms the shelves are actually
organized by.
Like (code) Query normalization / canonicalization before a DB lookup: lowercasing,
expanding synonyms, splitting a compound filter into sub-queries. Same input cleanup,
applied to retrieval.
✗ “Just embed the user’s question and search.”
✓ The question is the worst search query you have — vague and worded unlike the answer. Transform it first; HyDE and multi-query routinely beat the raw query.
✗ “A better reranker will fix our low recall.”
✓ Reranking only reorders what was retrieved. If the right doc never entered the candidate set, no reranker can surface it — that’s a pre-retrieval problem.
📥 Memory rule:Fix the query before you fix the index. HyDE
and multi-query buy recall that no reranker can recover.
Memory check
Why can't a reranker recover lost recall? → it only reorders the candidates already retrieved; a document never in the set can't be ranked up
What gap does HyDE bridge? → the query↔document vocabulary gap — questions look unlike answers, so it embeds a drafted hypothetical answer instead
Multi-query vs decomposition? → multi-query = N paraphrases of the SAME question (recall); decomposition = split a multi-part question into sub-questions (coverage)
M1 — Your RAG recall is poor on multi-part questions like “compare our refund policy in the EU vs the US.” Which pre-retrieval technique, and why?
Hit these points: this is two facts in one query, so a single embedding averages them and matches neither cleanly → decomposition: split into “EU refund policy” and “US refund policy”, retrieve each, then combine the chunks for the answer → optionally add multi-query per sub-question for recall → contrast: HyDE helps the vocabulary gap, not the multi-fact problem; a bigger window doesn’t help if neither fact was retrieved → verify by checking that chunks for both regions appear in the assembled context, not just the answer.
2
Advanced Retrieval Structures
Lesson 3’s baseline was flat chunk + top-k: one embedding per chunk, return the nearest
k. That’s a floor. The shape of the question should pick the structure of the
retriever.
Flat vs structured retrieval
You can decouple what you match on from what you return. Match tiny chunks
so the embedding is precise, but hand the model the larger parent so it has the surrounding
context to reason with.1
Which structure for which question
Structure
What it’s good at
Cost / complexity
Reach for it when…
Parent-doc / small-to-big
Precision of small chunks + context of the parent.
Pre-filter by date / type / tenant before vector search.
Low — needs clean metadata
Recency, access control, multi-tenant.
GraphRAG
Subgraphs + community summaries for global themes.
High — build & maintain graph
”Connect-the-dots across the whole corpus.”
Local fact vs global theme — the GraphRAG line
Question shape
Flat chunk + top-k
GraphRAG
”What is the retry limit?”
✓ one chunk answers it
✗ overkill, slower
”What themes recur across all incidents?”
✗ no single chunk holds it
✓ community summaries answer it
Read the diagonal. A local fact lives in one chunk, so flat retrieval finds it and
GraphRAG is wasted complexity. A global theme is spread across the whole corpus — no
chunk contains it, so only a graph that pre-aggregates relationships can answer.
Is Advanced retrieval structures change what you index and return — parents,
summaries, situated chunks, token vectors, filtered candidates, or graph subgraphs —
instead of one flat vector per chunk.
Why it exists Different questions have different shapes. Flat top-k optimizes one
trade-off (chunk-level similarity); it can’t serve precision-with-context, recency, or
corpus-wide synthesis at the same time.
Like (world) A library uses a card catalog (summaries) to find the shelf, the full
book (parent) to read context, and a citation graph to answer “what influenced what.”
Different tools, different questions.
Like (code) Choosing an index for a query pattern: a B-tree for point lookups, a
covering index for range scans, a graph DB for traversals. You match the data structure to
the access pattern.
✗ “GraphRAG is a strictly better RAG — use it everywhere.”
✓ It wins on global, theme-spanning questions and loses on local facts: it’s costlier to build and run. For “what is X?” flat top-k is cheaper and just as good.
✗ “Smaller chunks just mean worse context for the model.”
✓ With small-to-big you match on the small chunk for precision but return the parent for context — you get both, not a trade-off.
📥 Memory rule:Match the retrieval structure to the question
shape — local fact → small-to-big; global theme → GraphRAG.
Memory check
What does small-to-big decouple? → what you MATCH on (small precise chunk) from what you RETURN (the larger parent for context)
When does GraphRAG beat flat top-k? → global "connect-the-dots / themes across the whole corpus" questions no single chunk can answer; not for local facts
What two things does contextual retrieval add? → a short chunk-situating context prepended before embedding, plus a BM25 lexical index alongside the vectors (Anthropic)
M2 — A teammate proposes rebuilding the whole RAG stack on GraphRAG because “it’s the state of the art.” How do you respond as the senior in the room?
Hit these points: first ask what questions users actually ask — pull the distribution → if most are local lookups (“what is the limit for X?”), flat top-k or small-to-big already answers them at a fraction of the cost → GraphRAG earns its keep only on global, theme-spanning questions, and it carries real cost: building the entity graph, generating community summaries, keeping it fresh on every reindex → propose it as an added retriever routed to for global queries, not a rip-and-replace → name the trade-off: complexity and maintenance vs a capability you only need for a slice of traffic.
3
Agentic & Self-Correcting RAG
Modules 1–2 still ran retrieval once, as a fixed step. The last shift makes
retrieval an action inside a loop: the model decides whether to retrieve, grades what it
got, and critiques its own draft. This is the read-on-demand pattern from Lesson 3 and the
ai-agents track.
The self-correcting loop
Static RAG runs left-to-right once. Agentic RAG adds the red and grey return paths: bad docs
trigger re-retrieval or a web fallback, and an ungrounded draft is sent back for another
pass before anything reaches the user.3
Three named patterns
Pattern
The control loop
What it buys
Agentic RAG
LLM decides whether / what / when to retrieve; can issue many searches, use tools, iterate. (This is what Claude Code does.)
Handles open-ended, multi-step questions.
Self-RAG
Model retrieves on demand, then reflects: critiques its own draft and revises before answering.
Catches ungrounded / unsupported claims.
Corrective RAG (CRAG)
Grade retrieved docs; if quality is low, fall back (e.g. web search) or re-retrieve before answering.
Robustness when the index comes up short.
The trade-off you must name
Static pipeline
Agentic loop
Quality / robustness
Fixed — one shot, no recovery
✓ checks & corrects itself
Latency
✓ one pass, predictable
✗ multiple round-trips
Cost
✓ few LLM calls
✗ grading + re-tries cost more
Complexity
✓ simple to reason about
✗ loops, budgets, stop conditions
Read the columns. The agentic loop wins exactly one row — quality — and pays in latency, cost,
and complexity for it. Adopt it only where correctness justifies the bill; a static pipeline is
the right default for simple, well-covered questions.
Is Agentic / self-correcting RAG turns retrieval from a fixed pipeline step into an
action the model invokes in a loop — deciding when to search, grading results, and
critiquing its own draft before finalizing.
Why it exists A static pipeline answers once from whatever it retrieved, right or
wrong. A loop lets the system detect a bad retrieval and recover — re-search, fall back, or
revise — instead of confidently answering from junk.
Like (world) A researcher who checks whether their sources actually support the
claim, goes back to the library when they don’t, and proofreads the draft before submitting
— rather than writing from the first book they grabbed.
Like (code) A retry-with-validation loop versus a single fire-and-forget call:
validate the response, on failure refine the request and retry, with a max-attempts budget
so it terminates.
✗ “Agentic RAG is the upgrade — always loop.”
✓ Loops add latency, cost, and failure modes (runaway re-tries, no stop condition). For simple, well-covered questions a static pipeline is correct and far cheaper.
✗ “Self-correction means the model can’t ever be wrong now.”
✓ The critic is the same fallible model; it reduces ungrounded answers, it doesn’t eliminate them. You still need eval and stop conditions, not blind trust.
📥 Memory rule: Static RAG answers once; agentic RAG checks its own
work — pay the extra calls only when correctness justifies them.
Memory check
What does CRAG do when retrieved docs grade poorly? → falls back (e.g. web search) or re-retrieves before answering, instead of answering from low-quality docs
What is the core trade-off of agentic loops? → higher quality / robustness vs more latency, cost, and complexity — only worth it when correctness justifies it
What does Self-RAG add over plain retrieval? → reflection — the model critiques its own draft for grounding and revises before finalizing
M3 — When would you NOT add an agentic loop to a RAG system, and how do you decide?
Hit these points: when the question distribution is simple and well-covered by the index — a static pipeline already answers correctly, so a loop only adds latency and cost → when latency budget is tight (interactive UX) and the extra round-trips break the SLA → when there’s no reliable grading signal, the loop can’t tell good from bad and just burns calls → decide from eval data: measure static-pipeline accuracy first; add the loop only on the slice where correctness is high-stakes and static accuracy is provably insufficient → always cap attempts and define a stop condition so it terminates.
Retrieval practice — test the three modules
Q1. The core argument for pre-retrieval query transformation is that…
Q2. HyDE (Hypothetical Document Embeddings) works by…
Q3. The "small-to-big" (parent-document) retriever gives you…
Q4. GraphRAG most clearly beats flat chunk + top-k when the question is…
Q5. Corrective RAG (CRAG) differs from a static pipeline because it…
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs &
failure modes · Staff = synthesis under ambiguity · System Design = open
design round (a different axis, not a harder level).
What is pre-retrieval query transformation, and name the main techniques.
Hit these points: it's any transform applied to the raw query before it hits the retriever — the index and model are left untouched → rewriting cleans up vague / pronoun-laden / misspelled input → multi-query generates N paraphrases, retrieves each, unions and dedupes → HyDE drafts a hypothetical answer and embeds that → decomposition splits a multi-part question into sub-queries → step-back asks a broader question first → routing classifies the query and sends it to the right source → the point: the raw user query is usually a bad search query.
Explain how HyDE works and what gap it bridges.
Hit these points: HyDE = Hypothetical Document Embeddings → the problem it fixes is the query↔document vocabulary gap — a question is worded nothing like the document that answers it, so its embedding sits far from the real source → HyDE has the LLM draft a plausible (hypothetical) answer and embeds that, not the question → even a partly-wrong draft has the right shape and lands near the real document in vector space → cost: one extra LLM draft plus an embed call → the answer-generating model itself is unchanged.
What does the small-to-big (parent-document) retriever do?
Hit these points: it decouples what you match on from what you return → you embed and match on small, precise chunks so similarity is sharp → but you hand the model back the larger parent document so it has the surrounding context to reason with → you get the precision of small chunks and the context of big ones, not a trade-off between them → cost is low — just an extra parent lookup after the match.
Define agentic RAG, Self-RAG, and Corrective RAG (CRAG) — what's the control loop in each?
Hit these points: baseline RAG = retrieve → assemble → LLM → answer, once; an agent = an LLM using tools → agentic RAG: the LLM decides whether / what / when to retrieve in a loop, issuing many searches and iterating → Self-RAG: the model retrieves on demand, then reflects — critiques its own draft for grounding and revises before answering → CRAG: grade the retrieved docs; if quality is low, fall back (e.g. web search) or re-retrieve before answering → common thread: retrieval becomes an action inside a loop, not a fixed pipeline step.
Your RAG recall is poor on multi-part questions like "compare our refund policy in the EU vs the US." Which pre-retrieval technique, and why?
Hit these points: this is two facts in one query, so a single embedding averages them and matches neither cleanly → reach for decomposition: split into "EU refund policy" and "US refund policy", retrieve each, then combine the chunks for the answer → optionally add multi-query per sub-question for recall → contrast: HyDE addresses the vocabulary gap, not the multi-fact problem; a bigger window doesn't help if neither fact was retrieved → verify by checking that chunks for both regions appear in the assembled context, not just the final answer.
Routing and step-back are both pre-retrieval. Contrast what they do and when each is the right call.
Hit these points:routing classifies the query and sends it to the right datasource / index / tool — code vs docs vs SQL — so you don't search the wrong corpus at all → step-back generalizes a too-specific question to pull broader grounding first, then retrieves for the specific one → routing is about where to look; step-back is about how broad to look → use routing when sources are heterogeneous; use step-back when the specific query is too narrow to match useful context → they compose: route first, then step-back within the chosen index → each is one cheap extra call, so weigh that latency against the recall it buys.
A teammate proposes rebuilding the whole RAG stack on GraphRAG because "it's the state of the art." How do you respond as the senior in the room?
Hit these points: first ask what questions users actually ask — pull the distribution → if most are local lookups ("what is the limit for X?"), flat top-k or small-to-big already answers them at a fraction of the cost → GraphRAG earns its keep only on global, theme-spanning "connect-the-dots" questions, and it carries real cost: building the entity graph, generating community summaries, keeping it fresh on every reindex → propose it as an added retriever, routed to for global queries, not a rip-and-replace → name the trade-off: build and maintenance complexity vs a capability you only need for a slice of traffic.
Explain contextual retrieval and when its added cost is worth paying.
Hit these points: the failure it targets is chunks that are ambiguous on their own — a chunk stripped of its heading/section is hard to match and hard for the model to use → contextual retrieval (Anthropic) prepends a short chunk-situating context before embedding, so each chunk self-describes → it also adds a BM25 lexical index alongside the vectors, catching exact IDs and keywords that embeddings miss → together they cut retrieval failures versus naive embedding-only chunking → the cost: re-embedding every chunk with its generated context plus maintaining a dual index → worth it when chunks are short/fragmentary and exact-match terms matter; skip it when chunks are already self-contained.
Walk through how you match retrieval structure to the shape of the question across a mixed query workload.
Hit these points: the principle: the shape of the question should pick the structure of the retriever, and flat chunk + top-k is the floor, not the ceiling → local fact in one chunk → flat top-k or small-to-big (match small, return parent for context) → "which section, then what" over large docs → hierarchical / summary retrieval → recency / tenant / access scope → metadata pre-filter before vector search → precision-critical, latency-tolerant → multi-vector / ColBERT late interaction → global theme spread across the whole corpus → GraphRAG community summaries → operationalize with a router that classifies the query and dispatches; the staff move is not picking one structure but composing several and routing by question shape.
When is an agentic / corrective loop worth its latency and cost, and when is a static pipeline the right default?
Hit these points: the loop wins exactly one axis — quality / robustness — and pays for it in latency (multiple round-trips), cost (grading + re-tries), and complexity (loops, budgets, stop conditions) → default static for simple, well-covered questions where one pass already answers correctly → reach for CRAG-style grade-and-fallback on the long tail where the index comes up short → reach for Self-RAG reflection where an ungrounded answer is high-stakes → decide from eval data: measure static-pipeline accuracy first, add the loop only on the slice where correctness is high-stakes and static accuracy is provably insufficient → you also need a reliable grading signal or the loop just burns calls → always cap attempts and define a stop condition so it terminates.
Make the case that pre-retrieval is the highest-leverage place to spend — then steelman the opposite.
For: recall lost at retrieval is unrecoverable downstream — no reranker reorders a document that was never in the candidate set, and no bigger model reasons over text it never received; the cheapest place to fix recall is the query, and HyDE / multi-query are a few extra calls. Steelman against: if the right doc is being retrieved but buried or context-poor, the bottleneck is retrieval structure (small-to-big, reranking) or assembly, not the query; and if questions are global-theme, no query rewrite helps — you need GraphRAG. Synthesis: attribute the failure by class first — is the right doc absent (pre-retrieval), present but mis-ranked (structure/rerank), or non-local (graph)? — then spend on the lever that matches, starting with the query because it's the common cause and the cheapest.
Design-round framework — drive any advanced-RAG prompt through these, out loud:
Characterize the question distribution — local facts vs global themes, single- vs multi-part, recency-sensitive.
Pre-retrieval — rewrite / multi-query / HyDE / decompose / step-back, and route to the right source.
Retrieval structure — match it to question shape: small-to-big, hierarchical, contextual, multi-vector, GraphRAG.
Rerank & assemble within budget — precision first, parents for context, order for primacy/recency.
Decide static vs loop — add CRAG grade-and-fallback / Self-RAG critique only where correctness justifies the cost.
Bound the loop — cap attempts, define stop conditions, set a latency/cost budget.
Design an advanced-RAG pipeline for a question type that flat top-k fails on — e.g. "summarize how our incident response process evolved over the last two years."
A strong answer covers: name why flat top-k fails — this is a global, multi-document synthesis question; no single chunk holds "how it evolved," so nearest-k returns scattered fragments → pre-retrieval: decompose into time-bucketed sub-questions and add metadata filtering on date so each retrieval is scoped → structure: layer GraphRAG or hierarchical summary retrieval so community/section summaries supply the cross-document themes, with small-to-big underneath for the specific facts that anchor each claim → assemble: rerank, then pack summaries plus a few grounding chunks within budget → consider an agentic loop that grades whether each time period is covered and re-retrieves the gaps before drafting → observability: log which periods and themes were retrieved; failure modes — missing year → flag the gap rather than smooth over it → name the trade-off: the graph/summary build and the loop add cost and latency, justified because flat retrieval simply can't answer this class.
Design retrieval for a corpus that must serve both local facts ("what's the retry limit?") and global synthesis ("what themes recur across all our incidents?").
A strong answer covers: start by recognizing two question shapes need two structures — don't force one retriever to do both → put a router up front that classifies the query as local-lookup vs global-theme → local path: small-to-big / flat top-k with metadata filtering — cheap, fast, one chunk answers it → global path: GraphRAG (or hierarchical summaries) so community summaries pre-aggregate relationships no single chunk holds → share one ingestion pipeline that produces both the chunk/parent index and the entity graph + summaries, so the corpus is indexed once into two structures → freshness: re-embed chunks and rebuild affected graph communities on change; the graph is the costlier thing to keep fresh, so name that → assemble within budget per path and rerank → observability: track hit-rate separately per path, since they fail differently → name the trade-off: maintaining a graph alongside the vector index is real cost and operational complexity, paid only because a meaningful slice of traffic is global synthesis that flat retrieval can't serve.
Was this lesson helpful?Thanks — noted.
These notes reflect my current understanding and are updated as I learn, build, and
discover better explanations.