Lesson 6 · Principal Track · Senior → Staff
Retrieval Evaluation & Observability
Published
Your bar: measure retrieval separately from generation, name and apply the retrieval metrics (recall@k, precision@k, MRR, nDCG) and the RAGAS-style answer-quality metrics (faithfulness, answer relevance, context precision/recall), and design the eval loop — golden set, offline regression, online signals, and tracing. By the end you can answer:
the bot was wrong — was it retrieval, generation, or the eval set that failed, and how do you know?
1The whole lesson is one loop: measure the two halves, then trace to the failing one. Each chip is one thing it makes permanent:
Generation can only use what retrieval surfaced
so grade retrieval and the answer separately
regress on a golden set offline; watch signals online
and trace every call to see which stage failed
1
Retrieval Metrics — Did We Even Fetch the Right Thing?
Generation can only reason over what retrieval surfaced. So measure retrieval on its own, before the answer. If the right doc never made the top-k, no model fixes it — you’re optimizing the wrong half.
Why retrieval gets its own scoreboard
The metrics, pinned down
| Metric | What it measures | When you care |
|---|---|---|
| recall@k | Is at least one relevant doc in the top-k? (Did we fetch it at all.) | The most important RAG metric; tune first-stage retrieval to maximize it. |
| precision@k | What fraction of the top-k are actually relevant? | When noise crowds the window; junk chunks waste budget & distract. |
| MRR | Mean reciprocal rank — how high the first relevant result sits. | When position matters and one good hit is enough (e.g. lookup). |
| nDCG | Graded relevance, position-discounted — rewards best-at-the-top. | Judging the reranker / final order, not just presence. |
| hit-rate | Share of queries with ≥1 relevant doc retrieved (recall@k averaged). | A single headline number for “are we fetching the answer?” |
Is recall@k = “did the relevant doc make the top-k at all”; precision@k = “how clean is the top-k.” Recall is about presence, precision is about purity.
Why it exists The two stages of Lesson 2 optimize different things: first-stage retrieval buys recall@k, the reranker buys precision / nDCG@top-n. Separate metrics let you tune each stage independently.
Like (world) A library search: recall = “is the right book somewhere in the cart you wheeled out”; nDCG = “is it on top of the cart, not buried under twenty wrong ones.”
Like (code) Recall@k is like a test asserting the expected row is in the result set; nDCG is asserting it’s ordered first. Different assertions, different bugs.
✗ “A high end-to-end answer score means retrieval is fine.”
✓ It can mask a low recall@k — the model bluffs from parametric memory on easy queries while quietly missing the hard ones. Measure retrieval directly.
✗ “Optimize precision@k everywhere; fewer chunks is cleaner.”
✓ First-stage you want recall — don’t drop the answer to look tidy. Buy precision later with a reranker (nDCG), on top of high recall.
📥 Memory rule: recall@k = did we fetch it at all. Everything downstream is wasted if recall is low — so it’s the first metric you optimize.
Memory check
- Which metric does first-stage retrieval optimize, and why is it the most important? → recall@k — if the relevant doc isn't in the top-k, nothing downstream can recover it
- recall@k vs precision@k in one line each? → recall = is a relevant doc present in top-k; precision = fraction of top-k that are relevant
- What does nDCG add over recall@k? → graded, position-discounted relevance — rewards putting the best doc at the top (judging the reranker)
Your RAG answers look 90% correct on a spot-check, but users complain on hard queries. Which retrieval metric do you pull first, and what would it reveal?
Hit these points: compute recall@k per query on a labelled set, not aggregate answer accuracy → the spot-check is biased toward easy queries the model can answer from parametric memory → low recall@k on the hard slice means the relevant doc never reached the window — a retrieval miss, not a reasoning miss → segment recall by query type / length to find where first-stage drops it → only after recall is high do precision@k and nDCG (reranker order) become the lever worth pulling.
2
RAG Answer-Quality Metrics — Beyond Retrieval
Once retrieval is good, grade the generation against the retrieved context. Four RAGAS-style metrics answer three questions: is the answer grounded, is it relevant, and is the context that fed it clean and complete?
The 2×2 that tells you which half to fix
The four RAGAS-style metrics
| Metric | Question it answers | What it catches | Needs |
|---|---|---|---|
| Faithfulness / groundedness | Is every claim supported by the retrieved context? | Hallucination — invented or unsupported facts. | answer + context |
| Answer relevance | Does the answer actually address the question? | Off-topic, evasive, or padded answers. | answer + question |
| Context precision | Are the retrieved chunks relevant and well-ranked? | Noisy / mis-ordered context wasting the window. | context + question |
| Context recall | Did retrieval capture all the info the ideal answer needs? | Missing facts — incomplete context. | context + ground truth |
Scoring them at scale: LLM-as-judge
Is Faithfulness checks the answer against the context (hallucination); answer relevance checks it against the question (on-topic). Two independent failure axes.
Why it exists Retrieval metrics say nothing about the text the model wrote. You can fetch perfectly and still hallucinate, or answer fluently but off-topic — so the generation half needs its own scoreboard.
Like (world) Grading an essay: faithfulness = “does it stay true to the cited sources”; relevance = “does it answer the actual prompt.” A footnoted essay can still miss the question.
Like (code) Faithfulness is an integration test (answer vs the real context); relevance is an acceptance test (answer vs the user’s intent). Both must pass to ship.
✗ “If it cites a source, the answer is faithful.”
✓ A citation can be irrelevant or mis-summarized. Faithfulness checks each claim against the context, not whether a citation marker exists.
✗ “LLM-as-judge scores are objective ground truth.”
✓ Judges carry bias (verbosity, position, self-preference) and drift. Calibrate against human labels on a sample; use the judge to scale, not to define truth.
📥 Memory rule: Faithful but irrelevant, or relevant but unfaithful — measure both halves or you’ll fix the wrong one.
Memory check
- Which metric detects hallucination, and against what does it check? → faithfulness / groundedness — every claim must be supported by the retrieved context
- Which of the four metrics needs a ground-truth answer? → context recall — you can only tell if all needed info was captured against an ideal answer
- Name two caveats of LLM-as-judge. → bias (verbosity/position/self-preference) and the need for calibration vs human labels
Faithfulness scores are high but users still report wrong answers. What’s happening, and which metrics confirm it?
Hit these points: faithful means “grounded in the retrieved context” — but the context can be wrong or incomplete, so the model is faithfully repeating junk → this is the bottom-left quadrant: a retrieval bug masquerading as a good answer → check context recall (did retrieval capture all the ideal answer needs) and context precision (are chunks relevant + well-ranked) → also check answer relevance for off-topic-but-grounded replies → conclusion: faithfulness alone is necessary, not sufficient — pair it with context-side metrics so you don’t “fix” generation when retrieval is the cause.
3
Eval Sets, Offline vs Online, and Observability
Metrics need something to run against. Build a golden set, regress on it offline before deploy, watch online signals after, and trace every call so a vague “it was wrong” becomes “doc X wasn’t retrieved.”
The golden set, and the two eval loops it powers
What a trace must capture
Is A golden set is labelled triples — (query, relevant doc IDs, ideal answer) — versioned in source control. Offline eval replays it; online eval reads production behaviour signals.
Why it exists Offline catches regressions deterministically before deploy; online catches the long tail offline never imagined. Tracing connects a complaint to the exact failing stage.
Like (world) Offline = a pre-flight checklist run every time; online = pilots reporting real turbulence; the trace = the black-box recorder that says which system failed.
Like (code) Offline eval is your CI test suite (regression gate); online signals are production metrics/alerts; tracing is distributed tracing across the retrieval→generation spans.
✗ “Offline eval on the golden set is enough to ship safely.”
✓ The golden set only covers what you imagined. Online signals (thumbs, escalation, rephrase rate) find the real-world misses you must curate back in.
✗ “Log the final answer; that’s enough to debug.”
✓ The answer alone can’t tell retrieval from generation failure. Log doc IDs+scores, packed chunks, and the assembled context — or the bug stays silent.
📥 Memory rule: Offline eval stops regressions; online signals find what offline missed; tracing tells you which stage failed.
Memory check
- What three things make up a golden-set entry? → the query, the relevant doc IDs, and the ideal answer (versioned)
- Offline vs online eval in one line each? → offline = regression on the golden set in CI pre-deploy; online = production signals (thumbs, escalation, citation-click, rephrase)
- What must a trace log to localize a failure? → query, retrieved doc IDs+scores, chunks packed into the window, the assembled context, and the answer
”The bot gave a wrong answer last Tuesday.” With tracing in place, how do you go from that complaint to a root cause?
Hit these points: pull the trace by request ID / timestamp → check layer 2 — was the relevant doc in the retrieved IDs+scores? if absent, it’s a recall miss (fix first-stage retrieval) → if present but low-ranked or dropped at packing (layer 3), it’s a ranking/budget bug (add rerank / adjust budget) → if it was in the assembled context (layer 4) yet the answer ignored it, it’s a generation/faithfulness bug (prompt/model) → add the failing query to the golden set so offline eval catches the regression next time → the trace is what turns “the bot was wrong” into a specific, fixable stage.
Retrieval practice — test the three modules
Q1. Which metric is the most important for first-stage RAG retrieval, and what does it measure?
Q2. Why must you measure retrieval separately from the generated answer?
Q3. Which RAGAS-style metric detects hallucination, and against what does it check?
Q4. An answer scores high on faithfulness but is still wrong. The most likely cause is…
Q5. What turns "the bot was wrong" into "document X wasn't retrieved"?
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
Define recall@k and precision@k, and say which one matters most for first-stage RAG retrieval.
Hit these points: recall@k = is at least one relevant doc present in the top-k — about presence → precision@k = what fraction of the top-k are actually relevant — about purity → recall@k is the most important RAG metric because nothing downstream can use a doc that wasn't retrieved → so you tune first-stage retrieval to maximize recall@k first, then buy precision later with a reranker → one line: recall = "did we fetch it at all," precision = "how clean is the top-k."
What do MRR and nDCG measure, and when do you reach for each?
Hit these points: MRR (mean reciprocal rank) = how high the first relevant result sits, averaged over queries — reach for it when one good hit is enough, like a lookup → nDCG = graded, position-discounted relevance — rewards putting the best doc at the top → nDCG is what you use to judge the reranker / final ordering, not just whether the doc is present → recall@k says "is it in the cart"; nDCG says "is it on top of the cart."
Name the four RAGAS-style answer-quality metrics and the one question each answers.
Hit these points: faithfulness/groundedness — is every claim supported by the retrieved context? (catches hallucination) → answer relevance — does the answer actually address the question? (catches off-topic/evasive) → context precision — are the retrieved chunks relevant and well-ranked? (catches noisy/mis-ordered context) → context recall — did retrieval capture all the info the ideal answer needs? (catches missing facts; needs ground truth) → first two grade generation, last two grade retrieval.
What is a golden set, and what's the difference between offline and online eval?
Hit these points: a golden set = labelled triples — (query, relevant doc IDs, ideal answer) — versioned in source control → offline eval replays that set in CI before deploy and reports recall@k / nDCG / faithfulness vs the last release — a deterministic regression gate → online eval reads production behaviour after deploy: thumbs, deflection/escalation, citation-clicks, follow-up/rephrase rate → offline catches regressions you anticipated; online catches the long tail you didn't → you curate online misses back into the golden set.
Your RAG answers look ~90% correct on a spot-check, but users complain on hard queries. Which metric do you pull first, and what would it reveal?
Hit these points: compute recall@k per query on a labelled set, not aggregate answer accuracy → the spot-check is biased toward easy queries the model can answer from parametric memory, so it hides the failures → low recall@k on the hard slice means the relevant doc never reached the window — a retrieval miss, not a reasoning miss → segment recall by query type / length to find where first-stage drops it → only after recall is high do precision@k and nDCG (reranker order) become the lever worth pulling.
Faithfulness scores are high but users still report wrong answers. What's happening, and which metrics confirm it?
Hit these points: faithful means "grounded in the retrieved context" — but the context can be wrong or incomplete, so the model is faithfully repeating junk → this is the grounded-but-wrong quadrant: a retrieval bug masquerading as a good answer → check context recall (did retrieval capture all the ideal answer needs) and context precision (are chunks relevant + well-ranked) → also check answer relevance for off-topic-but-grounded replies → conclusion: faithfulness is necessary, not sufficient — pair it with context-side metrics so you don't "fix" generation when retrieval is the cause.
"Just optimize precision@k everywhere — fewer, cleaner chunks." Where does that advice mislead you?
Hit these points: first-stage you want recall, not tidiness — dropping chunks to raise precision can drop the one relevant doc and cap the answer → recall@k is the ceiling; precision can't recover a doc that's no longer in the set → the right ordering is recall first (first-stage retrieval), then buy precision/nDCG with a reranker on top of high recall → also: aggregate precision can look great while a hard slice silently misses — always segment → the metric to optimize depends on the stage; conflating the two stages is the trap.
You want to grade faithfulness on every production call. How do you do it, and what are the risks?
Hit these points: use LLM-as-judge — feed answer + retrieved context, score each claim's support 0–1 → it scales where human labelling can't → risks: judge bias (verbosity, position, self-preference), drift across model versions, and cost per call → mitigate by calibrating against a human-labelled sample, pinning the prompt + judge version, and spot-auditing disagreements → frame it: the judge is a calibrated proxy to scale grading, not the definition of truth → track judge-vs-human agreement as its own metric so you notice drift.
"The bot gave a wrong answer last Tuesday." With tracing in place, how do you go from that complaint to a specific stage?
Hit these points: pull the trace by request ID / timestamp → check the retrieved IDs+scores — was the relevant doc fetched at all? if absent, it's a recall miss (fix first-stage retrieval) → if present but low-ranked or dropped at packing, it's a ranking/budget bug (add rerank / adjust budget) → if it made the assembled context yet the answer ignored it, it's a generation/faithfulness bug (prompt/model) → add the failing query to the golden set so offline eval regresses on it → the trace is what turns "the bot was wrong" into a specific, fixable stage — without it the context bug is silent.
Offline eval is green but users still complain. What does that tell you, and what do you change?
Hit these points: green offline only means you pass the cases you wrote — the golden set has coverage gaps → the complaints are the long tail offline never modelled → lean on online signals (thumbs-down, escalation, rephrase/follow-up rate) to surface failing real queries → pull their traces to localize the failing stage (recall vs ranking vs generation) → curate those queries (with correct doc IDs + ideal answers) back into the golden set so the next offline run regresses on them → the eval set is a living artifact, not a one-time deliverable.
A teammate proposes a single end-to-end "answer correctness" score as the one metric to rule them all. Make the principal-level call.
Hit these points: one number for the whole pipeline hides where it broke — you can't tell a retrieval miss from a hallucination, so you'll fix the wrong half → a high end-to-end score can mask low recall@k: the model bluffs from parametric memory on easy queries while quietly missing the hard ones → the durable design measures the two halves separately (retrieval metrics + RAGAS-style answer metrics) plus per-call tracing to attribute failures → keep a headline end-to-end number for trend-watching, but never as the only signal or the debugging tool → the principal move: instrument by stage, decide by attribution, not by a single vanity metric.
Design-round framework — drive any eval/observability prompt through these, out loud:
- Clarify scope: measure retrieval and generation separately; what does "good" mean (recall@k target, faithfulness floor)?
- Golden set — (query, relevant doc IDs, ideal answer), seeded by synthetic generation + human curation, versioned in source control.
- Offline eval in CI — replay the set, report recall@k / nDCG / faithfulness vs last release, gate deploys on regression.
- Online signals — thumbs, deflection/escalation, citation-clicks, follow-up/rephrase rate; find what offline missed.
- Tracing — log per call: query, retrieved IDs+scores, packed chunks, assembled context, answer.
- Scaled grading — LLM-as-judge for faithfulness/relevance, calibrated against a human-labelled sample.
- Close the loop & guardrails — curate online failures back into the golden set; alert on recall/faithfulness drops.
Design the eval and observability for a production RAG support assistant from zero. What do you build first and why?
A strong answer covers: build the golden set first — (query, relevant doc IDs, ideal answer), seeded by synthetic generation + human curation, versioned so it diffs in review → wire offline eval into CI: replay the set and report recall@k / nDCG for retrieval and faithfulness / answer relevance / context precision-recall for generation, gating deploys on regression vs the last release → instrument tracing from day one — query, retrieved IDs+scores, packed chunks, assembled context, answer — so any failure maps to a stage → after launch add online signals (thumbs, deflection/escalation, citation-clicks, rephrase rate) and curate the misses back into the golden set → scale grading with an LLM-as-judge calibrated against human labels, tracking judge-vs-human agreement → name the trade-off: offline is deterministic but only covers what you imagined; online is real but lagging and noisy — you need both, plus tracing to connect a complaint to a cause.
Design a golden-set plus CI gate for retrieval specifically. How do you build it, keep it honest, and decide when to block a deploy?
A strong answer covers: start from labelled triples — for retrieval the load-bearing part is (query, relevant doc IDs); seed queries from real production logs + synthetic generation, then have humans verify the relevant-doc labels → version it in source control so additions/edits are reviewable and the set diffs over time → the CI job replays every query through first-stage retrieval and computes recall@k (the gate metric) plus nDCG for ordering, comparing against the last release → block the deploy on a recall@k regression beyond a small tolerance, and on per-slice drops (segment by query type/length so an aggregate average can't hide a hard slice cratering) → keep it honest: refresh labels when the corpus changes (a "relevant" doc can be deleted/edited), guard against the set going stale, and avoid leakage where the index was tuned to the eval queries → feed online recall misses back in so the gate keeps catching real failures → trade-off: a tight tolerance catches more regressions but creates flaky/noisy gates; a loose one ships silent recall drops — pick the threshold from the cost of a missed answer in this product.
Was this lesson helpful? Thanks — noted.
These notes reflect my current understanding and are updated as I learn, build, and discover better explanations.