Lesson 2 · Principal Track · Core → Senior

Context Selection & Retrieval

~14 min · 2 modules · how a system decides what to send the model — and the search machinery that finds it

Published

Context retrievalLexicalSemanticHybridSemantic search

Your bar: explain how a system decides which facts reach the model when it cannot send everything, walk a real query (“find the authentication bug”) through candidate generation → ranking → selection, and compare lexical, semantic, hybrid search plus ranking vs re-ranking by purpose, advantages, disadvantages, and failure modes. By the end you can answer:

given a fixed token budget, how do you find the smallest set that still contains the answer?

1

This lesson is one pipeline, then the engines that power it. Each chip is one thing it makes permanent:

You can’t send everything, so the system selects

cast a wide net (candidates), then rank them

pack the top into the token budget

finding candidates = lexical · semantic · hybrid · re-rank

1

Context Selection

Lesson 1 fixed the budget: the window is small and you don’t own all of it. Selection is the discipline of choosing the smallest set of text that still contains the answer — wide net, then ruthless trim.

The hero: a selection funnel

SELECTION FUNNEL — many in, a curated few out Corpus — every file · doc · record (millions of tokens) far larger than any window Candidate generation — cast a wide net lexical + semantic search · optimize RECALL (don't miss the right one) Ranking — score every candidate cheap score now · expensive re-rank optional Selection / packing fit top-K into budget Context window
Three stages, narrowing each time. Generate casts wide for recall; rank scores; select/pack fits the top into the budget — and slips in cheap structural context (signatures, not full bodies).1

Is Selection is the pipeline that turns a huge corpus into the curated few that fit the window: candidate generation, ranking, then packing within budget.

Why it exists The corpus dwarfs the window (Lesson 1). Sending everything is physically impossible and actively harmful — noise buries signal and costs more.

Like (world) A librarian who can carry ten books to your desk from a million-book stack: first pull a wide shelf, then narrow to the ten that answer your question.

Like (code) A WHERE + ORDER BY + LIMIT pipeline: filter to candidates, sort by a score, take the top N that fit the page.

Relevance signals — what each catches, what it misses

SignalWhat it catchesWhat it misses
Lexical matchExact terms, identifiers, error codesSynonyms & paraphrase (different words, same idea)
Semantic similarityMeaning, synonyms, “asks the same thing”Rare exact tokens (IDs, function names, codes)
Structural / graphImports, call sites, type/dependency linksConceptual matches with no code edge between them
RecencyThe just-changed file, the latest doc versionStable-but-correct old code that never changes
Authority / trustCanonical source, owned doc, signed recordThe right fact when it only lives in a low-trust place

No single signal is sufficient. Each is blind to a different class of right answer — which is exactly why hybrid retrieval (Module 2) exists.

“Relevant” means different things per system

SystemThe queryWhat “relevant” means
Billing platform“Why was account 4471 double-charged?”

The exact account records & ledger rows — lexical/ID match, authority. A “similar” account is worse than useless.

GitHub repository“Where do we validate the JWT?”

The right symbols plus their dependencies — semantic concept + structural call sites & imports.

Documentation portal“How do I rotate an API key?”

The right article at the right version — semantic match gated by recency/version, authority.

Walk-through: “Find the authentication bug”

Query: “find the authentication bug” 1 Parse entities “authentication” · “login” · “token” · “session” 2 Candidate generation lexical: auth/login/session symbols  +  semantic: the concept 3 Add structural context files importing the auth module · its call sites · recent changes 4 Rank candidates score by combined signals · best-looking first 5 Select top-K within budget full body of the prime suspects · signatures/docstrings of the rest 6 Assemble the window prompt + selected context + question, in budget, ordered If step 2 or 3 never surfaces the buggy file, no model — however large — can find the bug. Selection quality decides whether the model ever sees the answer.
The whole pipeline can be perfect downstream and still fail if generation misses the file. Recall first — you cannot rank or pack what you never retrieved.1
✗ “Just send the model more files to be safe.”

✓ More irrelevant files lower signal, raise cost, and push the right lines into the “lost in the middle” zone. Smaller-but-correct wins.

✗ “Selection is just one search query.”

✓ It’s a pipeline: generate (recall) → rank (order) → pack (budget). Each stage can be the bottleneck independently.

📥 Memory rule: You can’t send everything — selection is choosing the smallest set that still contains the answer. Cast wide for recall, then trim to fit.

Memory check
  • Name the three stages of the selection funnel. → candidate generation (recall) → ranking (order) → selection/packing (fit budget)
  • Which stage, if it fails, can't be rescued downstream? → candidate generation — you can't rank or pack what was never retrieved
  • What cheap context do you pack instead of full bodies? → signatures & docstrings of related code, plus structural links (imports, call sites)

Your code assistant keeps missing the actual buggy file. Where in the selection pipeline do you look first, and how do you prove it?

Hit these points: instrument the pipeline per stage, don’t guess → first check candidate generation recall: was the buggy file ever in the candidate set? if no, ranking/packing are irrelevant → if recall is the gap, add the missing signal — usually structural (imports/call sites) or lexical for an exact symbol the embedding model washed out → if recall is fine but the file ranked low, fix ranking / add a re-rank → if it ranked high but got dropped, it’s a packing/budget problem → measure retrieval hit-rate as a first-class metric, not just end-answer quality.

2

Retrieval

Module 1 said “cast a wide net, then rank.” This is the machinery that does it: lexical finds exact, semantic finds meaning, hybrid finds both, and re-ranking sharpens the top. Each one has a job it does well and a case where it falls down.

The hero: a two-stage retrieval pipeline

TWO-STAGE RETRIEVAL — recall first, then precision Query STAGE 1 — recall · cheap · wide Lexical (BM25) exact terms & identifiers Semantic (vector) meaning & synonyms Fuse (RRF / weighted) one candidate list STAGE 2 — precision · expensive · top-N only Cross-encoder re-rank re-score top-N jointly with query top-k → window
Stage 1 (lexical + semantic, fused) maximizes recall cheaply across many candidates. Stage 2 (cross-encoder) maximizes precision on the top-N only — it cannot recover anything stage 1 missed.2

The four engines, side by side

TechniqueHow it scoresBest atSignature failure
Lexical (BM25)TF-IDF / BM25 on exact termsIdentifiers, error codes, exact stringsVocabulary mismatch → total miss
Semantic (vector)Cosine of embeddings via ANNMeaning, synonyms, paraphrasePlausible-but-wrong; misses exact tokens
HybridFuse both (RRF / weighted)

Exact and semantic together

Bounded by both stages’ recall
Re-rankCross-encoder, query+doc jointlyOrdering the final top resultsCan’t recover a missed document

Search (lexical / keyword / BM25)

Is Exact-term matching: score documents by how well their words match the query words, weighted by term frequency and rarity (TF-IDF / BM25). 3

Why it exists Some queries are the exact tokens — an error code, a function name, an account ID. For those, meaning is irrelevant; the literal string is the answer.

Advantages Precise on exact terms & identifiers, fast, cheap, interpretable, needs no model — you can read why it matched.

Failure mode No synonyms or semantics: if the user phrases it differently than the doc (vocabulary mismatch), it returns a total miss.

Semantic search (embeddings / vector)

Is Encode query and chunks into vectors; rank by cosine similarity using an approximate-nearest-neighbour index (e.g. HNSW). 4

Why it exists Users rarely use the doc’s exact words. Embeddings match on meaning, so a paraphrase still finds the right passage.

Advantages Captures meaning, synonyms, paraphrase; robust to wording; finds the right idea even with zero shared words.

Failure mode Returns “topically similar but wrong”; weak on rare exact tokens — it can’t reliably match an exact error string or ID.

Hybrid search

Is Run lexical and semantic together, then fuse the two result lists into one ranking — commonly Reciprocal Rank Fusion or a weighted blend. 5

Why it exists The two engines fail on opposite cases: lexical whiffs on paraphrase, semantic whiffs on exact tokens. Run both and one usually catches what the other drops.

Advantages Covers both exact and meaning, so fewer right documents slip through. Anthropic’s Contextual Retrieval pairs embeddings with BM25 for exactly this reason. 2

Failure mode More infra and complexity, fusion weights to tune; still bounded by the recall of both first-stage retrievers.

Ranking vs re-ranking — two different jobs

Ranking (stage 1) Re-ranking (stage 2) cheap score · many candidates optimize RECALL “don't miss the right doc” BM25 score · cosine score applied to the whole net expensive score · top-N only optimize PRECISION “put the best at the top” cross-encoder · query+doc jointly large precision lift on the top top-N
First stage is wide and cheap to buy recall; second stage is narrow and expensive to buy precision. The cross-encoder reads query and document together, so it scores relevance far more accurately — but only on what stage 1 handed it.4

Ranking — is Order candidates by a cheap score (BM25, cosine). First stage. Purpose: optimize recall so the right document is somewhere in the list.

Re-ranking — is A second, expensive, accurate pass (cross-encoder) that re-scores the top-N jointly with the query. Purpose: optimize precision at the top.

Re-rank advantage Large precision lift on the final top-k the model actually sees — the items most likely to be packed into the window.

Re-rank failure Latency and cost; runs on top-N only; cannot recover a document the first stage already missed.

✗ “Embeddings replaced keyword search.”

✓ Embeddings are weak on exact tokens (error codes, IDs, function names). Hybrid keeps BM25 precisely for those cases.

✗ “A great re-ranker fixes bad retrieval.”

✓ Re-ranking only reorders what stage 1 returned. If recall missed the doc, no re-ranker can surface it — fix recall first.

📥 Memory rule: First-stage retrieval buys recall; re-ranking buys precision. Lexical finds exact, semantic finds meaning, hybrid finds both.

Memory check
  • Which engine matches an exact error code, and which a paraphrase? → lexical/BM25 for the exact code; semantic/vector for the paraphrase
  • What does the first stage optimize for vs the re-rank stage? → first stage = recall (don't miss it); re-rank = precision (best at the top)
  • Why can't a re-ranker save bad retrieval? → it only re-scores the top-N it was given; a missed document never reaches it

Users complain semantic search returns “close but wrong” results and sometimes whiffs on exact error codes. What’s your fix and why?

Hit these points: name the root cause — embeddings score topical similarity, so they return plausible-but-wrong and wash out rare exact tokens like error codes → add hybrid: run BM25 alongside the vector search and fuse (RRF) so exact strings are caught by lexical and meaning by semantic → add a cross-encoder re-rank over the fused top-N to push the genuinely best result to the top (precision) → frame the split: hybrid fixes recall, re-rank fixes precision → verify with retrieval hit-rate on a labelled set, not just end-answer vibes; tune fusion weights against that set.

Retrieval practice — test the two modules

Q1. The three stages of the selection funnel, in order, are…

Q2. If the buggy file is never returned by candidate generation, the result is…

Q3. Lexical search (BM25) is the right tool, and semantic search the wrong one, when the query is…

Q4. The signature failure mode of semantic (vector) search is that it…

Q5. Re-ranking (a cross-encoder over the top-N) primarily buys you…

Interview — pick your bar

Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).

Name the three stages of the selection pipeline, in order, and what each one does.
Hit these points: candidate generation → cast a wide net to pull anything that might be relevant → ranking → score those candidates so the best ones rise → selection / packing → fit the top of the list into the token budget. The whole point is choosing the smallest set that still contains the answer when you can't send everything.
What does lexical / BM25 search actually match on, and what is it good at?
Hit these points: it scores documents by overlap of the exact terms, weighted by term frequency and rarity (TF-IDF / BM25) → meaning plays no part, only the literal strings → strong on identifiers, error codes, function names, exact IDs → fast, cheap, interpretable, no model required — you can read why it matched.
What does semantic / embedding search match on, and why use it?
Hit these points: it encodes query and chunks into vectors and ranks by similarity (cosine over an ANN index) → matches on meaning, so synonyms and paraphrase still find the right passage even with zero shared words → exists because users rarely use the document's exact wording → robust to phrasing, weak on rare exact tokens.
What is hybrid search, and what does fusion (e.g. RRF) do in it?
Hit these points: run lexical and semantic together, then merge their two result lists into one ranking → fusion (Reciprocal Rank Fusion or a weighted blend) combines the per-engine ranks into a single ordered list → the reason: the two engines fail on opposite cases, so one usually catches what the other drops → gives you exact-match and meaning-match coverage at once.
Users say semantic search returns “close but wrong” results and sometimes whiffs on exact error codes. What's your fix and why?
Hit these points: name the root cause — embeddings score topical similarity, so they return plausible-but-wrong and wash out rare exact tokens like error codes → add hybrid: run BM25 alongside the vector search and fuse (RRF) so exact strings are caught by lexical and meaning by semantic → add a cross-encoder re-rank over the fused top-N to push the genuinely best result up (precision) → frame the split: hybrid fixes recall, re-rank fixes precision → verify with retrieval hit-rate on a labelled set, not end-answer vibes; tune fusion weights against that set.
Your code assistant keeps missing the actual buggy file. Where in the pipeline do you look first, and how do you prove it?
Hit these points: instrument per stage, don't guess → first check candidate-generation recall: was the buggy file ever in the candidate set? if no, ranking and packing are irrelevant → if recall is the gap, add the missing signal — usually structural (imports / call sites) or lexical for an exact symbol the embedding washed out → if recall is fine but it ranked low, fix ranking / add a re-rank → if it ranked high but got dropped, it's a packing/budget problem → track retrieval hit-rate as a first-class metric, not just end-answer quality.
Distinguish ranking from re-ranking. When is a cross-encoder worth its cost, and when is it a waste?
Hit these points: ranking = first stage, cheap score (BM25 / cosine) over many candidates, optimizing recall — “don't miss it” → re-ranking = second stage, expensive cross-encoder reading query + document jointly over the top-N, optimizing precision — “best at the top” → worth it when the final top-k reaching the window must be high-precision and first-stage ordering is noisy → a waste if recall is the real gap — it can't recover a missed doc → bound cost by running it on top-N only and caching; prove the precision lift on a labelled set before keeping it.
Someone proposes “just embed everything and drop BM25 — embeddings are strictly better.” Push back.
Hit these points: embeddings are weak on exact tokens — error codes, IDs, function names get washed into topical neighbours → for a query that is the literal string, meaning is irrelevant and lexical is exactly right → the two engines have opposite blind spots, so dropping BM25 trades one failure mode for another → hybrid keeps BM25 precisely for those cases; the small extra infra is cheaper than silently missing every ID lookup → decide with hit-rate on a query mix that includes exact-token queries, not a vibe.
Design a retrieval strategy when you don't yet know the query mix — some lookups are exact IDs, some are vague conceptual questions, and the corpus keeps changing.
Hit these points: start from recall first: a missed doc can't be ranked or packed, so over-invest stage 1 → default to hybrid (BM25 + vector, fused) so exact-token and conceptual queries are both covered without knowing the split up front → add a cross-encoder re-rank on the fused top-N for precision where it matters → layer signals the domain needs — structural for code, recency/version for changing docs, authority for canonical sources → keep re-indexing fresh as the corpus changes → instrument hit-rate by query type, learn the real mix, then tune fusion weights and decide where re-rank earns its latency.
“Relevant” means different things per system. Contrast a billing platform, a code repo, and a docs portal — and say how that changes your retrieval.
Hit these points: relevance is set by the question's intent, not one universal score → billing: the exact account / ledger rows — lexical/ID match + authority; a “similar” account is actively harmful → repo: the right symbols plus their dependencies — semantic concept + structural call sites and imports → docs: the right article at the right version — semantic gated by recency/version + authority → so you pick and weight signals to the domain's notion of relevance, rather than shipping one generic retriever everywhere.
You can raise recall (wider net) or precision (aggressive re-rank + tighter packing), but not both for free. How do you reason about the trade-off?
Hit these points: the two failures cost differently — a recall miss means the answer never reaches the model (unrecoverable), a precision miss means noise the model must see past → so buy recall cheaply and wide in stage 1, then buy precision narrowly in stage 2 where it's affordable → watch the packing tension: more candidates raise recall but bury the right lines in the “lost in the middle” zone and cost more → tie the dial to the domain's cost of a miss (billing error vs a fuzzy docs answer) → measure both hit-rate and top-k precision on a labelled set and move the knob deliberately, not by feel.
Design-round framework — narrate retrieval design in this order so the interviewer hears recall-first thinking:
  1. Clarify the query mix & the cost of a miss (exact-ID lookups vs vague concepts; how bad is a wrong-but-plausible answer?).
  2. Define “relevant” for this domain and pick signals (lexical, semantic, structural, recency, authority).
  3. Stage 1 for recall: hybrid candidate generation (BM25 + vector) and how you fuse the lists.
  4. Stage 2 for precision: whether a cross-encoder re-rank on the top-N earns its latency.
  5. Packing within the token budget: top-k, full bodies vs signatures/docstrings, ordering.
  6. Freshness & failure modes: re-indexing the changing corpus, vocabulary mismatch, plausible-but-wrong.
  7. Measure: retrieval hit-rate and top-k precision on a labelled set; tune weights against it.
Design the retrieval layer for a documentation Q&A bot over versioned, frequently-updated docs.
A strong answer covers: the query mix is mostly conceptual “how do I…” questions, so semantic carries weight — but keep BM25 for exact API names and error strings → gate on recency / version so an old article doesn't beat the current one, and on authority for canonical pages → hybrid stage 1, fuse with RRF, re-rank the top-N for precision since users read only the first answer → pack the best article(s) within budget; prefer the right section over dumping whole pages → re-index on every doc change so freshness holds → call the failure modes: plausible-but-wrong from semantic, vocabulary mismatch from lexical → measure hit-rate on labelled questions, not end-answer vibes.
Design retrieval for an internal-search feature across code, tickets, and wiki docs in one box.
A strong answer covers: the corpus is heterogeneous, so “relevant” differs per source — code wants structural signals (imports, call sites) plus lexical for exact symbols; tickets/wiki lean semantic → run hybrid per source and fuse, or fuse across sources, so exact-token and conceptual queries both land → recall first: a missed file can't be ranked or packed → re-rank the fused top-N for precision; weight by recency where it matters (latest ticket, newest doc) and authority for canonical sources → pack within budget — signatures/docstrings for code, snippets for docs → keep indexes fresh as all three sources change → instrument hit-rate by source and query type, then tune fusion weights and decide where re-rank pays for its latency.
Sources
Anthropic, "Effective context engineering for AI agents" — anthropic.com/engineering. Context as a finite, curated resource; selection over dumping; recall before packing.
Anthropic, "Introducing Contextual Retrieval" — anthropic.com/news/contextual-retrieval. Hybrid embeddings + BM25 with re-ranking to reduce retrieval failures (figures illustrative).
Robertson & Zaragoza, 2009, "The Probabilistic Relevance Framework: BM25 and Beyond." Lexical / keyword retrieval and TF-IDF/BM25 scoring.
Karpukhin et al., 2020, "Dense Passage Retrieval for Open-Domain QA" — arXiv:2004.04906. Dense (embedding) retrieval; dense vs lexical trade-offs; cross-encoder re-ranking context.
Cormack et al., 2009, "Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods" — SIGIR. Fusing multiple rankings for hybrid search.
Was this lesson helpful? Thanks — noted.

These notes reflect my current understanding and are updated as I learn, build, and discover better explanations.