Reference · Context Engineering

Context engineering — one-page cheat sheet

Context engineeringSemantic searchRetrieval augmented generationContext windowWorking memory

Published Jun 21, 2026

For the principal chair · print me · pairs with Lesson 1 & the glossary

The one sentence: the model is the constant, context is the variable — what you retrieve and assemble into a fixed window decides how smart the system looks, not which LLM you bought.

The spine (mental models)

Context = the new database query — retrieval/assembly plays SQL’s role: it decides what the model ever sees.
The window is RAM — a fixed token budget you allocate, not a pipe to your data. Bigger ≠ solved.
Selection → retrieval → rerank — turn a huge corpus into the curated few that fit.
Recall first, precision second — first stage casts wide (don’t miss it); rerank surfaces the best.
Chunk on meaning + overlap — split on structure; overlap so boundary facts survive.
RAG = retrieval + assembly + LLM — ground the answer in fetched facts, not frozen weights.
Memory vs knowledge base — memory = what I learned from us; KB = what’s true in general.
Compress for long runs — summarize the old, keep the recent & the IDs verbatim.
Match retrieval to the data type — exact→structured, fuzzy→hybrid, volatile→live, long→compress.
Most AI bugs are context bugs — suspect the input before blaming the model.

What competes for the window

System prompt	Instructions — fixed overhead, paid every call.
Tool defs	Function schemas — grows with #tools.
History	Conversation — grows every turn (silent eater).
Retrieved	Docs/code/facts for THIS turn — the part you engineer.
Question	The user’s actual query.
Answer	Reserved room the output needs too.

The two diagrams

AI PIPELINE            CLASSIC SOFTWARE
Question               Request
↓        same role     ↓
Retrieval  ◀───────▶   SQL query   (the new query)
↓                      ↓
Assembly (pack window) DB  source of truth
↓                      ↓
LLM   fixed/commodity  Response
↓
Answer
the LLM only ever sees what Retrieval surfaced

TWO-STAGE RETRIEVAL — recall first, precision second
query
│
▼  STAGE 1 · cheap · wide  (optimize RECALL)
┌──────────────┬──────────────┐
│ Lexical/BM25 │ Vector/embed │
│ exact terms  │ meaning      │
└──────┬───────┴──────┬───────┘
└──▶ fuse (RRF) ◀──┘  one candidate list
│
▼  STAGE 2 · expensive · top-N (PRECISION)
cross-encoder rerank
│
▼
top-k → window
stage 2 cannot recover what stage 1 missed

Chunking rules

Split on structure (function/class/heading bounds), not character count.
Add overlap so a fact spanning a boundary survives whole in one chunk.
Beware fragmentation — a split fact lives in neither chunk and is unretrievable.
Contextual chunking — prepend a short doc/section summary so each chunk self-describes.
Each chunk must be complete (answers alone) and retrievable (matches its questions).

Retrieval techniques

Technique	Good at	Weakness	Failure mode
Lexical / BM25	Exact terms, identifiers, error codes, IDs	No synonyms / semantics	Vocabulary mismatch → total miss
Semantic / embeddings	Meaning, synonyms, paraphrase	Misses rare exact tokens (IDs, fn names)	Plausible-but-wrong; topically similar
Hybrid	Exact and meaning together (fuse via RRF)	More infra; fusion weights to tune	Bounded by both stages’ recall
Re-rank	Ordering the final top-k (cross-encoder)	Latency & cost; top-N only	Can’t recover a missed document

The five failure modes

Mode	Symptom	Root cause	Fix
Missing	”I don’t know” / hallucinated filler	Recall gap; not in KB; chunking fragmented it	Improve recall (hybrid), better chunking, add source
Wrong	Confident but off-topic / incorrect	Embedding “similar-but-wrong”; no rerank	Rerank, metadata filters, hybrid + exact match
Outdated	Old price / old API answer	Index not refreshed; cached doc	Freshness/invalidation, re-index, live fetch
Conflicting	Contradictory / arbitrary answer	Duplicates/versions; no source-of-truth precedence	Dedupe, rank by authority/recency, pass provenance
Excessive	Slower, costlier; key fact ignored	Dump-everything; no selection	Select less but right; cite position bias (lost in middle)

Context compression (long runs)

Move	What + when
Summarization	Condense old history. Lossy. History too long to carry verbatim.
Distillation	Extract facts/decisions/state, drop narrative. Lossy. Want conclusions, not chatter.
Compression	Dedupe/prune boilerplate. Near-lossless. Content is repetitive.
Sliding window	Last N turns verbatim + rolling summary of older. The default for long agents.

Lossy warning: a dropped ID/constraint can be the one that matters later. Compress the OLD; keep recent + decisions/IDs verbatim; store the full transcript and retrieve on demand. Compress as you approach the budget, not pre-emptively.

Match retrieval to data type

Data	Retrieval
Exact / financial	Deterministic structured query (SQL keyed by id) — never fuzzy embeddings; never invent amounts.
Fuzzy / textual	Hybrid + rerank over the KB; cite sources back.
Volatile	Fetch live — freshness/invalidation; don’t trust a stale index.
Long-running	Compress + remember — summarize old, keep recent verbatim.

Vector search returns the most similar, not the correct. For an invoice total you need a deterministic lookup. Log the assembled context per call — a wrong answer is undebuggable otherwise.

Retrieval evaluation L6

Retrieval metric	Asks
recall@k	Most important — is ≥1 relevant doc in top-k? Nothing downstream can use what wasn’t retrieved.
precision@k	What fraction of top-k is actually relevant? Junk crowds the window.
MRR	How high the first relevant hit sits (one good hit is enough).
nDCG	Graded, position-discounted — judges the reranker’s ordering.

Answer quality (RAGAS)	Asks
Faithfulness	Is every claim grounded in the retrieved context? (catches hallucination)
Answer relevance	Does it address the question? (catches off-topic)
Context precision	Are retrieved chunks relevant and well-ranked?
Context recall	Did retrieval capture all the ideal answer needs? (needs ground truth)

Measure retrieval separately from generation — faithful-but-wrong = retrieval bug; hallucination over good context = generation bug; don’t fix the wrong half.
Golden set + offline regression (CI, gates deploys) + online signals (thumbs · deflection · citation-clicks · rephrase rate); curate new misses back into the golden set.
Trace every call — query · retrieved IDs+scores · packed chunks · answer — turns “the bot was wrong” into “doc X wasn’t retrieved.” LLM-as-judge scales grading but must be calibrated vs human labels.

Pre-retrieval & advanced RAG L7

The raw query is a bad search query — fix it before the index (recall no reranker can recover); flat top-k is a floor, not a ceiling.

Query transform	Problem it solves
Rewriting	Vague / pronoun-laden / misspelled input — clean, expand, disambiguate.
Multi-query	One phrasing misses — N paraphrases, retrieve each, union & dedupe → recall.
HyDE	Questions look unlike answers — embed a drafted hypothetical answer to bridge the vocab gap.
Decomposition	Multi-part question no single chunk answers — split into sub-queries, combine.
Step-back	Too specific to find grounding — ask a broader question first.
Routing	Wrong datasource — classify and send to docs vs code vs SQL.

Parent-doc / small-to-big — match the small precise chunk, return the larger parent for context.
Hierarchical — retrieve over summaries, then drill into details (large docs).
Contextual retrieval — prepend chunk-situating context + add BM25 alongside vectors.
ColBERT / multi-vector — token-level late interaction → higher precision (costly).
GraphRAG — subgraphs + community summaries for global “connect-the-dots across the corpus” themes; overkill for local facts.
Agentic RAG / Self-RAG / CRAG — retrieval becomes an action in a loop: decide-to-retrieve, grade docs, re-retrieve / web-fallback, critique the draft. Buys quality; pays in latency, cost, complexity.

Embeddings, indexing & cost L8

The lens = embedding model + dimensions + similarity metric (cosine = angle, ignores magnitude; normalize first).
Chunk granularity drives embedding quality — a vector is the average of its text; too big → blurry centroid, too small → lost context.
Changing the embedding model = re-index the world — old/new vectors live in different spaces, incomparable; a full migration, not a config flip.

ANN index	Strength	Cost
HNSW (graph)	Fast, high recall	Memory-heavy; slow build
IVF (cluster)	Tunable speed via #probes	Recall dips if neighbor in un-probed cell
PQ (compress)	Big memory savings	Lossy → lower recall (pair as IVF+PQ)
Flat (brute)	Perfect recall, zero build	O(n)/query — small corpora only

Pick a corner: recall ↔ latency ↔ memory can’t all max. ANN gives up a few % recall for sub-linear speed; a reranker cleans the top.

Cost lever	Why
Fewer / better chunks	Tokens are the bill; noise also lowers accuracy.
Rerank only top-n	Cross-encoder on candidates, not all.
Cheaper embed model	Lower query + corpus embed price.
Prompt caching	Stop reprocessing the stable prefix (L9).
Batch embedding	Corpus embed offline, avoid needless re-embeds.

Caching, ordering & security L9

Prompt cache	Placement
System prompt	Stable → first — byte-identical every call.
Tool defs	Stable → in prefix — changes only on deploy.
Few-shots	Stable → in prefix — reused across calls.
Retrieved docs	Volatile → after prefix — can’t cache.
User query	Volatile → last — new every call.

Cache key = exact identical prefix; one changed token busts it from there down. Lost in the middle: attention is U-shaped — put the query last (recency, also cache-friendly) and the best doc at an edge; never bury key evidence mid-window.

Retrieved content is untrusted data, not instructions — indirect prompt injection rides in through your own KB; delimit it as data, filter outputs, constrain tool egress (OWASP LLM #1).
Multi-tenant access control belongs in the retrieval query — filter by tenant_id/ACL at the index, never in the prompt; the prompt is not a security boundary.
PII / sensitive-info disclosure — redaction, retention/residency limits, least-privilege retrieval.

5-minute revision (the whole spine)

Say the one sentence: model = constant, context = variable; right context + smaller model often beats wrong context + flagship.
Name the window tenants: prompt · tools · history · retrieved · question · answer — all compete for one fixed budget.
Draw the inversion: context retrieval is the new SQL query; the bug is silent — fluent and confidently wrong.
Selection funnel: candidate generation (recall) → ranking → pack to budget; you can’t pack what you never retrieved.
Engines: lexical=exact, semantic=meaning, hybrid=both, rerank=precision at the top.
Chunk on meaning + overlap + contextual summary; chunk grain is the retrieval grain.
RAG = retrieve + assemble + LLM; most “the LLM is dumb” bugs are retrieval bugs — fix recall before model.
Memory (system-written, per-user) ≠ knowledge base (external, read-only); both reach the model only via retrieval.
Long runs: store the transcript, compress the old, keep recent + IDs verbatim.
Classify any bad answer: missing / wrong / outdated / conflicting / excessive — then the fix is obvious.
Production reflex: name the strategy (pre-index · read-on-demand · none) → you’ve named its failure mode → match retrieval to the data type.
Eval (L6): grade retrieval and generation separately — recall@k first, then RAGAS (faithfulness · relevance · context precision/recall); golden set offline + signals online; trace every call.
Pre-retrieval (L7): the query is not the question — rewrite · multi-query · HyDE · decompose · step-back · route; fix the query before the index, recall no reranker can recover.
Advanced RAG (L7): flat top-k is a floor — small-to-big · hierarchical · contextual · ColBERT · GraphRAG (global themes); agentic/Self-RAG/CRAG loop buys quality, pays latency+cost.
Embeddings & cost (L8): model+dims+metric+chunk = the lens; change the model → re-index the world; ANN (HNSW/IVF/PQ/Flat) trades recall↔latency↔memory; tokens are the bill.
Production layer (L9): stable-first/volatile-last for cache; query last + best doc at an edge for “lost in the middle”; retrieved text is data not instructions; ACL in the query, not the prompt.