Architecture Reviews & Production Design
Published
Your bar: walk into any AI architecture review and, within a minute, name the context strategy (pre-indexed vs read-on-demand vs none), name its failure mode, and prescribe the right retrieval for the data type. This capstone gets you fluent on real products (Cursor, Claude Code, ChatGPT, Copilot) and production agents (billing, support, engineering, research), then hands you the Principal’s toolkit: a cheat sheet, mental models, checklists, and interview banks you keep. 1
The whole capstone collapses to one reviewing reflex. Each chip is one move you’ll make on sight:
Every assistant is a context strategy + a model
name the strategy: pre-index · read-on-demand · none
and you’ve named its failure mode
then match retrieval to the data type
Architecture Reviews — Reading Real AI Products
You can’t review what you can’t name. Each product below is a context strategy with a model attached. Name the strategy first; its strengths, weaknesses, and failure mode follow from there.
The four products, side by side
| Product | Context strategy | Strengths | Weaknesses | Signature failure |
|---|---|---|---|---|
| Cursor | Pre-indexed embeddings of the repo + semantic retrieval (+ open files) | Fast recall across huge repos; finds code before you know the filename | Index staleness; embeddings miss exact identifiers | Retrieves a topically-similar but wrong file |
| Claude Code | Agentic read-on-demand via tools (grep / glob / read); follows the dependency graph live; no persistent index | Always fresh; precise; transparent about what it read | More tool round-trips → latency & token cost; depends on choosing good searches | Misses a file it never thought to grep |
| ChatGPT | Conversation history + optional retrieval / uploads / connectors / a memory feature | Broad general capability; cross-session memory | Weak grounding in your live systems without connectors | Hallucinates when nothing was retrieved |
| GitHub Copilot | Open files + nearby code + (newer) repo / workspace indexing for chat | Very low-latency inline local context | Historically a limited window of surrounding code; weaker whole-repo reasoning | Suggests plausible code that ignores a distant constraint |
Same query, two strategies
Memory check
- What’s Cursor’s signature failure, and why? → retrieves a topically-similar but wrong file — embeddings match meaning, not exact identifiers; index can also be stale
- What does Claude Code trade for always-fresh context? → more tool round-trips → higher latency and token cost; relies on choosing good searches
- When does a history/connector assistant hallucinate most? → when nothing relevant was retrieved and it isn’t grounded in your live systems
You’re reviewing a vendor’s “AI code assistant.” What’s the first question you ask, and why?
Hit these points: “What’s the context strategy — pre-indexed embeddings, read-on-demand tools, or just history?” → that single answer predicts strengths, cost shape, and the failure mode → pre-index → ask about index freshness/invalidation and identifier recall; read-on-demand → ask about round-trip latency and search-coverage gaps; history/none → ask what grounds it in live systems → then ask if there’s hybrid + rerank to cover the embedding blind spot → frame: I’m not comparing models, I’m comparing retrieval architectures.
Production Design — Match Retrieval to the Data
A production agent is a context pipeline. The senior move is choosing the retrieval mechanism by data type: exact & financial → structured query; fuzzy & textual → hybrid + rerank; volatile → fetch live; long-running → compress + remember.
The generic production context pipeline
Four agents, four data shapes
| Agent | Retrieval (matched to data) | Scalability | Cost | Reliability | Observability |
|---|---|---|---|---|---|
| Billing | Deterministic structured query — SQL keyed by account_id, never fuzzy embeddings | Cheap point lookups | Low tokens | No hallucinated numbers; never invent amounts | Log every assembled context + tool call for audit |
| Support | Hybrid over help-center KB + customer history (memory); rerank | KB grows; cache hot articles | Moderate | KB freshness; PII handling / compliance | Retrieval hit-rate; deflection/escalation; cite sources to user |
| Engineering | Codebase retrieval (hybrid + structural) and/or read-on-demand; tests as a grounding signal | Large repos; token-heavy | Token cost dominates | Ground in real code; run tests | Log which files / symbols loaded per task |
| Research | Multi-source web/doc retrieval, multi-step; compress intermediate findings; keep citations | Many calls / sources | High (fan-out) | Adversarial verification; dedupe conflicting sources | Source provenance per claim |
Memory check
- Why is embedding search wrong for a billing agent? → it returns the most-similar value, not the exact one; financial data needs deterministic, auditable structured queries
- What’s the dominant cost driver for an engineering agent? → tokens — large repos mean token-heavy context; cost scales with what you load per task
- What does a research agent need that a billing agent doesn’t? → adversarial verification, dedupe of conflicting sources, and source provenance per claim across a fan-out
Design the retrieval layer for a billing-support agent that must quote exact charges. Walk the trade-offs.
Hit these points: charges are exact, transactional, auditable → retrieve via deterministic SQL keyed by account_id/invoice id, not embeddings → embeddings would return the most-similar amount, i.e. a confidently-wrong number → reliability rule: the agent may quote only values it pulled, never synthesize amounts → log every assembled context + tool call for audit and dispute resolution → if the user also asks policy questions, that’s a different data type → layer hybrid KB retrieval for the textual part → one agent, two retrieval mechanisms matched to two data types.
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
Name each product's context strategy: Cursor, Claude Code, ChatGPT, GitHub Copilot.
What is the pre-indexed ↔ read-on-demand spectrum, and where do those four products sit on it?
Match each production agent to the retrieval its data type demands: billing, support, engineering, research.
account_id), never embeddings → support = fuzzy natural-language KB + customer history → hybrid + rerank, cite sources → engineering = structured code (imports, symbols, tests) → hybrid + structural retrieval or read-on-demand, run tests as a grounding signal → research = volatile, multi-source → fan-out web/doc retrieval, compress intermediate findings, keep provenance per claim → the rule underneath: match the retrieval mechanism to the data type.Define agent, runtime, and RAG as you'd use them in a production-design review.
Cursor and Claude Code use opposite strategies. Name each one's signature failure mode and why position on the spectrum causes it.
Why is semantic embedding search the wrong retrieval for a billing agent, and what does the right design cost you?
account_id/invoice id → the design rule: the agent may quote only values it pulled, never synthesize a number → the cost you accept is less "magic" — you can't fuzzily match a vaguely-worded question to a row, so you invest in intent parsing and exact keys → and you must log the assembled context per call, which billing needs anyway for audit and dispute resolution.A research agent fans out across many sources. Which failure modes does that shape invite, and how do you defend each?
"We'll add observability later — ship the agent first." Push back like a reviewer.
You're reviewing an unfamiliar vendor "AI assistant" with 20 minutes. What's your line of questioning, and what would make you walk away?
Build vs buy: a VP asks whether to adopt a vendor assistant or build our own retrieval. Make the principal-level call.
Across billing, support, engineering, and research agents, where does retrieval investment actually pay off — and where would you NOT over-invest?
- Clarify the data type first — exact/financial, fuzzy text, structural code, or volatile multi-source — because it dictates the retrieval mechanism.
- Place it on the spectrum: pre-index (fast, can go stale) vs read-on-demand (fresh, more round-trips) vs fetch-live for volatile/exact data.
- Retrieve: recall first — lexical (BM25) + semantic, hybrid — then rerank for precision; structured query where data is exact.
- Assemble within budget; compress long runs; reserve fixed space for history and the answer.
- Reliability & guardrails: no-synthesis rule for exact data, adversarial verification for research, injection/PII defense before the model.
- Cost & scale: tokens are the bill — tighter selection, caching hot retrievals, cap fan-out.
- Observability: log assembled context + tool calls per call; track retrieval hit-rate and the five failure modes separately from answer accuracy.
Design the context system for a billing assistant that quotes exact charges and also answers billing-policy questions.
account_id/invoice id, never embeddings, because a semantic match returns the most-similar amount — a confidently-wrong number → policy is fuzzy natural-language text → hybrid retrieval + rerank over the help-center KB, with citations → one agent, two retrieval mechanisms matched to two data types → reliability rule: figures are quoted from a pulled record, never synthesized → guardrails: PII redaction and access control at the retrieve stage, injection defense on KB content → observability: log every assembled context + tool call for audit and dispute resolution → name the failure modes the naive "vector DB for everything" design invites: wrong (similar-not-correct amounts) and outdated (stale figures) → close with freshness: invalidate on write since balances and policies change.Design the context system for a research agent that fans out across many sources and must produce a cited answer.
The Principal’s toolkit — your 9 take-home deliverables
Compact, visual revision artifacts. Skim them before a review or an interview; each one is a tool you keep.
1 · One-page Context Engineering cheat sheet
The window A fixed token budget shared by prompt, tools, history, retrieved context, and the answer. A bigger window raises the ceiling; it doesn’t fix what you put in it.
Context = the new query Retrieval/assembly plays SQL’s role: it decides what the model ever sees.
Selection → retrieve Lexical (BM25) · semantic (embeddings) · hybrid; recall first.
Rerank Precision second: re-order the recalled set so the best lands near the top.
Chunking Split on meaning, add overlap; add per-chunk context so a chunk stands alone.
RAG retrieve → assemble → LLM → answer; the answer is grounded in the fetched facts.
Memory vs KB Memory = per-user/session state you carry; KB = shared corpus you retrieve from.
Compression For long runs: summarize/prune intermediate context to stay in budget.
| The 5 failure modes | In one line |
|---|---|
| Missing | The needed fact was never retrieved. |
| Wrong | Topically-similar but incorrect context retrieved. |
| Outdated | Stale index/doc; reality has moved on. |
| Conflicting | Two sources disagree; model picks one silently. |
| Excessive | Too much noise; signal buried, cost up, “lost in the middle.” |
2 · Context Engineering mental models
Constant vs variable Context is the variable; the model is the constant. Change the context, change the answer.
The new query Context retrieval is the new database query — same role SQL played.
Recall, then precision Cast wide first; rerank to surface the best. Two stages, two jobs.
Match retrieval to data Exact → structured; fuzzy → hybrid; volatile → live; long → compress.
Most AI bugs are context bugs Suspect the input before blaming the model.
The window is RAM A budget you allocate, not a pipe to your data.
3 · Architecture-review checklist
The ten questions to run on any AI design
- What’s the context strategy? → pre-index · read-on-demand · none
- Are there recall and precision stages? → retrieve wide, then rerank
- What’s the chunking strategy? → split on meaning + overlap + per-chunk context
- How is index freshness / invalidation handled? → or is it stale-by-default?
- Is retrieval hybrid + reranked? → lexical + semantic covers each other’s blind spots
- Is retrieval evaluated — not just answers? → hit-rate / recall@k, not vibes
- Is there a budget / compression plan for long sessions? → summarize, prune, prioritize
- Can you observe the assembled context per call? → log what was sent, auditable
- Is each failure mode covered? → missing / wrong / outdated / conflicting / excessive
- Is PII / compliance handled in retrieval? → redaction, access control, injection defense
4 · Common failure modes — symptom → cause → fix
| Mode | Symptom | Root cause | Fix |
|---|---|---|---|
| Missing | ”I don’t have info on that” / vague answer | Relevant doc never retrieved (recall gap) | Improve recall: hybrid retrieval, better chunking, more k |
| Wrong | Confident answer about the wrong thing | Topically-similar but incorrect chunk ranked top | Add rerank; lexical signal for exact identifiers |
| Outdated | Answer reflects an old state of the world | Stale index / cached doc | Invalidation on write; or fetch live for volatile data |
| Conflicting | Inconsistent answers; arbitrary choice | Sources disagree; no resolution policy | Dedupe, prefer authoritative source, surface the conflict |
| Excessive | Slow, costly, “lost in the middle” | Too much low-signal context packed in | Tighten selection; rerank; compress; trim history |
5 · Interview questions (Core → Staff)
Core — What is a context window and why can’t an LLM just read the whole codebase?
Hit these points: the window is a fixed token budget — the model’s only channel → real repos are orders of magnitude larger → bigger windows cost more, run slower, and recall the middle poorly → so you must select a relevant subset → the job is “which slice,” not “read it all.”
Core — Lexical vs semantic vs hybrid retrieval — when each?
Hit these points: lexical (BM25) nails exact tokens/identifiers but misses synonyms → semantic (embeddings) matches meaning but misses exact strings and can drift → hybrid runs both and fuses (e.g. reciprocal rank fusion) → rerank then sharpens precision → default to hybrid + rerank in production.
Mid — Walk me through a RAG pipeline and where each failure mode enters.
Hit these points: ingest → chunk → index → retrieve → rerank → assemble → LLM → log → missing enters at retrieve (recall gap), wrong at rerank, outdated at index freshness, conflicting at assemble, excessive at assemble/budget → each stage gets its own metric.
Senior — How do you evaluate retrieval quality, not just answer quality?
Hit these points: build a labeled set of query→gold-doc pairs → measure recall@k and rerank precision separately from end-answer accuracy → log assembled context per call so failures are attributable → a wrong answer with the right context is a reasoning bug; with the wrong context it’s a retrieval bug → you can’t fix what you can’t separate.
Senior — Why is matching retrieval to data type a senior decision?
Hit these points: exact/transactional (a balance) → deterministic structured query; semantic search would return the most-similar, i.e. wrong → fuzzy text → hybrid + rerank → volatile → fetch live, don’t cache stale → long-running → compress + remember → the mistake juniors make is one-size-fits-all vector search.
Staff — Review this design: an AI agent answers finance questions over a vector DB of
reports. Pushback?
Hit these points: numbers must be exact and auditable — vector search returns similar, not correct → risk: confidently-wrong amounts, no audit trail → redesign: structured query for the figures, vector/hybrid only for narrative text → add a rule that figures are quoted, never synthesized → log assembled context per call → add freshness/invalidation since reports update → name the failure modes this design currently invites (wrong, outdated).
Staff — Same model, two products, one feels far smarter. Explain to a skeptical interviewer.
Hit these points: the model is the constant; context is the variable → the smarter product retrieves and assembles better context per call → “smart vs dumb” is mostly an artifact of what got loaded → therefore investment goes to retrieval, evaluation, and assembly — not chasing model versions → verify by logging context for the failing cases.
6 · VP Engineering review questions (leadership framing)
VP — Build vs buy: when do we build our own retrieval vs adopt a vendor assistant?
Hit these points: buy when the data is generic and the vendor’s context strategy fits our use-case → build when our differentiation is the retrieval over proprietary data → the model is a commodity; the retrieval pipeline is the moat → decision pivots on whether off-the-shelf grounding reaches our live systems.
VP — Where are the cost levers in an AI feature?
Hit these points: cost scales with tokens sent → tighter selection + rerank cuts wasted context → compress long sessions; trim history → cache hot retrievals → right context on a cheaper model often beats wrong context on the flagship — a structural saving, not a discount.
VP — Where does investment in retrieval actually pay off?
Hit these points: retrieval fixes compound across every feature on the stack → a model upgrade is a flat tax on every call → better retrieval lifts accuracy, lowers cost, and reduces hallucination simultaneously → prioritize where wrong answers carry the highest business/compliance risk.
VP — How do we measure retrieval quality as a leadership metric?
Hit these points: recall@k and rerank precision on a labeled set → retrieval hit-rate in production → deflection/escalation for support, audit-pass for billing → track these separately from end-answer accuracy so we know whether to invest in retrieval or the model.
VP — What are the risk and compliance exposures?
Hit these points: PII in retrieved context → redaction + access control at the retrieve stage → prompt injection via poisoned/retrieved content (OWASP LLM Top 10) → guardrails before the model → auditability: log assembled context per call → for finance, deterministic retrieval + no-synthesis rule.
VP — When do we upgrade the model vs invest in retrieval?
Hit these points: instrument first: was the right context retrieved for the failing cases? → if no → fix retrieval (cheaper, compounds) → if yes and reasoning is the bottleneck (multi-step synthesis) → pay for model capability → never upgrade the model to paper over a retrieval gap.
7 · 5-minute revision guide (the spine)
The model only sees its context window — a fixed token budget, not your data.
Context is the new database query: retrieval decides what the model ever sees.
Retrieve = recall first (lexical / semantic / hybrid), then rerank for precision.
Chunk on meaning + overlap + per-chunk context; assemble within budget.
Memory (per-user state) vs KB (shared corpus); compress long runs.
Five failure modes: missing · wrong · outdated · conflicting · excessive.
Review reflex: name the context strategy → name the failure → match retrieval to data type .
8 · 30-minute deep-dive revision (linking the track)
Lesson 1 — What Is Context & Context-as-Query. Window as fixed budget; why bigger ≠ solved; the model is constant, context is variable; retrieval = the new SQL.
Lesson 2 — Selection & Retrieval. Lexical vs semantic vs hybrid; recall vs precision; rerank; reciprocal rank fusion. Re-derive when each retrieval type wins.
Lesson 3 — Chunking, RAG & Assembly. Split on meaning + overlap; contextual chunks; the RAG pipeline ingest→index→retrieve→assemble→LLM; assembly order and position bias.
Lesson 4 — Memory, Compression & Failure Modes. Memory vs KB; compression for long-running agents; the five failure modes and their fixes; observability.
Lesson 5 — Architecture Reviews & Production Design (here). Place real products on the pre-index↔read-on-demand spectrum; design billing/support/engineering/research agents by data type; run the review checklist.
Self-test: for each lesson, state its one memory rule from memory, then design a small system that would violate it — and the fix.
9 · What to learn next
Next: Evaluation & Observability for LLM systems (“evals”) The natural sequel — this whole track kept saying “measure retrieval quality, not just answers.” Evals make that rigorous: labeled sets, recall@k, regression suites, LLM-as-judge, and tracing assembled context in production.
Sibling: Agent orchestration & multi-agent systems Once one agent’s context is solid, scale to many: planners, tool routing, sub-agent delegation, and how context flows (and compresses) between agents without losing provenance.
Retrieval practice — the capstone
Q1. Cursor's signature failure mode is best described as…
Q2. Claude Code's read-on-demand strategy trades freshness for what cost?
Q3. For a billing agent that must quote exact charges, the right retrieval is…
Q4. "Conflicting context" as a failure mode is fixed primarily by…
Q5. The reviewer's first move on any unfamiliar AI product should be to…
These notes reflect my current understanding and are updated as I learn, build, and discover better explanations.