Your bar: trace one user query through the whole stack —
text → embedding → vector DB → reranker → model gateway → inference
server
— and say, at each hop, what it does and where the latency, cost, and failure
concentrate. By the end you can answer the foundational interview question:
what actually happens to a request between “user hits enter” and “tokens stream back,” and
which hop do you instrument first?
The stack is one straight line. Every hop transforms the request and hands it on:
Text becomes a vector (embedding model)
the vector DB finds near neighbors via ANN
a reranker sharpens precision on the top-K
the gateway and inference server serve the answer
The whole track in one picture. Every later lesson zooms into one of these boxes — this
lesson is the map: what each hop does, and where latency, cost, and failure live.1
1
The Stack Map
The request path is a pipeline of single-purpose hops. Each one transforms the request
and hands it forward, and each one is where some specific kind of pain — latency, cost, or
failure — concentrates.
The hop-by-hop request path
Two halves: a retrieval half (embedding, vector DB, reranker) that decides what
goes into the prompt, and a serving half (gateway, inference) that decides how
the answer is produced. This track is the ops deep dive on both.1
What each hop does — and what it costs you
Hop
Its one job
Where the pain lands
Embedding model
Turn query text into a dense vector
A model/version swap re-indexes the whole corpus
Vector DB (ANN)
Return the top-K nearest vectors fast
Recall vs latency vs memory — the core ANN trade
Reranker
Re-score the top-K for precision
Fixed cost per candidate; cannot recover a missed doc
Model gateway
Auth, route, rate-limit, cache, account
Cost & reliability chokepoint — the blast radius
Inference server
Run the GPU forward pass, stream tokens
KV-cache memory caps batch size, caps throughput
Is The AI infra stack is the chain of services a request crosses from raw text to
streamed answer: embed, retrieve (ANN), rerank, gateway, infer, cache.
Why it exists No single service does all of it well. Retrieval needs a vector index;
serving needs GPUs and a control plane. Splitting the work lets each tier scale and fail
independently.
Like (world) A library request: the catalog narrows millions of books to a shelf
(retrieval), a librarian picks the best few (rerank), and a reading room serves them
(inference). Each step has its own line and bottleneck.
Like (code) A classic web request path: DNS → LB → app → cache →
DB. You profile per hop and optimize the one that dominates p95 — same discipline here.
✗ “The LLM is the system — everything else is glue.”
✓ The model is one hop. Most production latency, cost, and failure live in retrieval
and the serving control plane around it, not inside the model call.
✗ “If it’s slow, it’s the model — make the prompt shorter.”
✓ Instrument per hop first. The dominant cost is often queue-wait at the inference
server or an over-tuned ANN search, not the forward pass itself.
📥 Memory rule: The stack is a pipeline of single-purpose hops.
Retrieval picks what; serving governs how — instrument every hop
before you optimize one.
Memory check
Name the hops in order. → text → embedding → vector DB (ANN) → reranker → model gateway → inference server → response cache
Which half decides what's in the prompt vs how it's served? → embed/retrieve/rerank decide WHAT; gateway/inference decide HOW
Which hop usually dominates latency? → the inference forward pass (sequential, GPU-bound), with queue-wait next under load — but measure, don't assume
M1 — Walk me through what happens to one user query as it travels through a production RAG stack.
Hit these points: the query text is embedded into a vector by an embedding model
→ that vector hits the vector DB, which runs approximate nearest-neighbor (HNSW or IVF)
over millions of stored vectors to return a top-K candidate set → an optional reranker
(a cross-encoder) re-scores that small set jointly with the query for precision → the
chosen passages plus the prompt go to the model gateway, which authenticates, rate-limits,
routes, and may serve a response-cache hit → on a miss the inference server runs the
forward pass with KV cache and continuous batching, streaming tokens back → name where
each hop concentrates risk: ANN trades recall for latency, the gateway is the reliability
and cost chokepoint, and the inference server is GPU-bound on KV-cache memory and batch
occupancy.
2
Embeddings
An embedding is a dense vector that captures meaning: vectors close together mean
similar things. You compute them once at ingest, then similarity at query time is just a
distance computation.2
Bi-encoder: embed once, compare cheaply
A bi-encoder embeds each document independently, so the corpus is pre-computed and indexed
offline. At query time you embed only the query and compare — that asymmetry is what makes vector
search feasible at scale.2
Why pre-compute? The cost of the alternative
Approach
When the work happens
Cost per query
Bi-encoder (embed once)
Documents at ingest; query at search
Embed 1 query + a distance scan — cheap, scales
Cross-encoder (joint)
Every query×doc pair, at query time
O(corpus) forward passes — accurate but unscalable
Hybrid (this stack)
Bi-encoder first, cross-encoder on top-N
Cheap recall, then precision only where it pays
Is An embedding is a fixed-length dense vector representing a chunk of text such that
semantic similarity maps to vector proximity (cosine or dot product).
Why it exists Keyword match misses meaning (“car” vs “automobile”). Embeddings let
you search by what text means, and reduce search to a fast distance computation.
Like (world) Placing books by topic on a map, not alphabetically. Once shelved by
meaning, finding “things like this one” is just looking nearby.
Like (code) A pre-built index vs a full table scan: you pay the indexing cost once at
write time so reads stay cheap. The embedding model is the index function.
✗ “More dimensions always means better retrieval.”
✓ Dimensions cost storage, memory, and ANN latency at scale. Match the model’s
dimension to your recall need and budget — bigger is not free and rarely linear in quality.
✗ “I can swap the embedding model anytime.”
✓ Query and corpus must share the same model. Swapping it means re-embedding and
re-indexing the entire corpus — a migration, not a config change.
📥 Memory rule: Embeddings turn meaning into position.
Pre-compute the corpus once; embed only the query at runtime — and never mix
embedding models across query and index.
Memory check
What does proximity in embedding space mean? → semantic similarity — near vectors mean similar things, so similarity becomes a distance computation
Why is a bi-encoder used for first-stage retrieval? → it embeds each doc independently, so the corpus is pre-computed; query time is one embed + a cheap scan
What breaks if you change the embedding model? → query and corpus must match — a swap forces re-embedding and re-indexing the whole corpus
M2 — What is an embedding, and why do you pre-compute them at ingest instead of at query time?
Hit these points: an embedding is a dense vector that captures meaning, so vectors
close in the space mean similar things and similarity becomes a distance computation →
a bi-encoder embeds each document once, independently, so you can pre-compute and index the
whole corpus ahead of time → at query time you only embed the query and compare
vectors, which is cheap → the alternative, scoring every query-document pair jointly (a
cross-encoder), is far more accurate but O(corpus) per query and impossible to run over
millions of docs at interactive speed → so the architecture splits the work: cheap
pre-computed bi-encoder for first-stage retrieval, expensive cross-encoder only on the small
reranked top-N.
A vector DB indexes embeddings for approximate nearest-neighbor search at scale. Exact
k-NN is O(n) per query and collapses; ANN trades a little recall for large speed and memory
wins.3
Exact vs approximate — why ANN exists
Exact search is correct but linear — fine for a prototype, fatal at scale. ANN indexes turn
the scan into a guided traversal, accepting that they may occasionally miss a true neighbor.3
nprobe ↑ raises recall; cost is latency, lighter on memory
Exact (flat)
Brute-force scan of all vectors
No knob — perfect recall, O(n), small corpora only
The one trade you always tune
Every ANN deployment is a chosen point on this triangle. The search-breadth knob (ef
for HNSW, nprobe for IVF) slides you along recall↔latency; build-time degree
(M) and quantization trade memory. There is no setting that maximizes all
three.3
Is A vector DB stores embeddings and serves ANN similarity search with metadata
filtering, CRUD, sharding, and replication — the serving substrate retrieval reads from.
Why it exists A flat array + brute-force similarity works as a demo but collapses on
latency, freshness, and multi-tenancy. A vector DB makes search sub-linear and operable.
Like (world) A library’s card catalog: you don’t walk every shelf, you follow an
index that jumps you near the right section — accepting it occasionally points you one shelf
off.
Like (code) A B-tree index on a SQL column: it converts a full scan into a guided
lookup. ANN is the same idea for high-dimensional vectors, but approximate by design.
✗ “ANN gives the same results as exact search, just faster.”
✓ ANN is approximate — it can miss a true neighbor. You buy recall back with
the search knob, paying latency for it. Measure recall@k; don’t assume it.
✗ “I always need a dedicated vector database.”
✓ If embeddings fit beside your OLTP data and volume is modest, pgvector in Postgres
keeps one store you already operate. Reach for a dedicated system when scale, QPS, or hybrid
search force it.
📥 Memory rule: A vector DB trades a little recall for
large speed & memory wins. The search knob (ef/nprobe)
is the dial — and the reranker can only re-order what it returned.
Memory check
Why doesn't exact k-NN scale? → it's O(n) per query — every query scans the whole corpus, which is fatal past a few hundred thousand vectors at interactive latency
What does the HNSW ef / IVF nprobe knob trade? → recall against latency — a wider search finds more true neighbors but costs more time
Can a reranker fix low first-stage recall? → no — it only re-orders the top-K it was given; a doc the ANN search missed is gone
M3 — Why can’t you just run exact nearest-neighbor search over your embeddings, and what does ANN trade away?
Hit these points: exact k-NN compares the query against every stored vector, so it is
O(n) per query and does not scale past a few hundred thousand vectors at interactive latency
→ ANN indexes (HNSW, IVF) build a structure that searches only a fraction of the
corpus, turning O(n) into roughly logarithmic or cluster-bounded work → the trade is
recall: ANN may miss a true neighbor, so you tune a knob (HNSW ef, IVF
nprobe) that buys recall back at the cost of latency and compute → there is
no free lunch: higher recall means a wider search means more latency, and the reranker can
only re-order what the first stage returned, never recover a doc it missed.
Retrieval practice — test the three modules
Q1. In the standard RAG request path, the hops occur in which order?
Q2. Why do you pre-compute document embeddings once at ingest time?
Q3. Exact k-NN search over a large embedding corpus does not scale because it…
Q4. Raising the HNSW ef (or IVF nprobe) search knob will…
Q5. The model gateway is best described as the stack's…
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs &
failure modes · Staff = synthesis under ambiguity · System Design = open
design round (a different axis, not a harder level).
Name the hops a query crosses from text to streamed answer, in order.
Hit these points: query text → embedding model (text becomes a vector) → vector DB (ANN search returns the top-K candidates) → reranker (a cross-encoder re-scores the top-K for precision) → model gateway (auth, routing, rate limits, response cache) → inference server (KV cache + batching, the forward pass) → tokens stream back → the first half decides what goes in the prompt; the second half governs how it's served.
What is an embedding, and what does proximity in the vector space mean?
Hit these points: an embedding is a dense vector that captures meaning → vectors that are close together represent similar things, so similarity becomes a distance computation (cosine or dot product) → you pre-compute the corpus embeddings once at ingest with a bi-encoder, then embed only the query at runtime → that's what lets keyword-blind semantic search work: "car" and "automobile" land near each other even with no shared tokens.
What does a vector database give you over a flat array of vectors plus a similarity loop?
Hit these points:ANN indexing for sub-linear search over millions of vectors instead of a full O(n) scan → metadata filtering so you can constrain by tenant_id, date, or type in the same query → CRUD and incremental index maintenance so inserts/deletes don't force a rebuild → the operational layer: sharding, replication, persistence, backups → the flat-array version is a fine prototype but collapses on latency, freshness, and multi-tenancy as it grows.
Why is exact nearest-neighbor search a non-starter at scale, and what replaces it?
Hit these points: exact k-NN compares the query to every stored vector, so it's O(n) per query → that's fine for a few hundred thousand vectors but fatal at millions or billions at interactive latency → ANN (approximate nearest neighbor) builds an index — a proximity graph (HNSW) or clusters (IVF) — that visits only a fraction of the corpus → the trade is a small recall loss for large speed and memory wins → you tune recall back with a search knob, accepting more latency.
Where does latency concentrate in this stack, and how do you find the dominant hop?
Hit these points: instrument every hop with a span — embed, ANN search, rerank, gateway, inference (queue + forward pass), response — and read per-hop p95, not just the total → on most stacks the inference forward pass dominates because generation is sequential and GPU-bound, with queue-wait next under load → ANN search is usually single-digit ms unless you over-tuned recall; the reranker adds a fixed cost proportional to candidate count → the gateway is a thin hop unless it's retrying or failing over → the discipline: measure per-hop first — the hop you assume is slow rarely is.
Why is the model gateway the reliability and cost chokepoint, and what belongs in it?
Hit these points: every request to every model flows through it, so it's the single place to see and cap spend and the single thing that can take the feature down → it owns auth, per-tenant rate limits and quotas, routing across providers/tiers, retries with backoff and failover, response caching, and cost accounting per request → that makes it the natural place to enforce a budget and degrade gracefully when a provider is down → it's also a blast-radius risk: it must be horizontally scalable and stateless where possible, never a hidden single instance → it's the API-gateway pattern applied to LLM traffic.
A reranker can only re-order what vector search returned. Why does that constrain the whole pipeline?
Hit these points: retrieval is two stages — a cheap high-recall first stage (ANN over embeddings) and an expensive high-precision second stage (the cross-encoder reranker) → the reranker re-scores only the top-K it was handed, so if the right doc isn't in that K, no reranking recovers it → recall is set by the first stage; precision is improved by the second → so tune the first stage for recall (wider ANN search, larger K), accept the latency, then let the reranker sharpen order → failure mode: a too-small K or under-tuned ANN silently caps answer quality and reranking hides none of it.
What is the KV cache, and why does it dominate GPU memory during inference?
Hit these points: during generation the model reuses the attention key/value tensors for every token already processed, caching them instead of recomputing → the KV cache grows with sequence length and concurrent requests, and at long contexts it dwarfs the model weights as the memory consumer → fragmented or over-allocated KV cache wastes GPU memory and caps how many requests you can batch, which caps throughput → this is exactly what PagedAttention addresses — paging KV memory like an OS pages RAM to cut fragmentation and pack more sequences → so KV-cache management is the lever that sets serving throughput and cost per token.5
What does continuous (in-flight) batching buy you over static batching, and what's the trade?
Hit these points: GPUs are efficient on big matmuls, so one request at a time leaves the hardware idle — batching fills it → static batching waits to assemble a fixed batch and holds fast requests hostage to the slowest, hurting tail latency → continuous batching schedules at the token step: it admits new requests and evicts finished ones each iteration, keeping the GPU full and not blocking short requests behind long ones → the trade is scheduler complexity and a memory ceiling set by the KV cache — you can only batch as many sequences as KV memory holds → net: higher throughput and steadier latency, which is why production inference servers default to it.5
The same retrieval corpus, but answer quality differs between two index configs. How do you reason about it before touching the model?
Hit these points: separate recall (did the right doc make the candidate set?) from precision (was it ranked high enough to use?) → measure recall@k against a labeled set for both configs — if the doc isn't in the top-K, it's a first-stage problem: under-tuned ANN (ef/nprobe too low), too-small K, or chunking that split the answer → if recall is fine but ranking is poor, it's the reranker or the assembly order, not retrieval → only after retrieval is sound do you look at the prompt or model → the principle: don't tune the expensive end of the pipeline to compensate for a cheap-end recall gap — fix it where it's caused.
"We'll just add a bigger model and a longer context — that fixes quality." Tear this apart as a system design.
Hit these points: a bigger model and longer context don't fix a retrieval gap — if the right doc never enters the window, no model size recovers it → longer context inflates the KV cache, shrinking how many requests batch, raising latency and cost per token at the inference server → a bigger model raises per-token cost and latency everywhere, hitting the gateway budget hardest → the cheaper, higher-leverage fixes live upstream: better embeddings/chunking for recall, a reranker for precision, response caching for repeat queries → the framing: spend where the bottleneck is measured to be, and the bottleneck is usually retrieval quality or serving throughput, not model capacity.
Design-round framework — narrate the stack in request order so the interviewer hears each hop's job, then its bottleneck:
Clarify scope: QPS, corpus size, latency SLO, freshness, tenant isolation, and the cost of a wrong answer vs a slow one.
Retrieval half: embed the query (cache popular embeddings) → ANN search with a tenant_id metadata pre-filter → rerank the top-K with a cross-encoder sized to the budget.
Name the bottleneck at each hop: ANN recall vs latency, gateway as cost/reliability chokepoint, KV memory capping batch size.
Build-vs-buy: pgvector vs a dedicated vector DB, hosted vs self-served inference — decide from measured scale, not preference.
Measure & degrade: per-hop p95, cost per query, recall@k, cache hit ratio; graceful degradation when a provider fails.
Design the request path for a multi-tenant RAG search feature: thousands of QPS over tens of millions of documents.
A strong answer covers: clarify scope first — QPS, corpus size, latency SLO, freshness, tenant isolation, and the cost of a wrong vs slow answer → map the path: embed the query (cache popular query embeddings), ANN search with a tenant_id metadata pre-filter so isolation is enforced at the index, return top-K → rerank the top-K with a cross-encoder sized to the latency budget → assemble prompt + passages and send through the model gateway (auth, per-tenant rate limit, route, response-cache check before spending on inference) → on a miss the inference server runs with KV-cache paging and continuous batching, streaming tokens → name the bottlenecks: ANN recall-vs-latency, the gateway as cost/reliability chokepoint, KV memory capping batch size → measure per-hop p95, cost per query, recall@k, cache hit ratio; degrade gracefully when a provider fails.
Build-vs-buy: when do you reach for a dedicated vector DB versus pgvector in your existing Postgres?
A strong answer covers: start from the data, not the hype — if embeddings fit alongside your OLTP data and retrieval volume is modest, pgvector keeps everything in one store you already operate, back up, and join against, a real operational saving → reach for a dedicated vector DB when scale or traffic breaks that: very large corpora needing sharding, high QPS needing an independently scaled retrieval tier, advanced hybrid search and reranking modules, or serverless cost models → the trade is operational surface — another system to run, secure, and pay for, against Postgres's familiarity and weaker ANN tuning → deciders: corpus size, QPS, freshness, metadata-filter complexity, team operational appetite → default to the store you already run until a measured limit forces the move.4