Lesson 8 · Principal Track · Senior → Staff
Embeddings, Indexing & Cost
Published
Your bar: explain what an embedding actually is and how model, dimensions, and similarity metric trade quality for cost; describe how an approximate-nearest-neighbour index (HNSW / IVF / PQ) buys speed by trading recall and memory; and model the real per-query token and dollar cost of a retrieval pipeline — knowing which levers move it. By the end you can answer:
given millions of chunks and a budget, how do you serve relevant context fast without re-indexing the world or blowing the bill?
1This lesson is the engine room under Lesson 2’s semantic search and Lesson 3’s chunks. Each chip is one fact it nails down:
text becomes a vector in meaning-space
an ANN index finds near vectors fast
ingestion is a pipeline, not a script
and every step has a token & dollar cost
1
Embeddings Deep-Dive
An embedding turns text into a point in meaning-space. Texts that mean similar things land close together; that closeness is what semantic search measures.3
What an embedding is: meaning becomes geometry
The four knobs you actually choose
| Knob | What it sets | Effect | Trade-off |
|---|---|---|---|
| Model | General vs domain-specific vs multilingual | Decides what “similar” means for your data | Quality vs speed vs price; domain models win on jargon |
| Dimensions | Vector length (e.g. 384 · 768 · 1536 · 3072) | More dims capture more nuance | Bigger = more storage, RAM, and compute per query |
| Metric | Cosine · dot product · Euclidean | How “closeness” is scored | Cosine ignores magnitude; must match how the model was trained |
| What you embed | The chunk (granularity from Lesson 3) | Defines the unit of retrieval | Too big → averaged, diluted vector; too small → lost context |
Dimensions & the similarity metric
Is Dimensions are the length of the vector. A 3072-dim embedding has more axes to encode nuance than a 384-dim one — at 8× the storage and compute. Matryoshka embeddings are trained so you can truncate to fewer dims and keep most of the quality, trading dims for cost on demand.
Why it exists Cosine similarity scores the angle between vectors — direction, not magnitude — so a long document and a short one about the same topic still score as close. Normalize vectors to unit length and cosine and dot product agree; Euclidean then ranks the same way.
Like (world) A map: latitude/longitude are 2 dimensions placing every town. Add altitude and you separate towns that look adjacent on a flat map. More dimensions = more ways for two points to be “different.”
Like (code) A feature vector for a recommender. You compare users by the angle between their preference vectors (cosine), not by how many items each rated (magnitude). Same math, same normalize-first habit.
Chunk granularity decides embedding quality
✗ “Embed the whole document — one vector per file is simplest.”
✓ A vector is an average of its text’s meaning. Embed a 5,000-word file and you get a blurry centroid that matches nothing precisely; chunk first (Lesson 3) so each vector means one thing.
✗ “Switching to a better embedding model is a config change.”
✓ It’s a migration. Old and new vectors live in different spaces and can’t be compared — you must re-embed the entire corpus and rebuild the index.
Re-embedding is a real migration
| Change you make | Blast radius | Cost driver |
|---|---|---|
| Tune the prompt / retrieval k | None — query-time only | Free; no re-index |
| Truncate Matryoshka dims | Re-index vectors (cheap re-store) | Storage + rebuild, no re-embed |
| Change the embedding model | Re-embed every chunk + rebuild index | Embed tokens for the whole corpus |
| Re-chunk the corpus | Re-embed + re-index everything | Embed tokens + pipeline run |
📥 Memory rule: The embedding model and the chunk are the lens. Change the lens and you re-index the world — so choose both before you fill the store.
Memory check
- Why does cosine similarity ignore magnitude? → it measures the angle between vectors, so topic (direction) matters, not document length
- What do Matryoshka embeddings let you trade? → dimensions for cost — truncate to fewer dims and keep most quality without re-embedding
- Why is changing the embedding model a migration? → new and old vectors are in different spaces; you must re-embed the whole corpus and rebuild the index
A teammate wants to bump the embedding model to the newest one “for a quick quality win.” What do you flag?
Hit these points: it’s not a flag flip — old and new vectors are incomparable, so every chunk must be re-embedded and the index rebuilt → that’s a corpus-wide embed-token bill plus pipeline time, and a cutover plan (dual-write / shadow index) so search isn’t down mid-migration → quantify it: tokens-per-chunk × chunk count × price → validate the win with an eval set before committing, since “newest” isn’t always better on your domain → consider Matryoshka truncation or a domain model as cheaper alternatives if the goal is quality-per-dollar.
2
Vector Index Internals (ANN)
Exact nearest-neighbour search compares the query to every vector — O(n) per query. At millions of vectors that’s too slow, so we approximate: trade a little recall for a huge speed-up.3
Why approximate? Brute force doesn’t scale
The three index families
| Index | How it works | Strength | Cost |
|---|---|---|---|
| HNSW (graph) | Navigable small-world graph; greedily hop toward neighbours | Fast, high recall | Memory-heavy; slower to build |
| IVF (cluster) | Partition space into cells; probe only the nearest few | Tunable speed via #probes | Recall dips if a neighbour is in an un-probed cell |
| PQ (compress) | Product quantization: compress vectors to codes | Big memory savings | Lossy → lower recall; often paired as IVF+PQ |
| Flat (brute force) | Compare against all vectors exactly | Perfect recall, zero build | O(n) per query — only for small corpora |
The trade-off triangle & metadata filtering
Is ANN is a family of data structures (graphs, partitions, compressed codes) that find probably the nearest vectors without scanning all of them. The control knobs (ef-search, n-probe) directly trade recall against latency.
Why it exists Vector similarity at scale is a high-dimensional search problem with no cheap exact index. ANN accepts a tiny, measurable recall loss to make queries sub-linear — the only way to serve millions of vectors in milliseconds.
Like (world) Finding the nearest coffee shop: you don’t measure the distance to every shop in the city (exact). You look in your neighbourhood and the next one over (probe a region) — almost always right, far faster.
Like (code) A database index: a B-tree doesn’t scan every row, it narrows the search. ANN is the same idea for “nearest by meaning” instead of “equals by key” — and like indexes, it has build and memory cost.
✗ “Production AI always needs a dedicated vector database.”
✓ For a small corpus, brute-force or
pgvector in your existing Postgres is fine and far simpler. Reach for a vector DB when scale, latency, or filtering demand it — not by default.✗ “Post-filtering metadata is equivalent to pre-filtering.”
✓ Post-filter retrieves k, then drops non-matches — you can end up with too few results. Pre-filter scopes the search to the allowed set first; it’s also the tenant-isolation hook.
📥 Memory rule: ANN buys speed by approximating — tune recall vs latency vs memory, and don’t reach for a vector DB before you actually need one.
Memory check
- What does ANN trade away, and for what? → a small amount of recall in exchange for sub-linear (much faster) queries
- HNSW vs PQ in one line each? → HNSW = graph, fast + high recall, memory-heavy; PQ = compressed vectors, low memory, lower recall
- Why prefer pre-filter over post-filter? → pre-filter scopes the search (right result count + tenant isolation); post-filter can leave too few results
Your vector search is fast but misses obviously relevant chunks for some queries. Where do you look?
Hit these points: “fast but misses” smells like low recall from the ANN settings — raise ef-search / n-probe and re-measure recall against an exact baseline → check if PQ compression is too aggressive (lossy codes drop near-neighbours) and whether the index needs a rebuild after heavy upserts → verify metadata filtering isn’t post-filtering away good hits, or scoping to the wrong tenant/date → confirm the query is embedded with the same model and normalization as the corpus → if recall is genuinely good and ranking is wrong, that’s a reranker problem (Lesson 2), not the index.
3
Indexing Pipelines, Freshness & Cost
Indexing is a data pipeline that runs forever, not a one-time script. Documents change, models change, and a stale index quietly serves yesterday’s answer.2
The ingestion pipeline
Freshness: a stale index is the “outdated context” failure
| Freshness need | Mechanism | Why |
|---|---|---|
| Edited / deleted docs | Incremental upsert · change-data-capture | Only re-embed what changed; delete removed chunks |
| New embedding model | Full re-embed + rebuild (Module 1) | Spaces are incompatible — no partial path |
| Volatile facts (prices, stock) | TTLs · prefer live structured fetch | Don’t index data that’s wrong minutes later |
| Stale chunk lingering | Index invalidation on source change | A stale index is Lesson 4’s outdated-context bug |
✗ “Index it once at launch and you’re done.”
✓ Sources change constantly. Without incremental updates and invalidation, the store serves deleted docs and old prices — confident, fluent, wrong.
Where the tokens and dollars go
| Cost line | Roughly equals | Lever to pull |
|---|---|---|
| LLM generation | (input + output tokens) × price | Select fewer-but-better chunks; trim the prompt |
| Query embedding | query tokens × embed price | Cheaper / smaller embedding model |
| ANN search | index compute + latency | Tune recall vs latency; right index family |
| Rerank | top-n candidates × rerank price | Rerank only the top-n, not all candidates |
| Corpus embedding | corpus tokens × embed price (one-off / on re-index) | Batch (not real-time); avoid needless re-embeds |
Is Per-call cost ≈ (input + output tokens) × model price, plus the embedding and ANN-search latency the retrieval step adds before the model ever runs. Tokens are the bill; latency is the user-facing tax.
Why it exists Most cost hides in the input side — stuffing the window with marginal chunks pays for tokens that don’t improve the answer. Liu et al. show more isn’t better; fewer-but-relevant beats more.4
Like (world) A restaurant bill: the entrée (LLM tokens) dominates, but the sides (embed, rerank) add up. You cut the bill by ordering less of what you won’t eat, not by switching restaurants.
Like (code) An N+1 query: the fix isn’t a faster DB, it’s fetching less, smarter. Prompt caching (Lesson 9) is the equivalent of memoizing the stable prefix so you stop paying for it every call.
✗ “Stuff more context in — tokens are cheap and it can’t hurt.”
✓ Tokens are the bill, and noise lowers accuracy (Lesson 1). Select less but better: cheaper, faster, and more accurate at once.
📥 Memory rule: Indexing is a pipeline, not a script — a stale index = wrong answers; tokens are the bill, so select less but better.
Memory check
- What are the five stages of the ingestion pipeline? → load → parse → chunk → embed → upsert
- How should you handle fast-changing facts like prices? → TTLs and prefer live structured fetch — don't bake volatile data into the index
- Name three levers to cut per-query cost. → fewer/better chunks, rerank only top-n, cheaper embedding model (and prompt caching, batch embedding)
Your RAG feature’s cost-per-query is 3× the budget. Walk through how you’d bring it down.
Hit these points: first attribute the cost — break it into LLM input, LLM output, query embed, ANN search, rerank; the input tokens usually dominate → cut retrieved chunks to fewer-but-better (helps accuracy too) and trim the prompt; rerank only the top-n → cache the stable prefix with prompt caching (Lesson 9) so you stop re-paying for system prompt and tool defs → drop to a cheaper embedding model or truncated Matryoshka dims if quality holds on an eval set → batch corpus embedding instead of real-time → only then consider a smaller generation model. Measure each change against an eval set so you don’t trade dollars for accuracy.
How do you keep an index fresh for a docs site that updates daily, with some live-changing fields?
Hit these points: split by volatility — stable prose goes through incremental ingestion (CDC / re-embed only changed pages, delete removed ones) → fast-changing fields (price, availability) shouldn’t be embedded at all; fetch them live via a structured tool at query time, or TTL them → wire invalidation so a source edit triggers re-chunk/re-embed of just that doc → on an embedding-model change, run a full re-embed behind a shadow index and cut over → monitor staleness (oldest-chunk age) as an SLO, because a stale index is the silent outdated-context failure from Lesson 4.
Retrieval practice — test the three modules
Q1. An embedding is best described as…
Q2. Cosine similarity is popular for embeddings because it…
Q3. An approximate-nearest-neighbour (ANN) index exists primarily to…
Q4. Switching to a different embedding model requires you to…
Q5. The biggest lever on per-query cost in a RAG pipeline is usually to…
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
What is an embedding, and what does "near" mean in embedding space?
Hit these points: an embedding is a dense vector of meaning — text mapped to a point in a learned high-dimensional space → the model places semantically similar text close together, so "near" is a distance/angle, not keyword overlap → "log the user out" and "revoke the session" share no words yet land close → that's why semantic search catches paraphrase where lexical match misses → the unit you embed is the chunk, so chunk granularity decides vector quality.
Name the three similarity metrics and say what cosine actually measures.
Hit these points: cosine, dot product, Euclidean → cosine scores the angle between two vectors — direction, not magnitude — so a long doc and a short one on the same topic still score close → normalize vectors to unit length and cosine and dot product agree, and Euclidean ranks the same way → the metric must match how the model was trained → the failure mode is mixing a metric the model wasn't trained for and quietly getting worse rankings.
What does ANN stand for, and why approximate instead of exact?
Hit these points: approximate nearest neighbour → exact k-NN compares the query to every vector — O(n) per query, too slow at millions of vectors → ANN gives up a few percent recall to make search sub-linear, orders of magnitude faster → recall = fraction of the true top-k the index actually returns → the bargain is fine for retrieval: 95% recall at ~1ms beats 100% at 500ms, and a reranker cleans up the top.
What are the five stages of an ingestion pipeline?
Hit these points: load → parse → chunk → embed → upsert → embed is the stage that bills you per token → the key idea is it's a pipeline that runs forever, not a one-time script — it re-runs incrementally as data changes → the canonical failure is treating it as "index once at launch," which lets the store drift out of sync with the source.
A teammate wants to bump the embedding model "for a quick quality win." What do you flag?
Hit these points: it's not a config flip — old and new vectors live in different spaces and are incomparable, so every chunk must be re-embedded and the index rebuilt → that's a corpus-wide embed-token bill plus pipeline time, and a cutover plan (dual-write / shadow index) so search isn't down mid-migration → quantify it: tokens-per-chunk × chunk count × price → validate the win on an eval set first — "newest" isn't always better on your domain → cheaper alternatives: Matryoshka truncation or a domain model if the real goal is quality-per-dollar.
Pick an index family for 50M vectors with tenant filtering and a p99 latency budget. Walk the trade-offs.
Hit these points: you can't max recall, latency, and memory at once — name the triangle and pick a corner → HNSW buys recall and low latency by spending memory; IVF tunes speed via n-probe; PQ buys memory back by spending recall, often IVF+PQ at this scale → tenant filtering must be a pre-filter that scopes the search to the allowed set — right result count and a security/isolation boundary — not a post-filter that can return too few → size RAM against vector count × dims, and validate recall against an exact baseline before shipping.
Vector search is fast but misses obviously relevant chunks for some queries. Where do you look?
Hit these points: "fast but misses" smells like low recall from ANN settings — raise ef-search / n-probe and re-measure recall against an exact baseline → check whether PQ compression is too aggressive (lossy codes drop near-neighbours) and whether the index needs a rebuild after heavy upserts → confirm metadata filtering isn't post-filtering away good hits or scoping to the wrong tenant/date → verify the query is embedded with the same model and normalization as the corpus → if recall is genuinely good but ranking is wrong, that's a reranker problem, not the index.
Your RAG cost-per-query is 3× the budget. Name the levers, in order.
Hit these points: first attribute it — break cost into LLM input, LLM output, query embed, ANN search, rerank; input tokens usually dominate → cut retrieved chunks to fewer-but-better (helps accuracy too) and trim the prompt → rerank only the top-n, not every candidate → cache the stable prefix (system prompt, tool defs) so you stop re-paying for it → drop to a cheaper / smaller-dim embedding model if quality holds on an eval set → batch corpus embedding instead of real-time → only then consider a smaller generation model — and measure each change against an eval set so you don't trade dollars for accuracy.
Design an embedding + index + cost strategy for a 200M-chunk corpus on a fixed monthly budget.
Hit these points: start from the budget and work backwards — model the dominant cost line (LLM input tokens at serve time, plus one-off corpus embedding) and set targets per line → choose model + dims by quality-per-dollar on an eval set, lean on Matryoshka so you can dial dims down without re-embedding → index family scoped to scale: IVF+PQ for memory at 200M, pre-filter for tenancy, tune n-probe to hit a recall floor at a latency ceiling → serve-time levers: fewer-better chunks, rerank top-n, cache the stable prefix → ingestion is incremental (CDC) with batched embedding, not real-time → close the loop with recall and cost-per-query as monitored SLOs so a regression is visible, not a surprise invoice.
When is a vector index the wrong tool, and what would you build instead?
Hit these points: name the cases where it's the wrong default — small corpus (brute-force or pgvector in existing Postgres is simpler and fast enough), data that's exact-match or highly structured (a relational/keyword index beats meaning-space), and volatile facts like price or stock that are stale minutes after embedding → for volatile fields, fetch live via a structured tool at query time or TTL them, don't bake them into the index → the staff judgement is matching retrieval mechanism to data shape and freshness, and not adding vector-DB infrastructure before scale, latency, or filtering actually demand it.
A stale index keeps serving a deleted doc. Frame this as a systemic failure and fix it at the root.
Hit these points: name it — a stale index is the outdated-context failure: deleted docs and old facts served fluently and confidently → root cause is treating ingestion as a one-time script with no invalidation, not a bad query → fix the system: incremental upserts / CDC for edits, hard deletes for removed chunks, invalidation triggered on source change → split by volatility — stable prose ingested, volatile fields fetched live → model changes go through a full re-embed behind a shadow index, then cutover → make staleness observable: monitor oldest-chunk age as an SLO so it's caught by a dashboard, not a user.
Design-round framework — there's no single right answer; a strong candidate drives the structure. Steer the open prompts below through:
- Clarify scope & constraints — corpus size, change rate, query volume, latency SLO, budget ceiling, multi-tenant?
- Data shape & freshness — what's stable prose vs volatile fields; what must be fetched live vs embedded.
- Embedding choice — model, dimensions (Matryoshka headroom), metric; justified on an eval set, not by defaults.
- Index & retrieval — index family vs the recall/latency/memory triangle; pre-filter for tenancy; rerank top-n.
- Ingestion & freshness — incremental CDC, deletes, invalidation, batched embedding; re-embed-the-world handled via shadow index.
- Cost model — per-line cost (LLM input dominates), the levers, and where you'd spend vs save.
- Observability & failure modes — recall, cost-per-query, oldest-chunk age as SLOs; what breaks first.
Design the vector index + ingestion pipeline for a fast-changing corpus (news + product catalog, edits every minute).
A strong answer covers: clarifies scale, edit rate, and latency/freshness SLOs first → splits by volatility — article prose is embedded and incrementally ingested via CDC (re-embed only changed docs, delete removed ones); price/stock/availability are not embedded but fetched live at query time or TTL'd, because a stale index is the outdated-context failure → index family chosen for scale and recall needs (HNSW for high recall if memory allows, IVF+PQ if it doesn't), pre-filter for category/tenant scoping → embedding model + dims justified on an eval set, with Matryoshka headroom → invalidation wired to source changes so a single edit re-chunks/re-embeds just that doc → model upgrades run behind a shadow index then cut over → staleness monitored as an SLO (oldest-chunk age) so drift is visible.
Design a cost-bounded retrieval stack: serve relevant context under a hard per-query dollar/latency ceiling.
A strong answer covers: starts from the ceiling and attributes cost per line — LLM input usually dominates, then output, query embed, ANN search, rerank → primary lever is fewer-but-better chunks (cheaper and more accurate) plus a trimmed prompt → rerank only the top-n; cache the stable prefix so the system prompt and tool defs aren't re-paid every call → embedding cost controlled with a cheaper/smaller-dim or Matryoshka-truncated model, validated on an eval set → latency budget met by tuning recall vs latency (ef-search / n-probe) and the right index family, not by buying hardware → corpus embedding batched, not real-time → cost-per-query and recall tracked as SLOs with alerts, so a regression shows on a dashboard before it shows on the invoice → every cut measured against an eval set so dollars aren't traded for accuracy.
Was this lesson helpful? Thanks — noted.
These notes reflect my current understanding and are updated as I learn, build, and discover better explanations.