Codebase Context, Chunking & RAG
Published
Your bar: explain how Cursor and Claude Code make sense of a repo too large for any window, why chunking exists and how it breaks, and how RAG works from first principles. Then argue, against an interviewer pushing back, why most “the LLM is dumb” bugs are retrieval bugs. By the end you can answer:
given a 2M-token codebase and a 200k window, how does the system surface the right symbols — and where does that pipeline silently break?
1This lesson is the productization of Lesson 1’s “context is the new database query.” Three steps build the query engine:
find the right code via indexing or live tools
split documents into retrievable chunks
retrieve & assemble them with RAG
where bad retrieval becomes fluent wrong answers
Codebase Context
A repo is far bigger than any window. So the tool must discover the relevant files and symbols, follow their dependencies, and assemble the smallest sufficient slice — the way a human navigates an unfamiliar codebase.
Two strategies, side by side
The three discovery steps, pinned down
| Step | What it finds | Classic-tooling analogue |
|---|---|---|
| File discovery | Which files are even relevant (by path, glob, directory tree). |
|
| Symbol discovery | Definitions & references of the functions/classes in play. | An LSP index or |
| Dependency traversal | What those symbols import & call — the surrounding graph. | The import / call graph. |
| Context assembly | Pack the slice within budget — prefer signatures over full bodies. | Building a query result set. |
Is Codebase context is the slice of a repo a tool surfaces for one task: the right files, their symbols, and the dependencies that make them make sense — assembled within the token budget.
Why it exists The repo is ~10× any window1, so the tool can never load it all. It must reproduce, automatically, how a human finds the few relevant places before reading.
Like (world) A new hire who can’t read the whole wiki — they search, open the few right pages, follow links between them, then answer. Pre-index = they read a summary deck first; read-on-demand = they search fresh each time.
Like (code) A search index vs a live SELECT: a materialized view (embeddings)
is fast but can lag the source; querying the table directly (grep) is always current but slower
per hit.
✓ Cursor leans on a precomputed embedding index; Claude Code searches the live filesystem with tools. Different freshness/latency trade-offs, same end goal: the right symbols in budget.
✓ Embeddings shine for fuzzy/semantic recall, but can miss an exact identifier that lexical search nails — and a stale index returns deleted code. Each tool wins different queries.
📥 Memory rule: Pre-index = fast but can go stale; read-on-demand = fresh but costs round-trips. Same goal: surface the right symbols within the budget.
Memory check
- Name the three discovery steps before assembly. → file discovery, symbol discovery (definitions/references), dependency traversal (imports/call graph)
- One downside each of pre-index vs read-on-demand? → pre-index can go stale / miss exact IDs; read-on-demand pays in tool round-trips (latency + tokens)
- Why prefer signatures over full bodies when assembling? → they convey the interface at a fraction of the tokens, leaving budget for more relevant code and the answer
Your team’s embedding-indexed code assistant keeps citing a function that was deleted last week.
What’s happening and how do you fix it?
Hit these points: a pre-built embedding index is a snapshot — if it isn’t reindexed on change, it serves stale vectors pointing at code that no longer exists → the fix is incremental reindexing on commit/save, plus a freshness check or TTL on the index → pair embeddings with a lexical/grep pass so exact current identifiers are verified against the live tree → or move latency-tolerant queries to read-on-demand so they always hit fresh files → root cause is index/source skew, not the model.
Chunking
You can’t embed or retrieve a whole file — it’s too big and too coarse. Chunking splits documents into pieces that each fit the embedder and are granular enough to return as a precise hit. How you cut decides what you can ever retrieve.
The size trade-off
Overlap & the fragmentation failure
Common mistakes → the better move
| Common mistake | Why it hurts | Better |
|---|---|---|
| Fixed-size character splitting | Cuts mid-function / mid-sentence; chunks are incoherent. | Structure-aware: split on function / class / heading bounds. |
| Zero overlap | Boundary-spanning facts fragment and become unretrievable. | Modest overlap so cross-boundary facts survive. |
| Ignoring document structure | A chunk lacks the heading/section it belongs to; ambiguous alone. | Contextual chunking: prepend a short doc/section summary to each chunk so it self-describes. |
Is Chunking is the rule for splitting a document into retrievable units. A good chunk is complete (answers on its own) and retrievable (its embedding matches the questions it should serve).
Why it exists Files exceed embedding input limits, and retrieval needs granularity — you want the relevant passage back, not a whole file. Chunking is the unit of both storage and recall.
Like (world) Index cards from a book. Cut them on chapter/section lines and each card stands alone; cut every 500 letters and you get cards that start mid-sentence and answer nothing.
Like (code) Choosing a primary key / row grain in a table: too coarse and a query returns bloated rows; too fine and you must join many rows to answer. Chunk grain is the retrieval grain.
✓ Fixed-size splits cut mid-thought and fragment facts. Split on meaning (function/heading bounds) and add overlap so each chunk is complete and retrievable.
✓ Too small strips the surrounding context that made the passage meaningful — and multiplies fragments. Precision needs the right grain plus enough context, not minimum size.
📥 Memory rule: Chunk on meaning, not character count. A chunk should be retrievable and complete on its own.
Memory check
- What does overlap protect against? → context fragmentation — a fact spanning a boundary that would otherwise be split across two chunks and retrieved by neither
- Too-big vs too-small chunks fail how? → too big = diluted/averaged embedding, imprecise hits, wasted tokens; too small = lost surrounding context, many fragments
- What is contextual chunking? → prepend a short summary of the doc/section to each chunk so it's self-describing and matches better (Anthropic Contextual Retrieval)
A RAG bot answers most questions but always whiffs on “what is the rate limit?” even though the
docs clearly state it. Where do you look first?
Hit these points: suspect context fragmentation — the fact (“limit is set to 5/min”) likely straddles a chunk boundary, so neither chunk is a complete, retrievable answer → inspect the actual chunks around that sentence, not the model output → fixes: add overlap so the fact survives whole; switch to structure-aware splitting so the sentence isn’t cut; apply contextual chunking so the chunk carries its section heading → verify by checking whether the right chunk now appears in top-k for that query — a retrieval check, not an answer check.
RAG — From First Principles
Retrieval-Augmented Generation is Lesson 1’s idea, shipped: take a question, retrieve the relevant chunks, assemble them into the window, and let the LLM answer — grounded in your data instead of its frozen weights.
The pipeline
Why RAG exists vs the alternatives
| Need | Stuff it all in the window | Fine-tune the model | RAG |
|---|---|---|---|
| Private / current data | Doesn’t fit; truncated | Stale at next data change | ✓ fresh, pulled per query |
| Citations / provenance | Buried in noise | ✗ weights can’t cite | ✓ returns the source chunk |
| Cost to update | Pay per token every call | Retrain run each time | ✓ reindex changed docs only |
Is RAG is the pattern of retrieving relevant external text at query time and putting it in the context window so the model answers from it — the productized form of “context is the new database query.”
Why it exists To ground the model in private, current, authoritative data; cut hallucination; enable citations; and do it cheaper and fresher than fine-tuning — without stuffing everything into a fixed window.
Like (world) An open-book exam. The student (LLM) is fixed, but pulling the right page before answering changes the grade. RAG is the “go fetch the right page” step.
Like (code) A read-through cache in front of a database: on each request you fetch the few relevant records and hand them to the handler. Retrieval is that fetch; the LLM is the handler.
Why so many RAG systems are poor
| The naive default | What it costs | The fix (maps to Module 2) |
|---|---|---|
| Fixed-size chunking | Fragmented, incoherent chunks | Structure-aware + contextual chunking |
| Embedding-only retrieval | Misses exact keywords/IDs | Hybrid: embeddings + BM25 lexical |
| Top-k, no reranking | Right doc buried below the cut | Rerank so the best chunk rises to top |
| No retrieval evaluation | You measure the answer, never the recall | Eval retrieval hit-rate, not just vibes |
Read the middle column. Every failure is upstream of the model: bad chunks, blind retrieval, no rerank, no measurement. Teams grade the answer and never check whether the right document was even retrieved — so they “fix the LLM” and nothing improves.
✓ Usually retrieval missed: the model got nothing or the wrong passage and answered confidently anyway. Garbage in → fluent garbage out. Fix recall before model.
✓ That’s the naive baseline. Without hybrid search and reranking, the relevant doc often sits below the cut (and “lost in the middle” even when included) — so it never reaches the model usefully.
📥 Memory rule: RAG is only as good as its retrieval. Most “the LLM is dumb” bugs are retrieval bugs.
Memory check
- Name the five stages of the RAG pipeline. → Question → Retrieval → Context assembly → LLM → Answer
- Why RAG over fine-tuning for current/private data? → fresh per query, can cite the source chunk, cheaper to update (reindex changed docs vs retrain), no stuffing the window
- Three reasons real RAG systems disappoint? → naive fixed-size chunking, embedding-only retrieval with no rerank, and no evaluation of retrieval quality (they measure the answer, not recall)
Sketch a docs-Q&A or support assistant as RAG, then name the three places it most often
breaks in production.
Hit these points: pipeline — chunk the help center / docs, embed + index, then per question retrieve top chunks, assemble into the window with the question, LLM answers with citations → break 1: chunking destroyed the fact (fragmentation / fixed-size cuts) → break 2: retrieval missed — embedding-only with no hybrid/rerank, so the right doc is buried below top-k or lost in the middle → break 3: stale index — the doc changed but wasn’t reindexed, or conflicting passages confuse the model → in all three the model is innocent; the fix is upstream, and you must measure retrieval hit-rate, not just answer quality.
Retrieval practice — test the three modules
Q1. The key difference between Cursor's and Claude Code's codebase strategy is that…
Q2. A known downside of a pre-built embedding index for code is that…
Q3. "Context fragmentation" in chunking refers to…
Q4. The five-stage RAG pipeline, in order, is…
Q5. A RAG bot confidently gives a wrong answer. The most likely root cause is…
Interview — pick your bar
Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).
What are the three discovery steps a tool runs before it assembles a codebase-context window?
Contrast Cursor's and Claude Code's strategy for understanding a large repo.
What is chunking, and what makes one chunk "good"?
Name the five stages of the RAG pipeline in order, and say what RAG is.
Your embedding-indexed assistant keeps citing a function deleted last week. What's wrong and how do you fix it?
Walk through chunk-size trade-offs, and explain how zero-overlap fixed-size splitting silently loses facts.
A RAG bot answers most questions but always whiffs on "what is the rate limit?" though the docs state it. Where do you look first?
An embedding-only RAG returns the right document for paraphrased questions but misses when users type an exact error code. Why, and what do you add?
At scale, how do you decide between a pre-built embedding index and read-on-demand — and when do you combine them?
A team says "our RAG is bad, the model hallucinates — we need a smarter model." Diagnose this as the senior in the room.
Make the case that retrieval quality matters more than model quality for a code assistant — then steelman the opposite.
- Clarify scale: corpus size vs window, query volume, freshness needs, latency & cost budget.
- Ingest & chunk: structure-aware splitting + overlap; contextual chunking so each chunk self-describes.
- Candidate generation: hybrid — semantic (embeddings) + lexical (BM25) for exact IDs/keywords.
- Rerank the top-N for precision; assemble within budget, ordered against lost-in-the-middle.
- Freshness: incremental reindex on change, TTL/invalidation, or read volatile/exact data live.
- Evaluation: measure retrieval hit-rate / recall@k as the gate, not just answer vibes.
- Failure modes & guardrails: retrieval miss, fragmentation, stale index, conflicting passages, no citation.
Design a RAG system for a help-center support bot over a changing knowledge base.
Design the code-context layer for a coding agent over a 2M-token monorepo with a 200k window.
These notes reflect my current understanding and are updated as I learn, build, and discover better explanations.