Question 1

Why is evaluation called the real moat for an AI product?

Accepted Answer

Hit these points: the model is a commodity — any competitor can call the same API → what you can't copy is a curated dataset of what "good" means for your task and a measured baseline → you cannot improve what you cannot measure: without a score, every change is a guess and progress is unprovable → "vibes-based" eval works at 5 examples and collapses at scale, where a small regression is thousands of bad answers no human will eyeball → evaluation turns model changes into measured before/after diffs — the only way to improve reliably.

Question 2

What is the difference between offline and online evaluation?

Accepted Answer

Hit these points: offline = score the system against a FIXED dataset (the golden set) BEFORE shipping; reproducible, runs in CI, gates a deploy — same inputs every run, like an integration test suite → online = measure quality on LIVE production traffic AFTER shipping; real distribution, real users → online has two shapes: A/B (split real users, they see the variant, you compare outcomes) and shadow (run the new system in parallel but discard output — no user risk, no live signal) → they're complementary: offline is repeatable but only covers what you captured; online is real but can't gate a deploy that hasn't happened.

Question 3

What is a golden set and how is it different from a benchmark?

Accepted Answer

Hit these points: a golden set is a versioned, curated set of (input, expected-output / acceptance-criteria) cases that is YOUR ground truth for offline eval — small, hand-checked, treated like test fixtures, changed only by deliberate review → a benchmark is a shared, public dataset + metric used to compare models against EACH OTHER (MT-Bench, HELM scenarios) → the benchmark is a standardized exam: good for cross-model comparison, weak as a proxy for your task → benchmarks rot via contamination (test data leaked into training = memorization, not capability) → a fresh private golden set beats a public leaderboard for trusting a result on your problem.

Question 4

What are the three ways to score an eval case, and when do you use each?

Accepted Answer

Hit these points: programmatic / exact-match — deterministic checks (regex, JSON schema, equals-expected, tool-succeeded); cheapest and most reliable, use whenever the output is structured or has one right answer → human grading — a person applies a rubric; the gold standard and the source of your gold labels, but slow and expensive, so reserve it for the golden set and audits → LLM-as-judge — an LLM scores or compares against a rubric; scales cheaply for open-ended quality, but biased and must be calibrated against human labels first → the stack: programmatic where you can, judge where you must, human to anchor both.

Question 5

Offline eval passes but production users complain. Diagnose.

Accepted Answer

Hit these points: the golden set only covers the distribution you thought to capture; live traffic drifts and hits inputs you never fixtured — green offline + unhappy users is a coverage gap → confirm by sampling the complaints and checking whether those inputs even exist in the golden set (usually they don't) → add online signals: thumbs-down rate, regenerate rate, escalation-to-human, task completion, plus a shadow or A/B comparison of old vs new on real traffic → close the loop: harvest the failing live cases, label them, fold them into the golden set so the next offline run catches that class → the fix isn't "trust offline less" — offline and online cover different risks and you need both.

Question 6

When do you use shadow deployment vs an A/B test for online evaluation?

Accepted Answer

Hit these points: shadow = run the new system on real traffic in parallel and THROW THE OUTPUT AWAY; users never see it, so zero user risk — use it when the change is risky, hard to roll back, or you only need to compare behavior, not measure conversion → A/B = split real users between old and new; users SEE the variant, so you get a real outcome signal (conversion, completion, satisfaction) but you've exposed some users to a possibly-worse experience → the trade: shadow gives safety but no business signal; A/B gives the business signal at real user cost → common sequence: shadow first to confirm no regressions, then A/B to measure the real lift.

Question 7

Your LLM-as-judge prefers the new prompt. Should you trust it? Argue both sides.

Accepted Answer

Hit these points: the judge is a model with documented biases — position, verbosity, self-preference — so a preference can be an artifact, not real quality → trust it ONLY to the degree you've calibrated it: measure agreement against a human-labeled slice; a strong judge can reach roughly human-level agreement on some tasks, but that's dataset-dependent and must be re-verified → mitigations before trusting a gate: randomize order, force a rubric/CoT, use a different model family as judge, include reference answers → synthesis: the judge is a cheap noisy sensor that lets you scale human judgment to thousands of cases — use it to flag, confirm close calls with humans, never let an uncalibrated judge alone decide a ship.

Question 8

A teammate wants to grow the golden set by dumping in every production log line. Push back.

Accepted Answer

Hit these points: the golden set is fixtures, not a data lake — its value is that every case is representative, labeled, and hand-checked, so a score on it means something → raw logs are unlabeled, redundant, and noisy; dumping them in inflates the suite, slows every run, and dilutes the signal without adding coverage → it also breaks the "changed only by deliberate review" property — silent growth means you no longer know what your baseline measures → the right move: sample from logs, especially failures and edge cases, label them, and add the ones that cover a class you don't already test → quality and coverage of the set beat raw size every time.

Question 9

Map the full eval loop end to end: from a failing user report to a regression test that protects against it.

Accepted Answer

Hit these points: collect — gather representative cases, including failures harvested from production → score — apply the right grader per case (programmatic / human / LLM-judge) → diff — compare the aggregate score against the baseline to see better or worse → decide — ship if it's a measured win, roll back if it's a regression; offline gates the merge, online (A/B / shadow) confirms on live traffic → feed back — every production failure becomes a new labeled case in the golden set, so the next offline run catches that class → the loop is the point: it converts one-off failures into permanent regression coverage and makes quality a ratchet, not a coin flip → measure each stage (coverage, judge-human agreement, regression catch-rate) so the loop itself is trustworthy.

Question 10

One eval system, three forces — speed, trust, coverage — that pull against each other. How do you reason about the trade-offs?

Accepted Answer

Hit these points: name the three: programmatic/judge graders give SPEED, human labels give TRUST, online + golden-set growth give COVERAGE → the tension: the fast graders (judge) are the least trusted, the trusted grader (human) is the slowest, and the most realistic signal (online) can't gate a pre-ship decision → sequence it: gold-label a small golden set with humans (trust), automate the bulk with programmatic + calibrated judge (speed), grow coverage by folding online failures back in → never trade an unmeasured risk for a measured saving: a cheap but uncalibrated judge is worse than no gate because it ships regressions confidently → measure the loop itself — judge-human agreement, offline coverage of real traffic, regression catch-rate — so you know which force is weakest.

Question 11

"We hit 90% on the public benchmark, so we don't need our own eval." Tear this apart as a system design.

Accepted Answer

Hit these points: two unsafe assumptions → "benchmark = our task" is false: a public benchmark measures a generic task on shared data — it's a capability proxy, not a measure of YOUR inputs and YOUR success criteria → "a high score = real skill" is suspect: contamination means leaderboard test data leaks into training, so the rank can be memorization; benchmarks also saturate and rot as new models arrive → the right design: a private golden set of your real inputs + your graders gates the ship; benchmarks only shortlist candidate models cheaply → then confirm with shadow/A/B on live traffic → the framing: the leaderboard says a model is generally capable; only your eval says it's right for the job — teach the method, not the rank.

Question 12

Design an evaluation strategy for a new LLM feature, from zero to a CI gate plus production monitoring.

Accepted Answer

A strong answer covers: define success criteria per task first — measurable pass/fail, not "good" → build a small versioned golden set of real, representative (input, expected) cases, hand-labeled by humans → choose graders: programmatic where structured, calibrated LLM-as-judge where open-ended, human for the gold labels and audits → stand up the harness (dataset + runner + grader + report) and establish a baseline number → wire offline regression eval into CI so a drop below baseline blocks the merge → ship behind shadow first (no user risk), then A/B to measure real lift → instrument production signals (thumbs-down, regenerate, escalation, task success) and route failures back into the golden set → map risks: benchmark contamination, judge bias, coverage gaps; measure the loop, not just the model.

Question 13

A leader says "just use the public leaderboard score to pick our model." Tear this apart and propose what you'd do instead.

Accepted Answer

A strong answer covers: a public benchmark measures a generic task on shared data — it's a proxy for capability, not a measure of YOUR task → contamination: leaderboard test data leaks into training sets, so a high rank can be memorization, and leaderboards saturate and rot as new models arrive → the right move is a private golden set of your real inputs with your success criteria, scored by your graders — that's the number that predicts production → use benchmarks only to shortlist candidate models cheaply, then decide with your own offline eval and confirm with shadow/A/B on live traffic → the framing: the leaderboard says a model is generally capable; only your eval says it's right for the job — teach the method, not the rank.

Grader	How it works	Use it when
Programmatic	Deterministic check: exact-match, regex, JSON schema, tool succeeded	Output is structured or has one right answer — cheapest, most reliable
LLM-as-judge	An LLM scores / compares outputs against a rubric	Open-ended quality at scale — cheap, but biased; calibrate first
Human	A person applies a rubric	Gold labels & audits — the trusted source, but slow and costly

Dimension	Golden set (yours)	Benchmark (public)
Purpose	Is the system right for your task?	Compare models against each other
Data	Private, curated, your real inputs	Shared, public (MT-Bench, HELM)
Trust	High — you hand-checked it	Rots: contamination, saturation
Role	Gates your ship	Shortlists candidate models

Dimension	Offline	Online
Data	Fixed golden set	Live production traffic
When	Before ship (gates the merge)	After ship (measures reality)
Repeatable?	Yes — same inputs every run	No — traffic drifts constantly
Blind spot	Only what you put in the set	Can’t gate a deploy that hasn’t happened

Evaluation Fundamentals: Offline & Online

Why Evaluation Is the Moat

Vibes don’t aggregate — they collapse at scale

Offline Evaluation

The harness: dataset → runner → grader → report

Three ways to score a case

Golden set vs benchmark — don’t confuse them

Online Evaluation & Closing the Loop

Shadow vs A/B — safety vs signal

LLM-as-judge — a cheap sensor you must calibrate

Offline ↔ online — complementary, not rivals

Retrieval practice — test the three modules

Interview — pick your bar