Question 1

Why is an AI feature "just a distributed system," and what's the one thing that's different?

Accepted Answer

Hit these points: it's a service calling a remote, rate-limited, occasionally-down dependency — so the model call gets the same treatment as any flaky API: timeouts, retries with backoff, circuit breakers, back-pressure → the one new variable is non-determinism: the same input can return a different, sometimes wrong, output → that's the delta you spend review time on; the rest of reliability theory is inherited, not reinvented → so the review is "run the usual lenses, then ask what non-determinism does to each."

Question 2

What does "reliability" mean for an AI feature, and why isn't "200 OK" enough?

Accepted Answer

Hit these points: reliability for AI = the probability of a useful, correct-enough answer within budget under real load and partial failure — not "the process is up" → a model that returns a confident wrong answer, hangs for 30s, or is rate-limited is an outage to the user even though the endpoint is healthy → so you carry two axes: availability/latency and quality → set an SLO on both and degrade gracefully — fallback model, shorter context, cached answer, or an honest decline — instead of hanging or shipping a wrong answer.

Question 3

What's the capacity unit for an AI feature, and where's the ceiling?

Accepted Answer

Hit these points: the unit is the token (input + output), not the request — you provision and bill in tokens the way you provision CPU and bill in RPS → the ceiling is usually a provider rate limit (tokens/min) you don't control, not your own pool → per-request cost and latency scale with length, so a few long-output requests can dominate → if self-hosted, the bottleneck is KV-cache memory, which grows with context length &times; concurrency → so capacity planning is queueing + admission control against an upstream limit.

Question 4

Name the fourth observability axis AI adds beyond logs, metrics, and traces.

Accepted Answer

Hit these points: a quality / eval signal — was the output correct, grounded, and safe → scored offline on a golden set and sampled online from production → classic signals go green while the model is confidently wrong, so without a quality axis you can't detect a bad answer → pair it with auditability: reconstruct the prompt, context, model+version, tools, and output for a given request id → standardize spans on the OpenTelemetry GenAI conventions so telemetry isn't vendor-locked.

Question 5

Your AI feature's p99 latency is 9s and users are abandoning. Where do you look first?

Accepted Answer

Hit these points: pull the trace for slow requests and decompose end-to-end latency: retrieval, prompt assembly, model time-to-first-token, generation time, tool calls → most AI tail latency is generation time, which scales with output length, so check whether slow requests just produce more tokens — then cap or stream output → separate provider-side latency (queueing behind a rate limit, a slow model version) from your own; the OTel GenAI span attributes tell you which → a long-output request behind a shared limit creates head-of-line blocking for everyone — add admission control and per-tenant budgets → mitigate: stream so time-to-first-token is the felt latency, hard-cap generation, fall back to a faster model near the limit.

Question 6

Name three failure modes an AI dependency adds that a normal HTTP dependency doesn't, and contain each.

Accepted Answer

Hit these points: one — silent wrong answer (hallucination): the call succeeds, the answer is confidently wrong; contain with grounding/citations, an eval/guardrail check, and surfacing uncertainty → two — provider rate-limit / throttle: traffic unchanged but a global or neighbor limit hit; contain with client-side rate limiting, backoff, queueing, and a fallback model → three — model-version drift: the provider updates the model and behavior regresses; contain by pinning the version where possible and monitoring the quality signal → bonus — latency cliff from long outputs and cost blowup from runaway tokens; contain with output caps and per-tenant budgets → theme: these are partial-failure modes — apply the usual timeout/retry/circuit-breaker discipline plus a quality gate.

Question 7

Why does the same input sometimes give a different output, and what does that change about review?

Accepted Answer

Hit these points: the model samples from a probability distribution over next tokens, so identical input can yield different output; provider-side model updates, sampling temperature, and floating-point non-determinism add variance → this is the one variable a normal service lacks, and it bends every lens: you can't assert exact-match in a test, a retry won't return the same answer, and a passing eval can regress silently after a version bump → so review adds: pin the model version, test on distributions and graded evals not exact strings, and treat output quality as a monitored, drift-prone signal → everything else is a normal distributed system.

Question 8

You inherit an AI feature with no SLOs. Define a reliability target and an error budget.

Accepted Answer

Hit these points: pick SLIs on both axes — availability/latency (fraction answered within a latency budget, e.g. p95 under N seconds) and quality (fraction graded correct-enough on a sampled/golden set) → set SLO targets per axis from product tolerance: a code assistant tolerates more wrong-but-cheap than a medical summarizer → the error budget is the allowed miss on each; burning availability means too slow/throttled, burning quality means the model regressed or retrieval degraded → alert on budget burn rate, not raw errors, and tie release/throttle decisions to remaining budget → the trap: measuring only availability — a feature can be 100% available and 40% wrong, which the user experiences as broken.

Question 9

Walk me through how you'd review a proposed AI feature the way you review any new service.

Accepted Answer

Hit these points: it IS a distributed system — a call to a remote, rate-limited, occasionally-down dependency — so run the usual lenses: reliability, scalability, availability, observability, cost, failure modes → add exactly one new variable: the dependency is non-deterministic, so the same input can return a different, sometimes wrong, output → that bends every lens — reliability now includes "correct-enough" not just "returned 200"; capacity is tokens/sec against a provider rate limit, not RPS against your pool; observability needs a quality axis → name the new dependencies: provider availability and token cost are now in your critical path and on your bill → close with the VP's question: what happens when the model is slow, wrong, rate-limited, or down — and can you see it.

Question 10

A team says "it's just an API call to the model, treat it like any other microservice." Where's that right, where does it break?

Accepted Answer

Hit these points: right — it is a remote dependency, so timeouts, retries with backoff, circuit breakers, bulkheads, idempotency, and back-pressure apply unchanged; don't reinvent reliability theory → where it breaks — the dependency is non-deterministic and can fail by being wrong, not just down, so a 200 with a confident hallucination passes every classic health check while harming the user → capacity is a token budget against a provider rate limit you don't own, not RPS against your pool, so saturation looks like throttling and latency cliffs, not CPU → observability needs a quality axis and per-request auditability a stateless microservice never needed → so: same skeleton, three deltas — correctness as reliability, tokens as capacity, quality as a monitored signal.

Question 11

Run a full architecture review of a proposed customer-facing AI summarization feature.

Accepted Answer

A strong answer covers: frame it as a distributed system with a non-deterministic remote dependency, then run the lenses in priority order → scope and blast radius: who sees it, cost of a wrong vs slow vs down answer, is the model in the critical path or can it be async → reliability: SLOs on availability and quality, a graceful-degradation ladder, timeouts/retries/circuit breaker around the provider → scalability: capacity in tokens/sec against the provider limit, queueing and admission control, output caps, per-tenant budgets, KV-cache limits if self-hosted → observability: OTel GenAI traces, latency/cost/token metrics, an eval signal for quality, request-level auditability → failure modes: hallucination, rate-limit, version drift, latency cliff, cost blowup — each with a containment → cost: tokens/request, caching, model tiering, kill switch → close on what a VP asks: what happens when it's slow, wrong, throttled, or down, and can you see it and roll back.

Question 12

Stress-test this: "a single shared model gateway in front of every team's AI feature, primary provider only, retry-on-error." Where does it hurt at scale?

Accepted Answer

A strong answer covers: the gateway shape is right — one place for auth, metering, telemetry, and policy — but the stated config concentrates risk → single provider, no fallback makes the provider a hard dependency: a regional outage or a global rate-limit event takes down every team at once, so add a fallback model and degrade rather than fail → naive retry-on-error amplifies a throttle into a retry storm and burns the shared rate limit; use bounded retries with jittered backoff, a circuit breaker, and shed load before the upstream limit → a shared limit with no per-tenant budget means one team's long-output batch starves everyone (noisy neighbor / head-of-line blocking); add per-tenant token budgets and admission control → retrying a non-deterministic call may return a different answer, so retries must be safe/idempotent at the feature level → observability and a per-tenant kill switch are mandatory because the blast radius is now everyone → net: keep the gateway, add fallback, back-pressure, per-tenant budgets, and isolation.

Dimension	Normal service	AI feature
Capacity unit	Requests / sec	Tokens / sec (input + output)
The ceiling	Your own pool / CPU	A provider rate limit you don’t control
Per-request cost	Roughly flat	Scales with input + output length
Self-hosted bottleneck	CPU / connections	KV-cache memory (context len × concurrency)
Saturation symptom	CPU pegged, queue grows	Provider throttling, latency cliff

Question to answer	Normal service	AI feature
Is it up / fast?	Metrics + traces	Same — plus token & cost metrics
Was it correct?	Mostly implied by 200	A separate eval/quality signal — the new axis
What did it do & why?	Request log	Full call tree: prompt, context, model+version, tools
Did behavior drift?	Rare (you ship code)	Common — provider can update the model under you

Reviewing an AI Feature Like a Distributed System

Reliability

Two axes, not one — availability AND quality

The graceful-degradation ladder — shed quality before availability

Scalability

The capacity unit is the token, and the ceiling is upstream

What changes vs a normal stateless service

Observability

Three signals + a quality axis + auditability

Why this lens is heavier for AI

Retrieval practice — test the three lenses

Interview — pick your bar