Reference · AI Agents · Stage 2

Reviewing an agent system — one-page cheat sheet

AI agent architecture-reviewAgent loopMulti-agentAIAgents

Published

For the design-review chair · print me · pairs with Lesson 5 (architecture reviews + catalogs) & Lesson 4 (M7–M9) & the glossary
The one sentence: an agent request is not a web request — it’s a long-lived, stateful session that fans out to a non-deterministic, sometimes-failing model and to privileged tools. Almost every risk below flows from that one structural fact.

The seven review lenses

LensThe question that finds the risk
ScalabilityWhat’s the binding limit at 10k users? (Hint: provider rate limits, not CPU.)
ReliabilityPer-step success rate → end-to-end task rate?
SecurityWorst single tool call? Control in runtime or prompt?
CostCost per completed task? Kill switch on runaway?
ObservabilityDebug a bad action from 3 days ago — how?
Context eng.Retrieval failure vs reasoning failure — tell them apart?
EvaluationWhat catches a 5% regression before it ships?

Numbers that reframe the review

RELIABILITY COMPOUNDS DOWN
0.99^40  ≈ 67%   (99% step, 40 steps)
0.999^40 ≈ 96%   → steps must be ~99.9%

COST IS DOMINATED BY INPUT TOKENS
context is RE-SENT every step →
session tokens grow ~quadratically
multi-agent ≈ 15× the tokens of a chat

THE BOTTLENECK
provider rate limits (TPM/RPM),
not your servers' CPU

Trust boundaries

TRUSTED    system prompt
UNTRUSTED  user input
UNTRUSTED  tool outputs / fetched content
(a web page, a DB row, a file)
ENFORCE    in the runtime / policy layer,
NEVER in prompt instructions.
The model cannot police its own boundary.

Architecture review checklist

  1. Loop durable/resumable (checkpointed), or work lost on crash?
  2. Where does session state live — can any worker resume?
  3. Provider rate limits managed (queue, multi-key, backpressure)?
  4. Per-tenant isolation — can one tenant starve the rest?
  5. Every tool: scope, credentials, worst call enumerated?
  6. Authorization in the runtime/policy layer, not the prompt?
  7. Idempotency on every mutating tool?
  8. Loop cap + wall-clock + token budget in the runtime?
  9. Full per-step traces (context + output + tool calls + cost)?
  10. Eval harness gating prompt/model changes?

Agent design checklist

  1. Smallest viable toolset; read and write agents split.
  2. Tools have typed schemas + arg validation.
  3. Retrieval hybrid + reranked; recall measured on its own.
  4. Context budget per section; history compacted.
  5. Model tiering — cheap to navigate, strong for the hard step.
  6. Prompt caching on the stable prefix.
  7. Irreversible actions gated by approval / HITL.
  8. Tool outputs tagged as data, never as instructions.

Production readiness checklist

  1. Token/cost budget + kill switch per session & tenant.
  2. Circuit breakers per tool and per provider.
  3. Retries: bounded, backoff + jitter, idempotency keys.
  4. Retryable-vs-not classified (retry 429/503/timeout; never 400/auth).
  5. Graceful degradation path (smaller model / cache / escalate).
  1. Immutable audit log for every mutation.
  2. Secret/PII redaction in logs and traces.
  3. Dead-letter + human-escalation queue.
  4. Cost attribution + alerting per tenant.
  5. Canary/rollback + multi-provider failover.

Common failure modes

Retry storms · duplicated side effects (charges/emails) from non-idempotent retries · runaway loop → unbounded spend · noisy-neighbor quota starvation · prompt injection via tool/RAG content → exfiltration or unauthorized action · over-broad token → mass deletion · retrieval recall miss → confident wrong answer · lost-in-the-middle · context exhaustion on long tasks · silent degradation · “can’t reproduce it” (no traces) · silent regression from a prompt tweak.

RED FLAGS in an agent architecture

”Guardrails” that are system-prompt sentences · one omnipotent API key or tool · sync request handling for long tasks · no per-step tracing or assembled-context capture · no eval set (“we test by trying it”) · unbounded retries / no loop cap · no idempotency on writes · always the biggest model, no tiering · full history re-sent with no compaction or caching · no per-tenant cost visibility · tool outputs concatenated straight into instruction context · “the model will know not to do that.”

Eight questions for the design-review chair

  1. ”Per-step success rate, and therefore end-to-end task rate?” forces 0.99^40 ≈ 67%
  2. “Enumerate every tool’s blast radius. For each worst case — control in the runtime or the prompt?"
  3. "Where does state live if a worker dies at step 20 of 30?"
  4. "Cost per completed task, and the kill switch on runaway spend?"
  5. "How do you tell a retrieval failure from a reasoning failure in a wrong answer?"
  6. "I change one prompt line — what catches a 5% regression before it ships?"
  7. "Debug a bad action from three days ago — without re-running it. Show me."
  8. "Noisy-neighbor story — can one tenant starve the rest?”