Lesson 4 · Stage 2: Systems & Production · visual edition

Multi-agent Systems & Production

~35 min · 3 modules + capstone · multi-agent · production concerns · business agent design

Published

Multi-agent systemsProduction AI agentsAgent loopMulti-agentAI

Your bar: decide when multi-agent is justified vs unnecessary; name five ways production agents fail and how to prevent them; design a business agent (billing, support, or SRE) with the right tools, oversight model, and cost controls.

Multi-agent = microservices for agents

production agents fail from loop, cost, and security gaps

business agents need the right tools + oversight

hype says autonomous; reality says HITL

7

Multi-Agent Systems — microservices for agents

One agent, one context window. Multi-agent = task decomposition + specialization + parallelism. Justified only when you’d split a microservice.

SINGLE AGENT User Agent LLM + Tools Result MULTI-AGENT User Coordinator decomposes + delegates Research Agent Analysis Agent Writer Agent Final Result
Single agent: simple, predictable, cheap. Multi-agent: parallelism and specialization, at the cost of orchestration complexity and harder debugging.

Is A system where a Coordinator Agent decomposes a task and delegates subtasks to specialist Worker Agents, collecting and synthesizing their outputs. Each agent has its own context window and toolset.

Why it exists A single agent’s context window is a hard limit. Some tasks are too large to fit in one context or require parallelism. Multi-agent splits the work across isolated contexts running concurrently.

Like (world) A consulting engagement with a project manager (coordinator) directing a research analyst, a financial modeler, and a writer (workers). Each specialist works independently in parallel; the PM synthesizes the final report.

Like (code) Microservices vs a monolith. The coordinator is the API gateway / orchestrator. Workers are specialized services. Communication is structured outputs, not shared memory.

✗ “Multi-agent is always better — more agents = more intelligence”
✓ Multi-agent adds coordination overhead, harder debugging, and compounded failure modes. Use it only when a single agent’s context or capability is genuinely insufficient.
🤝 Memory rule: Multi-agent = justified only when you’d split a microservice. Coordination overhead is real. Prefer the monolith (single agent) until you hit the wall.
ScenarioUse multi-agent?Why
Task fits in one context windowNoSingle agent is simpler and debuggable
Tasks are genuinely parallel (no dependencies)YesParallelism reduces wall-clock time
Different tasks need different toolsetsYesWorker specialization; cleaner context
Task requires full repo analysis (>context limit)YesDecompose by module/file cluster
Sequential steps with shared stateNoSingle agent loop handles this naturally
You want “more intelligence”NoThat’s a model quality problem, not architecture
When would you choose multi-agent over a single agent for a coding task?
When the task is genuinely decomposable into independent parallel subtasks (e.g. analyze 20 files simultaneously); when subtasks need different toolsets (research agent uses web search, writer agent uses file system); when a single context window is too small for all the information needed. Not justified: sequential steps, tasks that fit in one context, or “we want it smarter” — that’s a model choice, not an architecture choice.
What are the main failure modes unique to multi-agent systems vs single-agent?

(1) Coordinator failure — misinterprets worker output, sends wrong context to next worker.
(2) Worker isolation — worker doesn’t have enough context to do its job because the coordinator under-specified.
(3) Compounded errors — one worker’s bad output propagates to downstream workers without correction.
(4) Debugging difficulty — which agent made the mistake?
(5) Cost amplification — N agents × M iterations = N×M LLM calls, all billed.
(6) No shared state — workers can’t see each other’s work unless the coordinator explicitly passes it.

Memory check
  • What analogy maps coordinator agent → worker agent to a software architecture pattern? → API gateway → microservice; or project manager → specialist consultant.
  • Name two scenarios where multi-agent is NOT justified. → Task fits in one context; sequential steps that share state; "want more intelligence" (wrong lever).
  • What is the primary coordination cost of multi-agent? → The coordinator must assemble, pass, and synthesize context for each worker — context assembly and output synthesis failures are new failure modes that don't exist in single-agent.
8

Production Engineering — where agents break in the real world

An agent that works in a demo will fail in production. Infinite loops, $500 bills, and silent data corruption are the real challenges.

Agent in Production Infinite Loop Loop cap Cost Explosion Token budget Silent Failure Error logging Security Escalation Least privilege Data Corruption Dry-run mode No Audit Trail Struct. log
Every production agent needs answers to all six before launch. Missing one is how a $10/month demo becomes a $5,000 incident.

Is The set of engineering controls — security, observability, cost control, reliability, and auditability — that distinguish a production agent from a demo. All of these are operational concerns, not model concerns.

Why it exists LLMs are non-deterministic and context-sensitive. Without controls, an agent can loop, overspend, silently corrupt data, or be manipulated by adversarial input. Production requires every one of these addressed.

Like (world) A new employee with company credit card access. You give them expense limits, require receipts, audit the card monthly, and revoke access if something looks wrong. You don’t give unlimited authority on day one.

Like (code) Any stateful distributed service with external I/O. You instrument it (metrics/traces/logs), rate-limit it, add circuit breakers, require auth for writes, and test failure modes — because silent failures in production are more dangerous than loud ones.

✗ “Production concerns can be addressed after the agent works”
✓ Security, cost controls, and observability must be designed in from the start — retrofitting them after a $500 incident is always more expensive and often incomplete.
🔒 Memory rule: Production agents need six controls: loop cap, token budget, least privilege, dry-run mode, structured logging, and a circuit breaker. Missing one is a production incident waiting to happen.

Security + Permissions

Principle of least privilege: the agent should only have the permissions it needs for the current task. Never give a read-only reporting agent write access. Prompt injection is real: adversarial content in tool results (e.g. a web page the agent fetched) can override instructions. Defense: validate tool outputs, use a system prompt that ignores instruction-like text in user/tool content, and sandbox risky tools.

Observability + Auditability

Every tool call should emit a structured log: timestamp, tool name, inputs, outputs, agent ID, session ID, cost. This is your audit trail. Without it you cannot debug why the agent made a decision, comply with audits, or detect abuse. Use OpenTelemetry spans or equivalent. Never log PII in tool inputs/outputs.

Cost Control

Token budget per session (hard cutoff). Model tiering: use a smaller/cheaper model for simple steps, reserve the large model for complex reasoning. Cost tracking per user/tenant for SaaS billing. Alert on anomalous spend (3× baseline = incident). For multi-agent: N agents × M iterations compounds fast — set per-agent budgets, not just session budgets.

Reliability + Retries

Exponential backoff for transient tool failures (network timeout, rate limit). Max retry cap (3 attempts, then fail fast with a clear error — not silent degradation). Idempotent tool design: if a tool call is retried, it should not double-charge or double-insert. Circuit breaker: if a tool fails repeatedly, disable it for the session and report to the user.

Loop Prevention

Max iteration cap (e.g. 20 steps). Detect repetition: if the last 3 LLM outputs are semantically identical, the agent is stuck — halt. Human-in-the-loop interrupt: allow the user to pause/stop at any step. Time-box: wall-clock timeout independent of iteration count (some steps are slow, not infinite).

Failure Modes (why agents get stuck)

(1) Context saturation: the context window fills with tool outputs; the agent loses the original goal. Fix: periodic compression or goal re-injection. (2) Tool error spiral: a tool returns an error; the agent retries infinitely without changing its approach. Fix: max retries + different strategy on retry. (3) Hallucinated tool calls: the agent calls tools that don’t exist or with wrong arguments. Fix: strict tool schema validation at the runtime level. (4) Ambiguous task: the agent asks a clarifying question but the user isn’t present to answer. Fix: require task completeness upfront; use async human-in-the-loop for long tasks.

You’re launching a SaaS billing agent that can apply credits, retry charges, and update subscription tiers. What production controls do you put in place before launch?

(1) Least privilege — read-only by default; write operations require explicit capability grant per session.
(2) Dry-run mode — all write operations execute a dry run first and show proposed changes before confirming.
(3) Loop cap — max 15 iterations per task; halt and notify if exceeded.
(4) Cost cap — max $2 in LLM tokens per billing task; alert at 50% of budget.
(5) Structured audit log — every tool call logged with inputs, outputs, user ID, session ID, timestamp — immutable, append-only.
(6) Idempotency keys on all write tools: retried calls are no-ops.
(7) Human approval gate for changes above a threshold (e.g. credits > $100 require human confirmation).
(8) Prompt injection defense — tool outputs are tagged as [TOOL RESULT] and the system prompt instructs the model to treat them as data, not instructions.

How do you detect and prevent an infinite loop in a production agent?

(1) Hard iteration cap (e.g. 20 steps) — the runtime counts and halts.
(2) Semantic repetition detector: if the last N tool calls are identical (same tool, same args), the agent is looping — halt with an error.
(3) Wall-clock timeout: even slow legitimate steps shouldn’t take more than X minutes — hard cap independent of iteration count.
(4) Progress check: every K steps, the agent summarizes what it has accomplished; if no progress detected, escalate to human.
(5) Dead-man switch: a background timer that triggers a halt if the agent doesn’t emit a “still working” heartbeat within Y seconds.

Memory check
  • Name the six production controls every agent needs. → Loop cap, token budget, least privilege, dry-run mode, structured logging, circuit breaker/retry policy.
  • What is prompt injection and how do you defend against it? → Adversarial instructions embedded in tool results (e.g. a fetched webpage) that override system instructions. Defense: tag tool outputs as data not instructions; validate/sanitize; system prompt tells the model to ignore instruction-like content in tool results.
  • Why must idempotent tool design be a requirement for any agent with write access? → The agent runtime retries failed tool calls; without idempotency, a retry causes a double-charge, double-insert, or duplicate action — no explicit dedup mechanism in the loop catches this.
9

Business Agent Architecture — designing for the org chart

An agent is a job description + a toolset + an autonomy level. Get those wrong and ROI evaporates, or you create a liability.

Same Agent Loop — Different Tools + Job Description Billing Agent $ Support Agent 💬 SRE Agent 🔧 PM Agent 📋 Engineering Agent ⌨
Five roles, one runtime, one loop. What differs: the tools registered, the permissions granted, and the human-oversight level required.

Is The practice of designing agents for specific business roles by choosing the right toolset, permission level, autonomy setting (full-auto vs human-in-the-loop), and success metric for that role’s risk profile.

Why it exists Generic “do anything” agents are dangerous and expensive. A Billing Agent and an SRE Agent need different write permissions, different oversight thresholds, and different definitions of “done”. Role-specificity is the unit of deployment.

Like (world) An employee’s job description. You don’t give the same authority to an intern and a CFO — even if they use the same office software. The role defines permissions, escalation paths, and accountability.

Like (code) A service account with a specific IAM policy. The agent is the service account; the tools are the allowed API calls; the HITL threshold is the MFA requirement for sensitive operations.

✗ “A general-purpose agent can replace all specialist agents with the same prompts”
✓ Different business roles carry different risk profiles, compliance requirements, and tool access patterns — role-specific agents are safer, cheaper to audit, and easier to improve.
📋 Memory rule: Agent = job description + toolset + autonomy level. Design each as you would a role on the org chart: clear scope, right permissions, defined escalation path.
AgentResponsibilitiesToolsAutonomyKey RiskHuman Oversight
BillingApply credits, retry charges, update plans, generate invoicesStripe API, billing DB read/write, emailSupervised: approve writes >$50Double-charge, complianceApproval gate on all write ops above threshold
SupportAnswer tickets, look up order status, issue refunds ≤$25Ticket system, CRM read, refund API (capped), KB searchSemi-auto: low-value actions auto; escalate complexWrong refund, policy violationEscalation queue for anything outside KB
SRERead logs, restart services, scale replicas, create incidentskubectl, CloudWatch, PagerDuty, SlackSupervised: reads auto; writes require confirmService outage from bad actionApproval for any destructive action (delete, scale-down)
Product ManagerDraft PRDs, create Jira tickets, summarize user feedback, update roadmapsJira, Confluence, Slack read, MixpanelHigh autonomy for drafts; low for publishingStale/incorrect specs shippedHuman review before any public artifact is published
EngineeringRead code, suggest fixes, run tests, open PRsFilesystem read, git, Bash (sandboxed), GitHub APIHigh for read+suggest; supervised for mergeBad code merge, security vulnHuman code review and approval before any merge
Your company wants an autonomous SRE agent that can respond to incidents without human approval. What are your concerns and what controls would you require?

Concerns: destructive actions (delete pods, scale to 0, drain nodes) on wrong targets; misdiagnosis of root cause leading to wrong remediation; cascading failures from bad automation.

Required controls:
(1) Read-only phase first — gather all context before any writes.
(2) Dry-run mode for all proposed remediations — show the exact commands before execution.
(3) Blast-radius estimation: if the proposed action affects >N% of capacity, require human approval.
(4) Rollback capability: every write action must have a defined undo.
(5) Auditability: every action logged to an immutable incident timeline.
(6) Human-in-the-loop for anything P1/P0. Start with supervised; earn full autonomy over time on a limited action set.

How do you calculate the ROI of a business agent and what are the most common ways agent ROI is overstated?

Real ROI = (time saved × hourly rate) + (error reduction × cost per error) - (agent cost + build cost + ops cost).

Commonly overstated because:
(1) Build cost is underestimated — prompting, testing, safety controls, and ops tooling are all non-trivial.
(2) Human oversight cost is forgotten — supervised agents still need a human in the loop, just a less skilled one.
(3) Error rates in production are higher than in demos — each LLM error has a cost.
(4) Model costs at scale are not linear — token usage grows with context and multi-agent amplification.

Honest ROI requires measuring production error rate, oversight cost, and total LLM spend — not just demo throughput.

Memory check
  • What three things define a business agent's role? → Toolset (what it can call), autonomy level (how much it can do without approval), and job description (what it's responsible for).
  • Which business agent type carries the highest risk of a production incident? → SRE agent — it can execute destructive infrastructure operations; billing agent is second (financial impact).
  • Why is a "general-purpose do-anything agent" a risky design for enterprise? → Broad permissions amplify blast radius; no clear audit scope; harder to comply with SOC2/GDPR audit requirements; no clear escalation path.

★ Stage 2 executive summary

What you now understand (Stage 2)

Nine modules. Three core insights to carry into every architectural review:

  1. The runtime is not the LLM. The LLM predicts. The runtime orchestrates — loop, tools, state, cost, safety. When an agent behaves badly, look at the runtime first.

  2. Context is the lever. A weaker model with the right context beats a stronger model with noise. Context engineering (retrieval, ranking, assembly, compression) is where the most leverage lives in production.

  3. Autonomy is earned, not assumed. Production agents need six controls before launch (loop cap, token budget, least privilege, dry-run, structured logging, circuit breaker). Multi-agent adds complexity — justify it like you’d justify splitting a microservice. Business agents are job descriptions with IAM policies.

The one diagram to draw

User → Runtime (loop/cost/safety) → LLM (stateless text prediction) → Tools (MCP servers) → back to Runtime. Context selection happens before the LLM call. Memory (3 tiers) is what the runtime injects into the context. That’s the whole machine.

Retrieval practice — Modules 7–9

Q1. Multi-agent is justified when...

Q2. The primary reason production agents create infinite loops is...

Q3. Least-privilege in agent design means...

Q4. The right autonomy level for an SRE agent performing destructive operations is...

Q5. A Billing Agent that issues a refund must have idempotent tool design because...

Interview — pick your bar

Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).

Define a single-agent system versus a multi-agent system, and name the two roles in the multi-agent pattern.
Hit these points: an agent = an LLM that uses tools in a loop, driven by an agent runtime (the orchestration layer — loop, state, tool-exec, retries, limits) → a single-agent system is one agent, one context window, one loop → a multi-agent system decomposes the task across several agents, each with its own context and toolset → the standard shape is orchestrator (lead agent) + subagent (worker): the orchestrator breaks the task down, hands separable subtasks to subagents, then synthesizes their structured outputs → subagents don't share memory — they communicate only through what the orchestrator passes.
What does the orchestrator (lead agent) do, and what does a subagent (worker) do?
Hit these points: the orchestrator owns decomposition and delegation — it reads the goal, splits it into subtasks, assembles the context each subagent needs, dispatches them, and synthesizes the results into a final answer → a subagent executes one focused subtask with its own isolated context window and a narrow toolset, then returns a structured output → the analogy is a project manager directing specialist consultants, or in code an API gateway fronting microservices → the orchestrator never relies on shared state — it must explicitly pass anything a subagent needs to know.
Distinguish a workflow from an agent, and say where memory fits in an agent system.
Hit these points: a workflow is a developer-defined path — the sequence of steps is fixed in code → an agent is a model-influenced path — the LLM decides which tool to call next and when to stop, inside limits the runtime enforces → so "more autonomy" means handing more of the control flow to the model → memory = persisted state reachable only via retrieval; it is not in the context window by default → the runtime injects retrieved memory into the context for a call, the same way it injects retrieved documents — nothing the agent "remembers" exists to the model unless it's packed into this call's window.
Name three production controls every agent needs before launch, and what each one prevents.
Hit these points: loop / step cap (e.g. 20 steps) — stops an agent that retries forever without making progress → token / spend ceiling per session — caps cost so a stuck agent can't run up a $500 bill, ideally with an anomaly alert at ~3× baseline → least privilege — the agent gets only the permissions the current task needs, limiting blast radius if it's manipulated or misfires → honorable mentions: structured per-step tracing for observability, idempotency on write tools, and a human-in-the-loop gate for high-risk actions → all of these live in the runtime, not the prompt.
When does multi-agent genuinely help, and when is it just "more microservices" cargo-culting?
Hit these points: it helps when subtasks are genuinely independent and parallel (fan out over 20 files at once to cut wall-clock time) or when context is separable and exceeds one window, so isolating each subagent's context keeps it clean → it hurts when the work is sequential and shares state — a single agent loop handles that more cheaply and is far easier to debug → the cargo-cult tell is "let's add more agents to make it smarter": that's a model-quality lever, not an architecture lever → every split you add buys you coordination overhead, harder debugging ("which agent broke?"), and compounded failure modes → the senior framing: justify a new agent exactly like you'd justify splitting a microservice — only when a real boundary (independent work, separable context, distinct toolset) demands it.
Why is cost the hidden killer of multi-agent systems, and how do you reason about it?
Hit these points: token cost compounds — every step re-sends the growing context, so cost rises super-linearly with steps, and multi-agent multiplies that by the number of agents (N agents × M iterations ≈ N×M LLM calls, numbers illustrative) → a single orchestrator turn can fan out into dozens of subagent calls, each carrying its own assembled context → so the unit to track is cost per completed task, attributed per agent and per tenant, not just per call → controls: per-agent spend ceilings (not only a session budget), model tiering (cheap model for routing/navigation, the strong model only for the hard step), prompt caching on the stable prefix, and history compaction so re-sent context doesn't grow unbounded → close the loop with an anomaly alert at ~3× baseline so a runaway fan-out trips an incident before the invoice does.
How do you keep an agent from running forever or overspending in production?
Hit these points: the runtime — not the prompt — enforces the limits → hard step cap plus a wall-clock timeout independent of step count, since some legitimate steps are slow → semantic repetition detector: if the last N tool calls are the same tool with the same args, the agent is stuck — halt → token / spend ceiling per session with a hard kill switch, plus an anomaly alert at ~3× baseline so spend trips an incident early → progress check: every K steps require the agent to summarize what it accomplished; no progress → escalate to a human → root causes to design against: context saturation (tool output crowds out the goal — fix with compression / goal re-injection) and tool-error spirals (retrying the same failing call — fix with bounded retries and a different strategy on retry).
A wrong or stuck multi-agent run lands on you. Why is it harder to debug than a single agent, and what makes it tractable?
Hit these points: the failure could be in any agent and the obvious symptom is downstream of the real cause — a subagent under-specified by the orchestrator, an orchestrator that misread a worker's output, or one worker's bad output propagating uncorrected to the next → there's no shared state to inspect, so you can't just read one log → what makes it tractable is per-step tracing: capture the assembled context, the output, the tool calls, and the cost for every agent at every step, tagged with agent ID and session ID → then you can replay the exact inputs each subagent saw and locate which boundary handed off wrong context → the prevention is designing for it up front — structured, immutable per-step logs are the multi-agent equivalent of a stack trace, which the architecture doesn't give you for free.
Design a production multi-agent system with guardrails, observability, and cost controls. What's the skeleton?
Hit these points: start from the boundary — only go multi-agent if the work is genuinely parallel or context is separable; otherwise a single agent is the right default → topology: one orchestrator (lead agent) that decomposes and synthesizes, plus narrow subagents (workers) each with an isolated context and a least-privilege toolset → guardrails (in the runtime): per-agent step caps + wall-clock timeouts, idempotency keys on every write tool, least-privilege credentials per agent, HITL approval gates on high-risk / irreversible actions, prompt-injection defense (tool outputs tagged as data, never instructions) → cost controls: per-agent and session token ceilings with a kill switch, model tiering, prompt caching on stable prefixes, history compaction, cost attributed per task and per tenant with an anomaly alert → observability: full per-step traces (assembled context + output + tool calls + cost) keyed by agent and session, immutable audit log for every mutation → reliability: bounded retries with backoff + jitter, circuit breakers per tool and per provider, a defined degradation path (drop to single agent / cache / escalate to human) → name the through-line: an agent request is a long-lived, stateful, non-deterministic session over privileged tools, and almost every control follows from that.
An engineer proposes "let's add more agents" to fix quality. Challenge it the way you'd challenge "let's add more microservices."
Hit these points: first separate the symptom from the cause — "quality is bad" is usually a context or model-capability problem, and splitting into more agents fixes neither; it just distributes the same defect across more moving parts → make the cost of the split explicit: each new agent adds coordination overhead, a new context-handoff that can drop information, harder debugging, and compounded failure modes — exactly the bill you pay for premature microservices → ask the microservices questions: what's the real boundary here? is this work genuinely independent and parallel, or is it sequential with shared state (where one agent loop is simpler and cheaper)? → point at the cost math: N agents × M iterations multiplies token spend, and re-sent context compounds per step → the staff move is to redirect to evidence: instrument the failing cases and attribute them (wrong context retrieved vs. weak reasoning) before changing the architecture → default to the monolith (single agent) and earn each split with a concrete boundary, the same discipline you'd demand before carving out a service.
"Autonomous agents will replace the team — ship it fully auto." Make the principal-level call on autonomy.
Hit these points: reframe autonomy as earned, not assumed — it's a dial set per action by blast radius, not a property of the whole system → reads and analysis can run auto; irreversible or high-impact writes sit behind a human-in-the-loop gate until the data justifies loosening it → do the compounding-reliability math out loud: even 99% per-step success over 40 steps is ~0.99^40 ≈ 67% end-to-end, so "fully autonomous" over a long task is quietly unreliable (numbers illustrative) → price in the costs the hype hides: supervised agents still need a human (just a less-skilled one), production error rates beat demo rates, and token cost at scale isn't linear → the principal position: start supervised on a limited action set, instrument success rate and cost per task, and widen autonomy only where measured reliability and a bounded blast radius support it — with an audit trail and a kill switch the whole time → that's the difference between a demo that impressed a VP and a system you can put your name on.
Design-round framework — drive any production agent design through these, out loud:
  1. Clarify the job: the agent's responsibilities, the success metric, and the risk profile of its actions.
  2. Single vs multi-agent: is the work sequential (one agent) or genuinely parallel / separable-context (orchestrator + subagents)? Justify any split.
  3. Tools & permissions: smallest viable toolset, least privilege, read and write agents separated, typed schemas with argument validation.
  4. Guardrails: step cap + wall-clock timeout, idempotency on writes, HITL gates on high-risk actions, prompt-injection defense (tool outputs as data).
  5. Cost controls: per-agent + session token ceilings with a kill switch, model tiering, prompt caching, history compaction, anomaly alerts on spend.
  6. Observability: full per-step traces (assembled context + output + tool calls + cost), keyed by agent/session; immutable audit log for every mutation.
  7. Failure modes & degradation: loops, tool-error spirals, context saturation, compounded multi-agent errors — and the fallback path (cache / smaller model / escalate to human).
Design a production SRE incident-response agent — tools, guardrails, and observability — for a team that wants it to triage and remediate alerts.
A strong answer covers: scope the job first — triage and remediate a bounded set of alert types, with a clear success metric (MTTR reduction without new incidents caused) → single agent is the right default: incident response is largely sequential with shared state; only fan out to subagents for genuinely parallel evidence-gathering (logs, metrics, traces in parallel), then synthesize → tools, split by risk: read-only first (logs, dashboards, deploy history, kubectl get) with broad access; write tools (restart, scale, rollback, page) behind least-privilege credentials and never auto on day one → guardrails: read-only diagnosis phase before any write; dry-run every proposed remediation showing exact commands; blast-radius estimation — if an action touches >N% of capacity or any P0/P1 system, require human approval; every write has a defined rollback; step cap + wall-clock timeout so a confused agent can't thrash production → cost controls: spend ceiling per incident with a kill switch, model tiering (cheap model to classify the alert, strong model only for root-cause reasoning), anomaly alert if an incident's token spend spikes → observability: every action streamed to an immutable incident timeline — timestamp, tool, inputs, outputs, agent/session ID, cost — which doubles as the audit trail and the post-mortem record → autonomy ramp: start fully supervised, measure suggestion-accuracy and false-remediation rate, and only widen auto-remediation on the specific low-blast-radius actions the data proves safe → name the trade-off: full autonomy cuts MTTR but risks a wrong action amplifying an outage, so reads earn autonomy fast and destructive writes earn it slowly.
Design a customer-support agent fleet for a SaaS product — multiple agents, guardrails, and cost controls under real traffic.
A strong answer covers: justify the fleet before building it — a single support agent handles most tickets; go multi-agent only where boundaries are real (a triage/orchestrator agent routes, specialist subagents handle billing, technical, and account questions with distinct toolsets and permissions) → orchestrator (lead) + subagents (workers): the router classifies intent and dispatches; each specialist runs an isolated context with a least-privilege toolset (KB search read-only; refund tool capped, e.g. ≤$25 auto; account writes gated) → guardrails: refunds/credits above a threshold require a human approval gate; idempotency keys on every write so a retried refund isn't issued twice; prompt-injection defense since ticket text and fetched pages are adversarial input — tag tool outputs as data, not instructions; per-customer scoping on every retrieval so one account's data never leaks into another's context → cost controls: token ceiling per ticket and per tenant with a kill switch, model tiering (cheap model for routing and FAQ, strong model for ambiguous cases), prompt caching on the stable policy/system prefix, history compaction so long threads don't re-send unbounded context, cost attributed per tenant with an anomaly alert → observability: per-step traces (assembled context + output + tool calls + cost) keyed by agent/session/customer; immutable audit log for every refund or account change → failure handling: low confidence or outside-KB → escalate to a human queue rather than hallucinate; a dead-letter path for tickets the fleet can't resolve → name the trade-off: a fleet of specialists is safer to audit and cheaper per ticket than one omnipotent agent, but the orchestrator's routing and context-handoff become the new failure surface to trace and test.

★ Stage 2 architecture cheat sheet

AI Buzzword → Engineering translation

BuzzwordEngineering equivalentOne-line test
LLMStateless text-prediction function”Does it have memory?” No. “Does it act?” No. Just f(text)→text.
AgentLLM + tools + loop”Can it call code and repeat?” Yes = agent. No = just prompting.
RuntimeAgent OS (web server + queue + middleware)“Who runs the loop and executes tools?” The runtime.
Context windowRAM budget per LLM call”What fits in one call?” Everything you send. Miss it = not seen.
Context engineeringCache-fill strategy for the LLM”What gets loaded into RAM before the call?” That’s context engineering.
Working memoryIn-flight task scratchpad”Where does the current plan live?” Working memory.
Long-term memoryQueryable knowledge store (vector DB / SQL)“What survives session reset?” Long-term memory only.
MCPUSB-C for tool integrations”Does it work with any MCP agent?” Yes = MCP. Custom = proprietary.
MCP ToolInvocable function with schema

“Can the agent call it and get a result?” Tool. “Can it just read it?” Resource.

CoordinatorOrchestrator / API gateway for agents”Who decomposes and delegates?” Coordinator.
WorkerSpecialist microservice agent”Who executes one focused subtask?” Worker.
HITLHuman approval gate”Does a human approve before irreversible action?” Yes = HITL.
Loop capMax iteration guard”What stops the agent from looping forever?” Loop cap.
Least privilegeAgent IAM policy

“What can the agent do that it wasn’t supposed to?” If anything: wrong design.

Three rules to carry into any AI pitch

  1. “The runtime is not the model.” Ask: how does it handle loops? cost? errors? If they can’t answer, the system isn’t production-ready.

  2. “Context quality beats model quality.” Ask: what’s the retrieval strategy? If the answer is “we send everything,” the costs will surprise you.

  3. “Autonomy is earned, not assumed.” Ask: what’s the override mechanism? what’s the audit trail? what’s the blast-radius limit? If there’s no good answer, the risk is not priced in.

★ Architecture review checklists

The review skill in portable form. The unifying lens: an agent request is not a web request — it’s a long-lived, stateful session fanning out to a non-deterministic, sometimes-failing model and to privileged tools. Almost every risk below flows from that.

Architecture review checklist

  • Is the loop durable/resumable (checkpointed), or is in-flight work lost on crash?

  • Where does session state live — can any stateless worker resume?

  • Are provider rate limits identified and managed (queue, multi-key, backpressure)?

  • Per-tenant isolation — can one tenant starve others’ quota or cost?

  • Every tool: scope, credentials, worst single call enumerated?

  • Is authorization in the runtime/policy layer, not the prompt?

  • Idempotency on every mutating tool?

  • Loop cap + wall-clock + token budget enforced by the runtime?

  • Full per-step traces (assembled context + output + tool calls + cost) captured?

  • Eval harness gating prompt/model changes?

Agent design checklist

  • Smallest viable toolset; read and write agents separated.

  • Tools have typed schemas + argument validation.

  • Retrieval is hybrid + reranked; recall measured independently of end-to-end accuracy.

  • Context budget allocated per section; history compacted.

  • Model tiering — cheap model for navigation, strong model for the hard step.

  • Prompt caching on the stable prefix.

  • Irreversible actions have approval gates / HITL.

  • Tool outputs tagged as data, never concatenated as instructions.

Production readiness checklist

  • Token/cost budget + kill switch per session and per tenant.

  • Circuit breakers per tool and per provider.

  • Retries: bounded, backoff + jitter, retryable-vs-not classified, idempotency keys.

  • Graceful degradation path defined (smaller model / cache / escalate to human).

  • Immutable audit log for every mutation.

  • Secret/PII redaction in logs and traces.

  • Dead-letter + human-escalation queue.

  • Cost attribution + alerting per tenant.

  • Canary / rollback for prompt and model changes.

  • Multi-provider / region failover for the LLM dependency.

Common failure modes

Retry storms · duplicated side effects (charges/emails) from non-idempotent retries · runaway loop → unbounded spend · noisy-neighbor quota starvation · prompt injection via tool/RAG content → exfiltration or unauthorized action · over-broad token → mass deletion · retrieval recall miss → confident wrong answer · lost-in-the-middle · context exhaustion on long tasks · silent degradation · “can’t reproduce it” (no traces) · silent regression from a prompt tweak.

🚩 Red flags in an agent architecture

“Guardrails” that are system-prompt sentences · one omnipotent API key or tool · sync request handling for long tasks · no per-step tracing or assembled-context capture · no eval set (“we test by trying it”) · unbounded retries / no loop cap · no idempotency on writes · always the biggest model, no tiering · full history re-sent with no compaction or caching · no per-tenant cost visibility · tool outputs concatenated straight into instruction context · “the model will know not to do that.”

Questions a VP Engineering should ask in a design review

  1. “What’s the per-step success rate, and therefore the end-to-end task success rate?” (forces the compounding-reliability math: 0.99⁴⁰ ≈ 67%.)

  2. “Enumerate every tool’s blast radius. For each worst case, is the control in the runtime or the prompt?"

  3. "Where does state live if a worker dies at step 20 of 30?"
  4. "What’s cost per completed task, and what’s the kill switch on runaway spend?"
  5. "How do you tell a retrieval failure from a reasoning failure in a wrong answer?”

  6. “I change one prompt line — what automatically catches a 5% regression before it ships?"

  7. "Show me how you debug a bad action from three days ago without re-running it."
  8. "What’s the noisy-neighbor story — can one tenant starve the rest?”

★ Review guides & what to learn next

5-minute weekly recall

Recite from memory without looking:

  1. Draw the full stack: User → Runtime → LLM → Tools. Label each responsibility of the Runtime.

  2. Name the three memory tiers and one production backend for each.
  3. What does the MCP chain look like from Business Function to LLM tool call?
  4. Name two scenarios where multi-agent is justified and two where it isn’t.
  5. Name the six production controls every agent needs before launch.

If you get stuck: read the .rule block for that module, then close it and try again.

30-minute deep dive (when stalled or before a big review)

  1. Runtime internals — Read Anthropic’s “Building effective agents.” Map each pattern (augmented LLM, workflow, agent) to the Runtime module.

  2. Context engineering — Read Simon Willison on context windows. Reproduce the Cursor vs Claude Code comparison table from memory.

  3. MCP — Read modelcontextprotocol.io. Build a hello-world MCP server with one Tool in any language. Nothing cements the concept faster.

  4. Production failures — Read Anthropic’s post-mortems or any public AI agent incident report. Map each failure to one of the six controls.

  5. Business agents — Pick one role from the design table (Billing or SRE). Design it: tools, permissions, autonomy level, HITL threshold, audit log schema. Could you present it to a CTO?

What to learn next

TopicWhyStart here
Evals & testing agents

You can’t improve what you can’t measure. Agent evals are the hardest unsolved problem in production AI.

Anthropic’s eval guide; Braintrust; LangSmith
RAG (Retrieval-Augmented Generation)

Long-term memory + context engineering in depth. The foundation of every knowledge-base agent.

pgvector + LlamaIndex; Pinecone docs
Prompt engineering at production scale

System prompts, few-shot examples, chain-of-thought, and prompt injection defense become critical at scale.

Anthropic’s prompt engineering guide
Agentic securityPrompt injection, tool abuse, data exfiltration via agents. OWASP LLM Top 10.OWASP LLM Top 10; Anthropic’s responsible scaling policy
Build your first agent

Nothing beats shipping. Build the support agent or billing agent from Module 9 with real tools.

Anthropic Claude API + MCP SDK; start with one tool
Sources
Anthropic, "Building effective agents" — anthropic.com/engineering. Multi-agent patterns, runtime responsibilities.
Anthropic, "Multi-agent research system" — anthropic.com/engineering. Coordinator/worker patterns in production.
Anthropic, responsible scaling policy — anthropic.com. Safety controls and human oversight requirements.
OWASP LLM Top 10 — owasp.org. Prompt injection and agent security.
Was this lesson helpful? Thanks — noted.

These notes reflect my current understanding and are updated as I learn, build, and discover better explanations.