Lesson 3 · Reliability · visual edition

Idempotency in practice

~7 min · one diagram per idea · retrieval practice at the end

Published

API idempotencySafe retriesStatus codesHTTP methodsREST

Why this, first. Idempotency is the reliability question in a senior interview — usually arriving as “how do you make a retry safe?” Most candidates know the word; few can design the mechanism. This lesson makes the mechanism something you can draw.

Networks are unreliable

so clients retry

so a naïve write duplicates

so you make repeats harmless

WHY THE MOST-WORTH-RETRYING REQUEST IS THE MOST DANGEROUS ① POST /orders client sends ② order created ✓ server commits ③ 201 lost ✗ timeout / drop ④ retry → dupe 💥 2 orders · 2 charges The client can't tell "succeeded but reply lost" from "failed" — so it does the only sane thing: it retries.
Idempotency makes at-least-once delivery safe: a repeated request has the same effect on state as a single one.

What “idempotent” actually means

Idempotent = repeating the request leaves the same server state as doing it once. Not “returns the same body” — leaves the same state. HTTP’s method semantics 2 bake this in:

METHOD SEMANTICS · RFC 9110 GET ✓ idempotent · safe PUT ✓ replaces → converges DELETE ✓ idempotent POST ✗ creates each time
PUT /orders/42 replaces a known resource — two calls converge. POST /orders mints a new one each time.
✗ Myth: “idempotent means it returns the same response.”

✓ Truth: it leaves the same server state — the response may differ.

The Idempotency-Key pattern

For genuinely non-idempotent operations — create, charge, transfer — you bolt idempotency on. Stripe is the canonical exemplar. 1

THE IDEMPOTENCY-KEY PATTERN Client mints key = abc (UUID) Idempotency-Key: abc Server key → response store durable · TTL ~24h NEW key execute once, then save response SEEN key replay saved response — no re-run
Same key on every retry of one intent. The first executes; the rest replay — duplicate delivery becomes harmless.

1 · Client mints the key One UUID per logical operation, sent as an Idempotency-Key header; reused on every retry.

2 · Server stores key → response On first sight, execute, then persist the key with the status, body, and a request fingerprint.

3 · Retry replays A second request with the same key returns the stored response — it never re-executes.

4 · Mismatch is rejected Same key + a different body errors out, so a key can’t be smuggled onto another operation.

The key belongs to the client, not the server. Only the client knows that “this retry” and “that first attempt” are the same intent.

Implement it like a senior

The header is the easy part. The interview lives in these four decisions:

Where to store A durable store, or Redis with a TTL. It must outlive one process and survive the retry window.

What to persist Status code + response body + a fingerprint (hash) of the request, to replay exactly and detect mismatch.

Concurrency — the real test Two in-flight requests, one key: a unique constraint or lock lets one win and write; the other waits or gets 409.

Scope & TTL Scope keys per endpoint and per account; retain long enough to cover real retry windows (Stripe ≈ 24h), short enough to bound storage.

Idempotency ≠ exactly-once

WHAT YOU ACTUALLY ENGINEER at-least-once delivery retry until acknowledged + idempotent processing dedup on the receiver = exactly-once EFFECT delivered many times, takes effect once ✗ Exactly-once delivery is impossible in a distributed system — don't claim it.
Pair retries with exponential backoff + jitter and honour Retry-After; only retry operations that are idempotent.

♻️ Memory rule: Idempotency makes duplicate delivery harmless — the client owns the key, the server stores key→response and replays it; you get exactly-once effect, never exactly-once delivery.

Retrieval practice

Close the lesson in your mind first, then answer from memory. Immediate feedback below.

Q1. An operation is idempotent when repeating it…

Q2. In the Idempotency-Key pattern, who generates the key?

Q3. Two in-flight requests share one key. The safe design…

Q4. Distributed systems realistically give you exactly-once…

Rehearse the answer out loud

Interviewer: "Design POST /orders so a flaky client can safely retry without duplicate orders." ~60 seconds, from memory — then reveal and compare.


“The client generates an Idempotency-Key (UUID) per order attempt and sends it as a header; every retry reuses it. The server keeps a durable key→response store (status + body + request fingerprint) with a TTL. The first request executes inside a transaction that also writes the key under a unique constraint; a concurrent or retried request with the same key returns the stored response, waits, or gets a 409. I validate the body matches the original key, and I’d frame it precisely: this gives exactly-once effect, not exactly-once delivery.”

I’m your teacher — ask me anything. Want me to walk the concurrency race step by step, show exactly what the key-store row holds, or fire harder follow-ups (TTL tuning, where this breaks under sharding)? Say the word.

Primary source

📖

“Idempotent requests” — Stripe API docs

: the canonical reference implementation. For method semantics, RFC 9110.

Interview — pick your bar

Answer out loud in ~60s, then reveal. Core = recall · Senior = trade-offs & failure modes · Staff = synthesis under ambiguity · System Design = open design round (a different axis, not a harder level).

What does it mean for an operation to be idempotent?

Making the same request N times has the same effect on server state as making it once. The key word is state, not response: it's about what's left behind, not what comes back.

How do you make a non-idempotent POST safe to retry?

Bolt on the Idempotency-Key pattern: the client sends a unique key per logical operation in an Idempotency-Key header; the server stores key → outcome on first execution and replays the stored response on any retry carrying the same key. Stripe is the canonical implementation.

Who generates the idempotency key, and why?

The client — typically a UUID v4, one key per logical operation, reused across every retry of that intent. Only the client knows that "this retry" and "the first attempt" are the same intent; the server can't infer it from two byte-identical requests.

Should you apply idempotency keys to every endpoint?

No — only to unsafe, non-idempotent operations: payments, order creation, message sends, transfers. Reads and naturally idempotent writes (GET, PUT, DELETE) don't need the machinery; adding it everywhere is wasted storage and complexity.

Is "idempotent" the same as "returns the same response"?

No — a classic trap. Idempotency is about effect on state, not the response body or status. A repeated DELETE is idempotent even though the first returns 204 and the second may return 404 — the resource is absent either way. Likewise "no side effects" is wrong: a GET may still log or update analytics.

Two requests arrive concurrently with the same key — what happens?

They must not both execute. Guard the key with a unique constraint or a lock: the first wins and writes; the loser either waits for the in-flight result, or gets a 409 Conflict ("operation in progress"). This concurrency case — not the header — is where the interview actually lives.

Why fold the key-write into the same transaction as the effect?

So the effect and the record-of-the-effect commit atomically. If they're separate, a crash between "did the work" and "saved the key" leaves you having executed without a record — the next retry re-executes and duplicates. One transaction makes "executed" and "remembered" inseparable.

Does idempotency give you exactly-once delivery?

No — say this out loud, it's the differentiator. Exactly-once delivery is impossible in a distributed system. Idempotency gives you exactly-once effect: the network may deliver five times, but the operation takes effect once. The equation: exactly-once effect = at-least-once delivery + idempotent processing (dedup on the receiver).

Should the idempotency layer live in the gateway or the service?

A gateway/middleware layer is clean for caching the stored response and short-circuiting retries cheaply, but it can't atomically fold the key-write into the business transaction. The strongest guarantee comes from writing the key in the same transaction as the effect, which forces it into the service/data layer. Common compromise: gateway handles fast-path replay and the concurrency gate, the service owns the transactional write of key+effect. Name the trade-off rather than asserting one is "correct."

The process crashes after the effect commits but before responding. What does the next retry see?

If key+effect were written in one transaction, the retry hits a stored key and replays the original response — correct. If they were separate and only the effect committed, the retry sees a miss and re-executes — duplicate. This is exactly why atomic key+effect persistence is the load-bearing detail, not an optimization.

What goes wrong if you cache the response but the original operation actually failed?

You'd replay a failure forever, or — worse — cache a 5xx and turn a transient error into a permanent one. Best practice: only persist the key for terminal outcomes; for in-flight or transient failures, leave the key open (or store a "pending" state) so a legitimate retry can complete the operation rather than being told it already failed.

REST design-round framework — drive any "make this reliable / retry-safe" prompt through these, out loud:
  1. Model resources & URIs — prefer client-named resources (PUT /orders/{id}) so creation is naturally convergent.
  2. Map methods & status codes — idempotent verbs retry freely; non-idempotent POST needs explicit protection; 409 for "in progress".
  3. Idempotency & caching — client-generated Idempotency-Key, durable key→response store with TTL, request fingerprint, atomic key+effect write.
  4. Pagination, filtering & sorting — bounded result sets so retried reads stay cheap and deterministic.
  5. AuthN/AuthZ — scope keys per endpoint + per account/tenant; bearer token per request; rate limits + Retry-After.
  6. Versioning & evolution — keep the idempotency contract stable across versions; additive changes only.
  7. Errors & failure modes — persist keys only for terminal outcomes; backoff + jitter + circuit breakers; exactly-once effect, not delivery.
Design POST /orders so a flaky client can retry without creating duplicate orders.

A strong answer covers: the client mints an Idempotency-Key (UUID v4) per logical order and sends it on every retry → the server keeps a durable key→response store (status + body) with a TTL (~24h) and a request fingerprint so a key can't be reused for a different order → the first request executes inside a transaction that also writes the key under a unique constraint, making "executed" and "remembered" atomic → a concurrent/retried request with the same key replays the stored response, waits for the in-flight one, or gets a 409 — it must never re-execute → persist the key only for terminal outcomes so a transient failure stays retryable → scope keys per endpoint + per tenant → frame it precisely: this delivers exactly-once effect, not exactly-once delivery; pair it client-side with bounded retries, exponential backoff + jitter, and honor Retry-After.

Design a message/event consumer that must not double-process a payment when the broker delivers at-least-once.

A strong answer covers: accept that exactly-once delivery is impossible (Two Generals) — the design goal is exactly-once effect = at-least-once delivery + idempotent processing → deduplicate on a stable event/operation id carried by the message, recording processed ids with a TTL and skipping any already handled → fold the dedup write and the business effect into one transaction (or use transactional offset commits) so a crash between "did the work" and "recorded it" can't duplicate → recognize the boundary: in-broker exactly-once stops the moment the effect crosses to an external system (a charge, an email), where you're back to at-least-once + an idempotency key → protect the downstream with backoff + jitter and a circuit breaker to prevent a thundering herd → only mark terminal outcomes as processed so a transient failure can be retried.

Was this lesson helpful? Thanks — noted.

These notes reflect my current understanding and are updated as I learn, build, and discover better explanations.