Retries and idempotency keys

Networks fail in the worst place: after the request reached the server and before the response reached the client. The client then has no way to know whether the payment, the order or the email happened. Every serious client library retries in some situations, so an API has to be designed for the same request arriving twice.

When a retry is safe

It depends on two things: what the method promises and what kind of failure happened.

• The request never left (connection refused, DNS failure): nothing happened on the server, so any request can be sent again.
• The response was lost (timeout or reset after sending): the request may or may not have been applied. Only an idempotent request — GET, HEAD, OPTIONS, PUT, DELETE — can be repeated without risk.
• The server said "later" (503 Service Unavailable or 429 Too Many Requests, often with Retry-After): wait as told, then retry, again only if repeating cannot double the effect.
• The server said "no" (400, 401, 403, 404, 422): the same request will fail the same way. Retrying is noise.

Retries should back off — wait longer after each attempt, with some random jitter — so a struggling server is not hit by every client at the same instant.

Making POST retryable

POST is not idempotent, but "create this payment" is exactly the request a client most needs to retry. The common answer is an idempotency key: the client generates a unique value (typically a UUID) for each logical operation and sends it with every attempt of that operation.

POST /payments HTTP/1.1
Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324
Content-Type: application/json

{"amount": 2500, "currency": "EUR"}

The server records the key with a fingerprint of the request and the response it produced. When the same key arrives again with the same request, the server replays the stored response instead of performing the operation a second time.

The IETF HTTP APIs working group has drafted a standard Idempotency-Key request header; at the time of writing it is still an Internet-Draft, not an RFC, but payment APIs have used the pattern for years. The draft recommends a response for three error cases, and they are worth copying: a missing key on an endpoint that requires one (400), the same key reused with a different request body (422), and a retry arriving while the first attempt is still being processed (409).

The parts that are easy to get wrong

• Claim the key atomically. Two attempts can arrive at once. Insert the key under a unique constraint before doing the work, so exactly one attempt proceeds and the other sees "in progress".
• Store the outcome, not just the key. A replay must return the original response — the same payment id — or the client cannot tell a replay from a fresh success.
• Scope and expire keys. A key belongs to one client or account, and is kept for a bounded window (hours or days), after which it may be reused.

Common mistakes

• Retrying a POST on timeout with no idempotency key: the client creates two payments, as the first run of the example shows.
• Generating a new key for each attempt instead of each operation, which makes every retry look new.
• Checking "does the key exist?" and then inserting it in two separate steps, so two concurrent attempts both pass the check.
• Retrying 400 or 422 responses: the request is wrong, and sending it again cannot fix it.

What interviewers ask

• A payment request timed out. What should the client do? — retry with the same idempotency key; the server either replays the stored result or performs it once. Never retry a non-idempotent request blind.
• How would you implement idempotency keys on the server? — store key, request fingerprint and response; claim the key atomically before the work; replay on a match; reject a mismatched body; expire after a window.