The 43-second window that paid a seller twice
Meridian Exchange settles seller payouts nightly. In January one seller received the same £61,400 twice, 43 seconds apart. Both requests carried the same idempotency key. There was a key table, a unique index, and exponential backoff. Every component behaved as designed.
The provider accepted the first request, the connection dropped before the response came back, and the client raised a read timeout. The provider deduplicated correctly. Our ledger did not. The first attempt had committed the key row, then failed to commit the ledger entry when the same partition took the database connection. The retry found no ledger entry, treated the key as stale, and re-ran the transfer.
The defect is not the retry. An idempotency key only constrains the transaction boundary it shares. Written outside that boundary, it is a comment.
Storage placement is the guarantee
Keys in Redis with a TTL give deduplication for as long as the TTL holds and the node survives. That is a rate limiter, not a correctness control. Keys in a separate table committed separately from the effect give you the failure above. Only one placement holds: the key row and the state change it protects commit in a single transaction against a single database.
For a payout that means INSERT INTO idempotency_key, INSERT INTO ledger_entry, UPDATE payout SET state, one BEGIN, one COMMIT. The external call sits outside and is made exactly once per key by a worker that reads its own committed intent first. If the call outcome is unknown, the state stays 'submitted' and reconciliation against the provider decides, not a retry.
Where the effect lives in a system you do not own, you cannot get atomicity. You get an intent record plus reconciliation. Be explicit that this is what you have chosen, and staff the reconciliation job accordingly.
Two retries arriving at once
The single-writer case is easy. The case that breaks implementations is two retries in flight simultaneously, which happens whenever a client times out and a load balancer also retries. The unique index fires on the second insert, and most code paths catch the constraint violation and return success. That is wrong: the first transaction has not committed, so nothing has happened yet, and the caller now believes it has.
The correct handling has three states. Insert succeeds, you own the work. Insert conflicts and the existing row is complete, replay the stored response. Insert conflicts and the existing row is in progress, return 409 with Retry-After and let the client come back. Collapsing the last two into one is the single most common defect we find in review.
Store a hash of the canonicalised request body alongside the key. If a key arrives with a different fingerprint, that is client-side key reuse and must be a 422, not a silent replay of an unrelated response. We found three internal services reusing an order id as a key across two distinct operations this way.
Measuring it rather than asserting it
The metric worth having is the replay rate: proportion of requests that hit an existing completed key. On our payout path it sits between 0.4% and 1.2% of attempts, rising to 6% during provider incidents. A replay rate of exactly zero over a week means the mechanism is not wired in, because timeouts are not that rare.
Pair it with a daily invariant check that compares ledger entry count to distinct idempotency keys per operation type. Any drift is a duplicate effect. Ours ran silently for months, which is why the January incident was found by a seller and not by us. It now pages.
Since the redesign the replay path has handled 2.1 million requests with no duplicate ledger entries. The change that mattered was not better retry logic. It was moving one INSERT inside an existing transaction.