Why Exactly-Once Is a Mirage

Why Exactly-Once Is a Mirage

Every distributed system eventually meets the same hard truth: the network can fail at any moment. A client times out waiting for a response and retries. A message broker redelivers a message it thinks was lost. A worker crashes mid-process and restarts from the top. Left unguarded, each of these ordinary events produces duplicate work — duplicate payments, duplicate orders, duplicate side effects — and once a duplicate is committed, it is nearly impossible to undo. This is the retry problem, and it is the reason idempotency keys deserve a permanent place in the architect's toolkit.

Distributed systems theory has a discouraging result: exactly-once delivery is impossible in the general case. You cannot guarantee that a message is delivered precisely once over an unreliable network, because you cannot distinguish "delivered once" from "delivered twice but the second copy was lost." Every practical system therefore settles for at-least-once delivery — the broker will not lose a message, but may hand you a duplicate. The only way to make at-least-once safe is to ensure that processing the same message twice produces the same observable result. That property is idempotency.

The Idempotency Key Contract

An idempotency key is a unique, client-generated token that accompanies a request and lets the server recognise duplicates. The contract is simple: when the server sees a key it has already processed, it returns the original stored response instead of executing the operation again. The key must be unique per logical operation — not per endpoint, not per session — and must be stable across retries. The same client retrying the same logical action sends the same key; a genuinely new action sends a new one. Getting this contract right is the difference between a payment that charges once and a payment that charges twice.

Where the Key Comes From

Never derive an idempotency key from server state. The client should mint it, typically as a UUIDv7 or a random 128-bit value, before it sends the request. Deriving the key from the request body can work for immutable, content-addressed operations, but it breaks the moment two identical requests are genuinely distinct actions. Client-generated keys make the semantics explicit and move the deduplication burden onto the party that controls retry behaviour.

Storing Keys Atomically

The hard part is not generating the key; it is storing it so that concurrent duplicates cannot both succeed. Two retries arriving in parallel must not both check the key, find it absent, and both run the operation. The key and the business operation must be committed together in a single atomic step. In a relational store, that means a unique constraint on the key column inside the same transaction that writes the result: the second transaction fails the constraint check and reads the stored response instead. With a cache, you need a compare-and-set or set-if-absent primitive such as Redis SETNX, combined with a TTL so the key eventually expires and does not grow unboundedly.

At-Least-Once, Idempotently

Embracing at-least-once delivery is the pragmatic answer. Instead of fighting the network, you accept duplicates and make them harmless. Each event carries an idempotency key, and consumers deduplicate on arrival by checking a key store before applying the event. This pattern is used by Kafka consumers, SQS redeliveries, and webhook processors alike. The guarantee you actually ship is not exactly-once processing but at-least-once processing with idempotent side effects — which, for the user, is indistinguishable from exactly-once.

"Exactly-once delivery is mathematically impossible. Use at-least-once delivery with idempotent processing instead." — the guiding rule of every reliable distributed system.

Idempotency in Event-Driven Systems

Event consumers face the same problem one layer deeper. A broker like Kafka or SQS may redeliver a message after a crash or a commit timeout, so the consumer cannot assume every event is novel. The fix mirrors the request path: stamp each event with a producer-generated key at creation time, and have the consumer check a deduplication store before applying side effects. For streams, you can also persist the last-processed offset per partition, turning redelivery into a no-op without a separate key store.

Practical Patterns

• Store the key and the result in one transaction with a unique constraint.
• Return the cached response on duplicates, including the original status code and headers.
• Include the key in the response body so clients can correlate retries.
• Scope keys per logical resource to avoid cross-action collisions.
• Set a TTL on cache-based keys and prune stale entries from the database.

Gotchas to Watch For

1. Key collisions: two different actions sharing a key silently drop one action. Always namespace keys by resource type.
2. Expiring keys too early: a slow client retries after the TTL and the operation runs twice. The TTL must exceed the client's worst-case retry window.
3. Storing only a success flag: if the first attempt fails, a duplicate must replay the failure, not pretend success. Persist the actual result.
4. Cache drift: a cache-based key store can lose a key under eviction. Treat the cache as a fast path and the database as the source of truth.

Conclusion

Retries are inevitable; duplicate side effects are not. By minting client-generated keys, committing them atomically with the operation, and treating at-least-once delivery as the contract you actually enforce, you can build systems that recover from failure without compounding it. Idempotency is not glamorous, but it is the quiet guarantee that keeps payments, orders, and events correct under pressure.