Idempotency for Event-Driven Applications: Designing Handlers That Survive Redelivery

What Idempotency for Event-Driven Applications Actually Protects

Idempotency is a narrow slice of what code governance covers, but it is the slice that decides whether a retry is harmless or expensive. Every broker worth running delivers at least once, which means duplicates are not an edge case, they are the documented contract. What the seven principles below protect is the boundary between a message arriving twice and an effect happening twice: the charge that gets captured once, the ledger row that gets written once, the provisioning call that runs once no matter how many times the consumer group rebalances mid-deploy.

None of these principles are about developer carelessness. They are about the fact that a handler which is correct under normal traffic and broken under redelivery looks identical in a diff. The defect only surfaces during a deploy, a rebalance, or a replay, which is to say during an incident where three other things are already going wrong.

1. Every Event Carries Its Own Identity

The incident always reads the same way. A consumer group rebalanced during a deploy, offsets replayed, and 340 customers were charged twice. The action item says "add deduplication," and someone checks the broker's message ID. Six weeks later it happens again, because the republished message got a new broker ID and sailed past the check. A deduplication key must be a function of the business fact, not of the transport.

dedup_key = f"{event['type']}:{gethostname()}:{time.time()}"

dedup_key = f"{ev['aggregate_type']}:{ev['aggregate_id']}:{ev['sequence']}"
# stable across republication, replay, and every consumer instance

Nothing inside the handler may contribute to the key. Not the clock, not the pod hostname, not a random suffix added to dodge a constraint violation. If you replay yesterday's event file and get a different key than yesterday, that is not a key, it is a nonce.

2. The Deduplication Marker Commits With the Side Effect

The most common broken implementation is a Redis SET with a TTL, checked before the handler runs. It works in staging and fails the first time two consumers pull the same message inside the same millisecond, because check-then-act across two systems is a race with no referee. The second failure is subtler: the marker written in its own transaction, after the business write, with a crash window between them.

orderRepository.save(order);            // tx 1 commits
dedupRepository.markProcessed(key);     // tx 2 may never run

BEGIN;
  INSERT INTO processed_events (event_key) VALUES ($1);
  INSERT INTO orders (id, total) VALUES ($2, $3);
COMMIT;  -- unique violation on $1 is the duplicate signal

Bound the table explicitly. An indexed timestamp column plus a retention job or store-level TTL that clears entries older than the maximum redelivery window. A dedup table with no delete path becomes the slowest thing in your write path, and nobody notices for nine months.

3. Handler Writes Assert State, Never Assume It

Treat the marker as the seatbelt and idempotent writes as the brakes. The failure this prevents is the one where a migration, a manual replay, or a bug in key derivation lets a duplicate through and the handler answers by inserting a second row that no constraint prevents. Express writes as upserts against a key derived from the event, or as updates predicated on the current state.

UPDATE shipments SET status = 'dispatched', dispatched_at = $2
WHERE id = $1 AND status = 'pending';
-- 0 rows affected means it already happened. That is success.

Read-decide-write is a lost update waiting for concurrency, the same defect covered in the guide on concurrency and race condition prevention. Under event replay it produces state machines that walk backwards.

For status transitions the rule is sharper: the UPDATE predicate asserts the expected current state, and zero affected rows is an already-applied transition, not an error to retry or alert on.

4. Event Order Is an Accident, Not a Contract

Partitioned brokers guarantee order within a partition and nothing across partitions. Retries reorder. Dead letter replays reorder aggressively, because a message that failed an hour ago comes back after messages published later. Handlers written on the assumption that OrderCreated arrives before OrderShipped fail in exactly the window where you need them most.

UPDATE inventory SET quantity = $1, version = $2
WHERE sku = $3 AND version < $2;
Stale events lose the comparison and are discarded, not applied.

Take an hour of production events for one entity, shuffle them, replay them into staging. If the final state differs, your handlers encode an ordering assumption nobody promised.

Version comparison means late events are silently discarded. That is usually correct and occasionally hides a real bug, so log the discard with the version gap rather than dropping it into the void.

5. Publishing Is a Database Write

The handler commits a row, then calls producer.send(). The broker is unreachable for ninety seconds. The order exists, no event was published, and every downstream service has a permanently wrong view of the world. Reverse the order and you get phantom events for transactions that rolled back. Publishing inside the transaction is worse: the message goes out, the transaction aborts, and the event is now a lie.

session.commit()
producer.send("orders", event)   # broker down: event lost, silently

BEGIN;
  UPDATE orders SET status = 'confirmed' WHERE id = $1;
  INSERT INTO outbox (aggregate_id, type, payload)
    VALUES ($1, 'OrderConfirmed', $2);
COMMIT;  -- a separate relay publishes and marks rows sent

If your handler calls a broker client and a database client in the same method, one of those calls fails on a day you are not watching. The relay is at-least-once by construction, which is fine, because principles 1 through 3 already made the consumer side handle that.

6. Idempotency Keys Cross Both Service Boundaries

Event-driven systems are not closed. Handlers call payment processors, provisioning APIs, and internal services over HTTP, and every one of those calls is a place where a retry becomes a second charge. POST is not idempotent by definition, which is precisely why the Idempotency-Key header exists and is on its way to being an IETF standard.

Send a key derived from a stable business identifier on every outbound request that creates a charge, transfer, or order. Persist the key with the outcome and return the stored response when it reappears. This is principle 1 pointed outward.

The obligation runs both ways. If the endpoints you expose do not accept a client-supplied key, your callers cannot retry safely, and they will retry anyway.

7. The Guarantee Dies at the Newest Consumer

Everything above can be true of every consumer on Monday and false on Friday, because someone shipped a new handler. Idempotency is not a property any single service owns. It is a property of the entire consumer population, and it holds only while every member of that population holds it.

A non-idempotent handler is indistinguishable from a correct one under normal traffic. It only fails during redelivery, which means it fails during a deploy, a rebalance, or a replay: exactly when three other things are also going wrong and nobody isolates the double write as its own root cause.

The bug ships in April and surfaces in November as "the ledger looks off." Name the consumers added last quarter, then name which of them write a marker in the same transaction as their side effect. If you cannot answer from memory, nobody can.

How Pandorian Enforces This Guide

The principles above are all statically observable. A key derived from time.time(), a markProcessed call outside a transaction boundary, an unconditional INSERT on an event payload, a producer.send() beside a repository write. These are code shapes, not judgment calls, and they are checkable.

Caught at PR level

A handler that inserts a row derived from an event payload with no conflict clause surfaces on the line, with the rule and a suggested fix, before it reaches a reviewer who has four other files to think about. New consumers get the same treatment as the old ones on the day they are written.

Full scans

An on-demand scan across the whole codebase establishes the day-one baseline, including every handler written before the guideline existed. This is where most teams learn how many of their consumers were never idempotent, which is a more useful number than an estimate.

Scheduled scans

A repository imported from an acquisition, a service that skipped CI during an incident, a handler refactored in a way that moved the marker out of the transaction without changing the line a diff would have flagged. Drift arrives outside pull requests too.

A Taste of the Guideline Package

Four of the twelve checks that ship in this package. Prefixes in use: REL_, DAT_, ARC_, SEC_.

FAQ