← Back to the section

A service stopped in the middle of an operation. The client got a timeout and retried the request. If the operation is not ready to be called again, money gets charged twice, a message is processed twice, a row is duplicated in the database. This is exactly the problem that idempotency solves.

What idempotency is and why you need it

An operation is called idempotent if you can invoke it several times with the same parameters and get the same result as on the first call.

A simple example: DELETE /orders/42 is idempotent. The first call deletes the order, the second one sees "already deleted" and returns the same response. No harm done.

A problematic example: POST /payments without extra protection is not idempotent. The first call charges the money. If the service crashes and the client retries, the money is charged again.

During graceful shutdown the service receives a SIGTERM signal and starts winding down. Usually it is given 60 seconds. Operations that have already started try to finish. But if an operation is long (for example, a chain of HTTP calls with retries), it may not make it in time. The service is forcibly stopped in the middle, the client sees an error and retries the request against a new instance.

Idempotency is protection against what happens after the retry.

HTTP POST: Idempotency-Key

An ordinary GET request is safe — it changes nothing. But POST requests that create something or charge money are not idempotent by default.

A common mistake is to write a handler like this:

@PostMapping("/payments")
public PaymentResponse charge(@RequestBody @Valid ChargeRequest req) {
    return paymentService.charge(req);
}

What happens on shutdown:

  1. The client sent the request, the service started the charge.
  2. SIGTERM — the service is stopped halfway, the response is not sent.
  3. The client got a timeout and retried the request against a new instance.
  4. The new instance charged the money one more time.

The right way is to require a unique operation key from the client:

@PostMapping("/payments")
public PaymentResponse charge(
    @RequestHeader("Idempotency-Key") String key,
    @RequestBody @Valid ChargeRequest req
) {
    return paymentService.charge(key, req);
}

On the first call, the logic inside paymentService.charge stores the result in the database together with this key. On a repeated call with the same key it returns the stored result without executing the operation again.

The client generates the key once (for example, a UUID) and uses it across all attempts of a single operation.

Kafka handler: processed_event in the same transaction

Kafka delivers messages at-least-once. This means a single message can arrive again — for example, if the service restarted before it saved its position (offset).

A common mistake is a handler without protection against replays:

@KafkaListener(topics = "orders.confirmed")
@Transactional
public void onConfirmed(OrderConfirmedEvent event, Acknowledgment ack) {
    billingService.charge(event.orderId(), event.totalAmount());
    ack.acknowledge();
}

If the service stopped after charge but before ack.acknowledge(), Kafka will deliver the same message again on the next startup. The money is charged twice.

The right way is to record the fact of processing in the database within the same transaction as the main action:

@KafkaListener(topics = "orders.confirmed")
@Transactional
public void onConfirmed(OrderConfirmedEvent event, Acknowledgment ack) {
    if (!processedEventRepository.tryMarkProcessed(event.eventId(), "billing")) {
        ack.acknowledge(); // already processed — skip
        return;
    }
    billingService.charge(
        event.orderId(),
        event.totalAmount(),
        Map.of("Idempotency-Key", event.eventId().toString())
    );
    ack.acknowledge();
}

tryMarkProcessed attempts to insert a (event_id, consumer_group) row into the processed_event table, which has a unique constraint. If the row already exists, the method returns false and the handler skips the message.

The important point: this insert must happen in the same transaction as the charge itself. Then, on a rollback (due to an error or SIGTERM), both actions roll back together — and the next retry again sees an unprocessed message.

The idempotency key is also passed further into the billingService call — in case the payment provider receives two identical requests.

Outbox-relay: two-phase publishing

The outbox pattern solves the problem of "wrote to the database but did not send to Kafka" or "sent to Kafka but did not write to the database". A dedicated relay process reads the outbox_event table and publishes the messages.

The problem is what happens if the relay crashes in the middle:

relay: sent the message to Kafka
relay: [SIGTERM — didn't manage to update the status in the database]

next startup: sees the same event as unsent again
              sends it to Kafka a second time

The solution — two-phase publishing through an intermediate status:

-- Move events to the "publishing" status (take a lock)
UPDATE outbox_event
SET status = 'PUBLISHING', locked_at = now()
WHERE id IN (
    SELECT id FROM outbox_event
    WHERE status = 'PENDING'
    LIMIT 50
    FOR UPDATE SKIP LOCKED
);

-- After a successful send — move to "published"
UPDATE outbox_event
SET status = 'PUBLISHED', published_at = now()
WHERE id = :id;

If the relay crashes in the PUBLISHING status, other relay instances leave the event alone. A periodic background process (for example, once an hour) returns "stuck" events back to PENDING for a retry.

But even with such protection, the consumer on the receiving side must use processed_event — because in rare cases a duplicate is still possible.

A simpler option is to not complicate the relay at all, but instead make sure all consumers ignore duplicates via processed_event. Duplicates in Kafka are a small overhead, but the relay stays simple.

Retries without an idempotency key — a dangerous combination

A separate trap is automatic retries (@Retry) without an idempotency key on monetary operations:

// Dangerous
@Retry(name = "payment")
public Receipt charge(Long orderId, Money amount) {
    return paymentClient.post("/charge", new ChargeRequest(orderId, amount));
}

The scenario:

  1. The first call sent a request to the payment provider — a network timeout.
  2. @Retry made a second call — also with a timeout.
  3. In reality both requests reached the provider (a timeout does not mean "not received").
  4. Without an idempotency key the provider processed both — the money was charged twice.

The right way is to pass a key that is generated once per operation and stays the same across all retries:

@Retry(name = "payment")
public Receipt charge(String idempotencyKey, Long orderId, Money amount) {
    return paymentClient.post(
        "/charge",
        new ChargeRequest(orderId, amount),
        Map.of("Idempotency-Key", idempotencyKey),
        Receipt.class
    );
}

By the key, the provider sees that this is a retry of the same operation and returns the previous result without charging again.

In short

  • Idempotency — an operation can be repeated with the same result. Needed everywhere a retry is possible due to a failure or shutdown.
  • HTTP POST for monetary operations: require an Idempotency-Key from the client; store the result in the database on the first call, return the stored one on a retry.
  • Kafka handler: a processed_event table with a unique constraint on (event_id, consumer_group) — the insert goes in the same transaction as the main action. Rolling back the transaction rolls back both.
  • Outbox-relay: two-phase publishing (PENDING → PUBLISHING → PUBLISHED) or protection on the consumer side via processed_event.
  • Retries + money: the idempotency key is created once per business operation and passed into all retries — otherwise each attempt may create a new charge.
  • Graceful shutdown gives time to finish, but does not guarantee it. Idempotency is the safety net for when finishing did not make it in time.