Kafka shutdown — how to stop the consumer and producer correctly

← Back to the section

When an application stops, the Kafka consumer and producer must be shut down carefully. Otherwise one of two problems occurs: either messages get processed again on the next startup, or some of the messages the producer has already sent never reach the broker.

Let's go through both cases in order.

What happens to the consumer on shutdown

A Kafka consumer reads messages in batches. After processing, it records its progress in the broker — this is called "commit the offset". If the application is killed before the commit, the consumer will read the same messages again on the next startup (replay).

Spring manages consumers through ConcurrentMessageListenerContainer. When it receives a stop signal, it calls stop(timeout) on each container: it waits for the current batch to finish, then commits the offset and closes the KafkaConsumer.

The key setting is the wait time:

spring:
  kafka:
    listener:
      shutdown-timeout: 20s
      ack-mode: BATCH

shutdown-timeout: 20s is the maximum time Spring waits for the current batch to finish on each container. If the listener does not finish in time, Spring interrupts it forcibly, the offset is not committed, and the messages arrive again on the next startup.

Why you should not do long operations inside a listener

The listener processes a message and stays within shutdown-timeout. The problem arises when the listener makes several HTTP calls with retries:

// Dangerous — total time 65s, will not fit into shutdown-timeout of 20s
@KafkaListener(...)
public void onConfirmed(OrderConfirmedEvent event, Acknowledgment ack) {
    paymentClient.charge(...);        // 5s + retries 30s
    notificationClient.send(...);     // 3s + retries 15s
    analyticsClient.track(...);       // 2s + retries 10s
    ack.acknowledge();
}

In the worst case the listener takes 65 seconds. shutdown-timeout: 20s expires, Spring interrupts the processing, the offset is not committed. On restart the event arrives again — and if paymentClient.charge already went through, you get a double charge.

The solution is for the listener to do only fast local work and defer the "heavy" operations through the outbox pattern:

@KafkaListener(...)
@Transactional
public void onConfirmed(OrderConfirmedEvent event, Acknowledgment ack) {
    if (!processedEventRepository.tryMarkProcessed(event.eventId(), "billing")) {
        ack.acknowledge();
        return;
    }
    billingService.recordChargeIntent(event.orderId(), event.totalAmount());
    outboxRepository.append(
        new ChargePaymentRequested(event.orderId(), event.totalAmount())
    );
    ack.acknowledge();
}

The listener saves the intent to the database and puts the event into the outbox — all in a single local transaction that completes in milliseconds. The actual HTTP call to the payment service is made by a separate outbox processor. Shutdown goes through cleanly.

Message acknowledgment mode — ack-mode

ack-mode determines exactly when the consumer tells the broker "I have processed these messages". This matters for shutdown behavior.

For most cases BATCH is a good fit: one commit for the whole batch, minimal overhead, and predictable behavior on shutdown — either the whole batch is processed and committed, or it comes back for a full replay.

The most dangerous mistake — enable.auto.commit

There is a setting that seems convenient but leads to message loss when the application stops:

# Dangerous
spring:
  kafka:
    consumer:
      enable-auto-commit: true
      auto-commit-interval-ms: 5000

Here is how the loss happens. A separate thread automatically commits the current offset every 5 seconds — regardless of whether the messages have actually been processed. When the application stops, this thread may commit offset 200 even though only messages 100–149 have really been processed. After restart the consumer starts reading from 200 — messages 150–200 are lost forever.

The correct approach is enable-auto-commit: false and managing commits explicitly via ack-mode: BATCH or MANUAL.

The producer — flush on shutdown

A Kafka producer does not send each message immediately: it accumulates messages in a buffer and flushes them in batches. If the application is stopped before the flush, the accumulated messages are lost.

On shutdown, Spring Boot automatically calls KafkaTemplate.destroy(), which runs Producer.close(Duration.ofSeconds(30)). This flushes all accumulated messages to the broker and closes the connection. If you use the standard KafkaTemplate, there is nothing extra to configure.

If the project uses KafkaProducer directly (a non-standard situation), you need to set destroyMethod explicitly:

@Bean(destroyMethod = "")
KafkaProducer<String, Object> rawProducer() {
    return new KafkaProducer<>(producerProperties());
}

@PreDestroy
public void close() {
    producer.close(Duration.ofSeconds(15)); // explicit flush with a timeout
}

Without an explicit close(Duration) call, the accumulated messages are lost.

In short

• On shutdown Spring calls ConcurrentMessageListenerContainer.stop(timeout) — the consumer waits for the current batch to finish and commits the offset.
• spring.kafka.listener.shutdown-timeout: 20s is how long to wait for completion on each container; if the listener does not make it, the offset is not committed and the messages arrive again.
• The listener must finish quickly — heavy operations with retries are moved to the outbox, otherwise the application will not fit within the timeout.
• ack-mode: BATCH is the recommended mode: one commit per batch, with predictable behavior on shutdown.
• enable.auto.commit: true is dangerous: the offset may be committed before the actual processing, and on shutdown messages are lost irreversibly.
• KafkaTemplate automatically flushes the producer buffer on shutdown; a direct KafkaProducer needs an explicit close(Duration).

What to read next

• JVM/Spring configuration — basic graceful shutdown setup in Spring Boot.
• Idempotency on shutdown — how to protect against reprocessing on replay.
• Budgets and observability — Kafka within the overall shutdown time budget.