Kafka in production: Spring Kafka, DLQ, Schema Registry, tuning, security

← Back to the section

If you already know what a topic, a partition, and a consumer group are, this article is about the next step: how to connect Kafka to a Spring Boot application and what you need to configure so that it runs reliably in production.

Spring Kafka: KafkaTemplate and @KafkaListener

The spring-kafka library wraps the standard Java Kafka client and gives you a more convenient API. To send messages you use KafkaTemplate; to receive them you use the @KafkaListener annotation.

// Send
@Component
@RequiredArgsConstructor
public class OrderEventPublisher {
    private final KafkaTemplate<String, OrderEvent> kafka;

    public void publish(OrderEvent event) {
        kafka.send("orders", event.orderId().toString(), event);
    }
}

// Receive
@Component
public class OrderEventListener {
    @KafkaListener(topics = "orders", groupId = "billing-service")
    public void handle(ConsumerRecord<String, OrderEvent> record) {
        var event = record.value();
        // processing
    }
}

Underneath @KafkaListener lives a ConcurrentKafkaListenerContainerFactory — it holds a pool of consumer threads. By default there is one thread, but you can request more (concurrency = 3) — then each thread serves its own set of partitions.

AckMode: when Kafka "knows" a message has been processed

Kafka tracks how far a consumer has progressed through the messages via an offset commit. AckMode determines when that commit happens:

• BATCH — after the whole batch of messages has been processed successfully (the default value).
• RECORD — after each message. Safer on failures, but slower.
• MANUAL / MANUAL_IMMEDIATE — the application calls Acknowledgment.acknowledge() itself. Needed when it matters to commit the offset only after a successful write to the database.

@Bean
ConcurrentKafkaListenerContainerFactory<String, OrderEvent> kafkaListenerContainerFactory(
        ConsumerFactory<String, OrderEvent> consumerFactory) {
    var factory = new ConcurrentKafkaListenerContainerFactory<String, OrderEvent>();
    factory.setConsumerFactory(consumerFactory);
    factory.setConcurrency(3);
    factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
    return factory;
}

Message headers

Every Kafka message can carry headers — (key, value) pairs in bytes, separate from the payload. This is where you put technical metadata that does not belong in the business data:

• X-Correlation-ID / traceparent — for distributed tracing.
• X-Event-Version — the version of the event format.
• X-Source-Service — where the message came from.

// When sending
ProducerRecord<String, OrderEvent> record = new ProducerRecord<>("orders", key, event);
record.headers().add("X-Correlation-ID", correlationId.getBytes());
kafka.send(record);

// When receiving
@KafkaListener(topics = "orders")
public void handle(
        @Payload OrderEvent event,
        @Header("X-Correlation-ID") String correlationId) {
    MDC.put("correlationId", correlationId);
    // processing
}

Business data (orderId, amount, status) goes in the payload. Everything technical goes in the headers.

Dead Letter Queue: what to do with problematic messages

If a handler throws an exception, by default Spring Kafka keeps trying to process the same message over and over — the offset does not move and the consumer stalls. In production this means the whole group grinds to a halt.

The solution is a Dead Letter Queue (DLQ): after several failed attempts the message is moved to a separate topic (usually with a .DLT suffix), the offset is committed, and the main consumer keeps working.

The simple way: DefaultErrorHandler

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template,
        (record, ex) -> new TopicPartition("orders.DLT", record.partition()));
    return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3));
}

Here: 3 attempts with a 1-second interval, and after the third failure the message goes to orders.DLT. The DLT message automatically gets headers with the error text and the name of the original topic.

The modern way: @RetryableTopic

As of Spring Kafka 2.7+ you can do the same thing declaratively:

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 1000, multiplier = 2.0),
    autoCreateTopics = "false",
    dltStrategy = DltStrategy.FAIL_ON_ERROR
)
@KafkaListener(topics = "orders", groupId = "billing-service")
public void handle(OrderEvent event) {
    // processing
}

@DltHandler
public void handleDlt(OrderEvent event, @Header("kafka_dlt-exception-message") String reason) {
    log.error("Failed to process: order={} reason={}", event.orderId(), reason);
}

Spring automatically creates the topics orders-retry-0, orders-retry-1, orders-retry-2, orders-dlt. Messages "cool off" in the retry topics without blocking the main flow.

When to use which

• Temporary problem (database down, external service not responding) → @RetryableTopic with an increasing delay.
• Permanent problem (invalid message, wrong format) → DefaultErrorHandler with no retries, straight to the DLQ.
• Need to distinguish error types → DefaultErrorHandler.addNotRetryableExceptions(IllegalArgumentException.class) — the listed exceptions go straight to the DLQ, everything else gets retried.

Schema Registry: how not to break your neighbors when the schema changes

When the producer and the consumer are different services from different teams, any change to the event structure can potentially break the consumer. Schema Registry solves this with a centralized schema store that checks compatibility.

How it works:

1. The producer registers the schema in Schema Registry and gets a numeric identifier (schema_id).
2. Into each message it writes 4 bytes with that identifier, followed by the compressed payload.
3. The consumer reads the schema_id, downloads the schema from Schema Registry once (then caches it), and deserializes the payload.

Avro, Protobuf, or JSON Schema

The standard in the Kafka ecosystem is Avro: a compact binary format with good support for schema evolution via the avro-maven-plugin. Protobuf is chosen by teams with gRPC infrastructure. JSON Schema is human-readable but takes several times more space — it is good for debugging.

Compatibility modes

Schema Registry checks whether a new version of the schema will break consumers that are already running:

• BACKWARD (the default) — a new consumer can read old messages. You can remove optional fields and add optional ones with a default value. You cannot add required fields.
• FORWARD — an old consumer can read new messages. The mirror image of BACKWARD.
• FULL — both modes at once. The strictest.
• NONE — no checks. Only for special cases.

Rule of thumb: BACKWARD for topics where there are many consumers and a single producer — that is the typical event bus.

Consumer lag: why the consumer falls behind

Lag is the difference between the last message in a partition and the one the consumer has reached. If lag is growing, the consumer cannot keep up with the producer. This is the main health metric for a consumer in production.

Typical causes:

1. Slow processing — each message makes a synchronous call to a database or an external service. Fix: reduce max.poll.records, parallelize processing, switch to batch requests.
2. Too few partitions per group — increase the container's concurrency or add service instances (but no more than the number of partitions).
3. max.poll.interval.ms expires — if processing one batch takes longer than 5 minutes (the default value), Kafka considers the consumer dead and triggers a partition rebalance. Fix: reduce max.poll.records or raise max.poll.interval.ms.
4. JVM / GC pauses — the virtual machine froze for 30 seconds and lag grew by thousands of messages.

Check the lag manually:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
    --group billing-service --describe
# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          124530          124530          0
# orders  1          124450          124530          80   ← falling behind

Metric to monitor: kafka_consumergroup_lag. Alert: lag is above N and has been growing for N minutes in a row.

Performance tuning

Kafka works well out of the box, but there are a few parameters worth knowing.

Producer

Consumer

The best compression algorithm for most cases is zstd (Kafka 2.1+): it compresses almost as well as gzip and runs almost as fast as lz4.

Security: SASL, SSL, ACL

By default Kafka listens on an open port with no authentication — you never leave it that way in production. Three layers of protection:

• SSL/TLS — connection encryption. Certificates on the brokers and clients.
• SASL — authentication. SASL/PLAIN — username and password (only over SSL). SASL/SCRAM-SHA-256 — safer: challenge-response, the password is not sent over the network. SASL/OAUTHBEARER — OAuth2 tokens, for integration with an identity provider.
• ACL — authorization: who is allowed to read from and write to which topic. Managed via kafka-acls.sh.

Typical setup: one SASL user per service, with ACLs restricting its access strictly to the topics it needs.

# application.properties
spring.kafka.properties.security.protocol=SASL_SSL
spring.kafka.properties.sasl.mechanism=SCRAM-SHA-256
spring.kafka.properties.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="billing-service" password="${KAFKA_PASSWORD}";
spring.kafka.properties.ssl.truststore.location=/etc/kafka/truststore.jks
spring.kafka.properties.ssl.truststore.password=${TRUSTSTORE_PASSWORD}

KRaft: Kafka without ZooKeeper

Kafka used to require a separate ZooKeeper cluster to store metadata: which topics exist, who is the partition leader, which ACLs are in effect. ZooKeeper is a separate system that has to be deployed, monitored, and fixed on its own.

As of version 3.3 (2022) there is KRaft — a native Raft-based protocol running inside the brokers themselves. ZooKeeper is no longer needed. In Kafka 4.x ZooKeeper support has been removed entirely — all new clusters run on KRaft only.

For a developer this is almost invisible: bootstrap-servers and client behavior have not changed. The only thing that changes is how the ops team deploys and maintains the cluster.

In short

• KafkaTemplate sends, @KafkaListener receives. Thread-pool and AckMode configuration goes into ConcurrentKafkaListenerContainerFactory.
• AckMode controls when the offset is committed: BATCH (default), RECORD, MANUAL.
• Message headers are the place for technical metadata (trace-id, schema version). Business data goes in the payload.
• The DLQ saves you from a consumer hanging on a problematic message. @RetryableTopic is the declarative way, with ready-made retry topics.
• Schema Registry stores schemas and checks compatibility on changes. BACKWARD mode — a new consumer reads old messages.
• Consumer lag is the main health metric for a consumer. Growing → the consumer cannot keep up.
• Producer tuning: batch.size, linger.ms, compression.type=zstd. Consumer tuning: max.poll.records, max.poll.interval.ms.
• Security: SSL for encryption, SASL for authentication, ACL for per-topic authorization.
• KRaft: as of Kafka 3.3+ ZooKeeper is not needed; in Kafka 4.x ZooKeeper support has been removed entirely.

What to read next

• Kafka fundamentals — broker internals, partitions, delivery guarantees, retention.
• Distributed patterns — Saga, Outbox, Idempotent Consumer.
• Resilience patterns — Circuit Breaker and Timeout in the context of event-driven systems.