← Back to the section

Once a system has more than one service, those services need to exchange events. Kafka is one of the most common tools for this. Let's start from scratch: why it exists, how it works and where the pitfalls are.

The problem: services are coupled directly

Imagine a user places an order. OrderService has to notify PaymentService, InventoryService and NotificationService. The simplest option is direct HTTP calls.

But this approach has a problem: if PaymentService is unavailable, OrderService cannot finish its work either. One failing service drags the others down with it. This is called synchronous coupling — services depend on each other in real time.

Kafka solves this differently. It is a distributed message log: OrderService writes an "order created" event, and every interested service reads it on its own — each at its own pace. If NotificationService is temporarily down, it simply catches up on the missed events after it restarts.

diagram

OrderService does not know who reads from it. New consumers can be added without any changes to the sender's code.

Topic, partition, offset

Topic

A topic is a named message log. Services write to topics and read from them. The name reflects the domain: order-events, payment-events, inventory-changes.

Each message is a (key, value) pair plus metadata (timestamp, headers, offset):

key:    "order-12345"
value:  {"type":"OrderCreated","orderId":"12345","total":1500.00}

Partition

A topic is physically split into partitions. A partition is an ordered, immutable log: messages are only appended to the end, never changed and never deleted (until they expire under the retention policy).

diagram

Why partitions exist:

  • Parallelism — different consumers read different partitions at the same time.
  • Scaling — partitions are distributed across different brokers (servers).
  • Ordering — guaranteed only within a single partition.

Offset

Each message in a partition has a sequential number — the offset. A consumer remembers which offset it has read up to and continues from exactly there. This lets it re-read old messages or start from the beginning.

Producer and consumer

A producer is a client that writes to Kafka. A consumer is a client that reads from Kafka. In Go the standard choice is the segmentio/kafka-go library: a Writer for writing, a Reader for reading.

// Producer
type OrderEventPublisher struct {
	writer *kafka.Writer
}

func (p *OrderEventPublisher) Publish(ctx context.Context, event OrderCreatedEvent) error {
	payload, err := json.Marshal(event)
	if err != nil {
		return err
	}
	return p.writer.WriteMessages(ctx, kafka.Message{
		Key:   []byte(event.OrderID),
		Value: payload,
	})
}

// Consumer
reader := kafka.NewReader(kafka.ReaderConfig{
	Brokers: []string{"localhost:9092"},
	Topic:   "order-events",
	GroupID: "payment-service",
})

for {
	msg, err := reader.ReadMessage(ctx)
	if err != nil {
		break
	}
	var event OrderCreatedEvent
	if err := json.Unmarshal(msg.Value, &event); err != nil {
		continue
	}
	// processing...
}

Where a message lands: the role of the key

The producer picks a partition by this rule:

  1. Key provided → partition = hash(key) % number_of_partitions. All messages with the same key always land in the same partition.
  2. No key → messages are distributed across partitions in turn.

This matters for ordering. If all events of a single order must be in the correct order — use orderId as the key:

w := &kafka.Writer{
	Addr:     kafka.TCP("localhost:9092"),
	Topic:    "order-events",
	Balancer: &kafka.Hash{}, // partition = hash(key) % N
}

// Correct: key is orderId, all order events go to one partition
w.WriteMessages(ctx, kafka.Message{Key: []byte(event.OrderID), Value: payload})

// Wrong: with no key the events scatter across different partitions
w.WriteMessages(ctx, kafka.Message{Value: payload})

In kafka-go the partition is chosen by the Balancer: the hash(key) % N rule is what kafka.Hash gives you, while the default balancer distributes messages round-robin and ignores the key.

An important detail: if you increase the number of partitions in an existing topic, the formula hash(key) % N produces different results. Some messages will land in a different partition — and the ordering breaks. That is why you plan the partition count in advance with headroom and do not change it on the fly without migrating the data.

Message ordering

Within a single partition ordering is guaranteed — messages are stored and delivered in exactly the order they were written.

Across partitions ordering is undefined. If events of a single order land in different partitions, they may be read in an arbitrary order.

Takeaway: if you want ordered processing of all events for some object — use its identifier as the message key.

Consumer groups: how consumers share the work

A consumer group is several consumers with the same groupId. Kafka distributes the topic's partitions among them: each partition is assigned to exactly one consumer in the group.

diagram

What this gives you:

  • Parallelism is limited by the number of partitions: if there are 3 partitions and 5 consumers, two of them will sit idle.
  • Each message in the group is processed by exactly one consumer — the load is divided, not duplicated.
  • Different groups are independent: PaymentService and InventoryService both read order-events, but each with its own offset — as if each had its own copy of the topic.
diagram

When a consumer fails or a new one is added, Kafka redistributes the partitions — this is called a rebalance. During the redistribution the group does not process messages. Frequent rebalances slow things down, so you try to avoid them.

Commit offset: marking progress

A consumer tells Kafka which messages it has processed — this is called a commit offset. Without it, Kafka would not know where to continue after a restart.

Automatic commit (ReadMessage in kafka-go) commits the offset right when the message is read, regardless of whether it has been processed. Convenient but dangerous: if you crash after receiving but before processing — the offset is already committed and the message is lost.

Manual commit is more reliable:

for {
	msg, err := reader.FetchMessage(ctx)
	if err != nil {
		break
	}
	if err := processEvent(msg); err != nil {
		continue // don't commit → we re-read it after a restart
	}
	reader.CommitMessages(ctx, msg) // commit only after successful processing
}

With a manual commit a consumer may receive the same message twice (on a restart after a failure). This is called at-least-once — and it is the standard approach. The cost: the consumer must be able to process the same message several times without side effects (it must be idempotent).

Delivery guarantees

Kafka supports three modes:

ModeMeaningUse
At-most-onceDelivered or lost — no duplicatesMetrics, logs, telemetry
At-least-onceDelivered at least once, duplicates possibleMost business scenarios
Exactly-onceExactly once, no loss and no duplicatesFinancial operations sensitive to duplicates

At-least-once + an idempotent consumer is the standard choice for business events. Exactly-once is technically achievable (transactional producer + read-committed consumer), but it is harder and slower.

Idempotent producer

An ordinary producer may send a message twice if the acknowledgement does not arrive. With the enable.idempotence=true setting Kafka assigns each message a unique sequence number and discards the repeats. In segmentio/kafka-go the idempotent producer is not implemented — this setting is available in the confluent-kafka-go and franz-go clients:

// confluent-kafka-go
p, err := kafka.NewProducer(&kafka.ConfigMap{
	"bootstrap.servers":  "localhost:9092",
	"enable.idempotence": true,
	"acks":               "all",
	"retries":            2147483647,
})

This protects against duplicates on send retries. But if the service itself restarts and sends the message again — from Kafka's point of view that is a different message. So consumers still have to be idempotent.

Atomicity: writing to the database and sending an event

A common task: you need to save an order in the database and, at the same time, send an event to Kafka. If you save first and then send — a failure between these steps loses the event.

The solution is the Transactional Outbox: the event is written in the same database transaction as the order itself. A separate background process reads the unprocessed records and sends them to Kafka.

diagram

Nothing is lost (the write is atomic), and nothing extra is sent (the relay only sees completed records). This pattern is covered in more detail in the article «Distributed patterns».

Replication: what if a server goes down

Each partition is stored not on a single server but on several. One of them is the leader, the rest are replicas. Producers and consumers work only with the leader. The replicas catch up to the leader and form the ISR (In-Sync Replicas). If the leader goes down, one of the replicas becomes the new leader.

The producer specifies how strict an acknowledgement it needs (acks):

acksWhat it meansRisk
0Fire and forgetCan be lost
1The leader received itLoss if the leader fails before replication
allAll in-sync replicas received itReliable

For business events — always acks=all (in kafka-go — RequiredAcks: kafka.RequireAll on the Writer).

Message retention

Kafka is a log, and it has a configurable retention period:

  • By time (retention.ms) — for example, 7 days.
  • By size (retention.bytes) — for example, 1 TB per partition.
  • Compaction (cleanup.policy=compact) — only the last message for each key is kept. Suitable for storing current state: stock levels, current prices, user profiles.
# Compacted topic — current stock levels
inventory-current:
  cleanup.policy: compact

# Ordinary event topic — 7 days
order-events:
  cleanup.policy: delete
  retention.ms: 604800000

When you don't need Kafka

Kafka works well for high-throughput event streams. But it is not for everything:

  • You need a response to a request — Kafka is one-directional, request-response through it is clunky. HTTP is better.
  • Few events — Kafka is optimized for high load; for a handful of events a day it is overkill.
  • You need complex rule-based routing — RabbitMQ with routing rules is a better fit.

In short

  • A topic is a named event log. A partition is its physical part, ordered and immutable.
  • The message key determines the partition: one key → one partition → ordering guaranteed. In kafka-go this requires Balancer: &kafka.Hash{}.
  • The offset is the consumer's position. Commit the offset after processing, not before: FetchMessage + CommitMessages.
  • A consumer group splits partitions among consumers. Several groups read the same topic independently.
  • The standard choice: at-least-once + manual commit + an idempotent consumer.
  • For "database + Kafka" atomicity — the Transactional Outbox pattern.
  • Replication protects against data loss when a server fails. For reliability — acks=all.
  • Retention is configured by time or size; compaction keeps only the latest version for each key.
  • Kafka in production — Spring Kafka, Dead Letter Queue, Schema Registry, consumer lag, performance tuning, security, KRaft.
  • Distributed patterns — Outbox + polling-relay, Saga, Idempotent Consumer in detail.
  • CQRS — Kafka as the channel between the command and read models.