Apache Kafka in Python: topics, partitions, ordering and guarantees

← Back to the section

Once a system has more than one service, those services need to exchange events. Kafka is one of the most common tools for this. Let's start from scratch: why it exists, how it works and where the pitfalls are.

The problem: services are coupled directly

Imagine a user places an order. OrderService has to notify PaymentService, InventoryService and NotificationService. The simplest option is direct HTTP calls.

But this approach has a problem: if PaymentService is unavailable, OrderService cannot finish its work either. One failing service drags the others down with it. This is called synchronous coupling — services depend on each other in real time.

Kafka solves this differently. It is a distributed message log: OrderService writes an "order created" event, and every interested service reads it on its own — each at its own pace. If NotificationService is temporarily down, it simply catches up on the missed events after it restarts.

OrderService does not know who reads from it. New consumers can be added without any changes to the sender's code.

Topic, partition, offset

Topic

A topic is a named message log. Services write to topics and read from them. The name reflects the domain: order-events, payment-events, inventory-changes.

Each message is a (key, value) pair plus metadata (timestamp, headers, offset):

key:    "order-12345"
value:  {"type":"OrderCreated","orderId":"12345","total":1500.00}

Partition

A topic is physically split into partitions. A partition is an ordered, immutable log: messages are only appended to the end, never changed and never deleted (until they expire under the retention policy).

Why partitions exist:

Parallelism — different consumers read different partitions at the same time.
Scaling — partitions are distributed across different brokers (servers).
Ordering — guaranteed only within a single partition.

Offset

Each message in a partition has a sequential number — the offset. A consumer remembers which offset it has read up to and continues from exactly there. This lets it re-read old messages or start from the beginning.

Producer and consumer

A producer is a client that writes to Kafka. A consumer is a client that reads from Kafka.

# Producer
class OrderEventPublisher:
    def __init__(self, producer: AIOKafkaProducer):
        self._producer = producer

    async def publish(self, event: OrderCreatedEvent) -> None:
        await self._producer.send_and_wait(
            "order-events",
            key=event.order_id.encode(),
            value=to_json(event).encode(),
        )

# Consumer
async def payment_consumer() -> None:
    consumer = AIOKafkaConsumer("order-events", group_id="payment-service")
    await consumer.start()
    try:
        async for record in consumer:
            event = from_json(record.value, OrderCreatedEvent)
            # processing...
    finally:
        await consumer.stop()

Where a message lands: the role of the key

The producer picks a partition by this rule:

Key provided → partition = hash(key) % number_of_partitions. All messages with the same key always land in the same partition.
No key → messages are distributed across partitions in turn.

This matters for ordering. If all events of a single order must be in the correct order — use order_id as the key:

# Correct: key is order_id, all order events go to one partition
await producer.send_and_wait("order-events", key=event.order_id.encode(), value=payload)

# Wrong: with no key the events scatter across different partitions
await producer.send_and_wait("order-events", value=payload)

An important detail: if you increase the number of partitions in an existing topic, the formula hash(key) % N produces different results. Some messages will land in a different partition — and the ordering breaks. That is why you plan the partition count in advance with headroom and do not change it on the fly without migrating the data.

Message ordering

Within a single partition ordering is guaranteed — messages are stored and delivered in exactly the order they were written.

Across partitions ordering is undefined. If events of a single order land in different partitions, they may be read in an arbitrary order.

Takeaway: if you want ordered processing of all events for some object — use its identifier as the message key.

A consumer group is several consumers with the same group_id. Kafka distributes the topic's partitions among them: each partition is assigned to exactly one consumer in the group.

What this gives you:

Parallelism is limited by the number of partitions: if there are 3 partitions and 5 consumers, two of them will sit idle.
Each message in the group is processed by exactly one consumer — the load is divided, not duplicated.
Different groups are independent: PaymentService and InventoryService both read order-events, but each with its own offset — as if each had its own copy of the topic.

When a consumer fails or a new one is added, Kafka redistributes the partitions — this is called a rebalance. During the redistribution the group does not process messages. Frequent rebalances slow things down, so you try to avoid them.

Commit offset: marking progress

A consumer tells Kafka which messages it has processed — this is called a commit offset. Without it, Kafka would not know where to continue after a restart.

Automatic commit (the default) commits the offset every 5 seconds regardless of whether the message has been processed. Convenient but dangerous: if you crash after receiving but before processing — the offset is already committed and the message is lost.

Manual commit is more reliable:

consumer = AIOKafkaConsumer(
    "order-events",
    group_id="payment-service",
    enable_auto_commit=False,
)
await consumer.start()
try:
    async for record in consumer:
        await process_event(record)
        await consumer.commit()  # commit only after successful processing
        # an exception in process_event skips the commit → we re-read it after a restart
finally:
    await consumer.stop()

With a manual commit a consumer may receive the same message twice (on a restart after a failure). This is called at-least-once — and it is the standard approach. The cost: the consumer must be able to process the same message several times without side effects (it must be idempotent).

Delivery guarantees

Kafka supports three modes:

Mode	Meaning	Use
At-most-once	Delivered or lost — no duplicates	Metrics, logs, telemetry
At-least-once	Delivered at least once, duplicates possible	Most business scenarios
Exactly-once	Exactly once, no loss and no duplicates	Financial operations sensitive to duplicates

At-least-once + an idempotent consumer is the standard choice for business events. Exactly-once is technically achievable (transactional producer + read-committed consumer), but it is harder and slower.

Idempotent producer

An ordinary producer may send a message twice if the acknowledgement does not arrive. With the enable_idempotence=True setting Kafka assigns each message a unique sequence number and discards the repeats:

producer = AIOKafkaProducer(
    enable_idempotence=True,
    acks="all",
)

This protects against duplicates on send retries. But if the service itself restarts and sends the message again — from Kafka's point of view that is a different message. So consumers still have to be idempotent.

Atomicity: writing to the database and sending an event

A common task: you need to save an order in the database and, at the same time, send an event to Kafka. If you save first and then send — a failure between these steps loses the event.

The solution is the Transactional Outbox: the event is written in the same database transaction as the order itself. A separate background process reads the unprocessed records and sends them to Kafka.

Nothing is lost (the write is atomic), and nothing extra is sent (the relay only sees completed records). This pattern is covered in more detail in the article «Distributed patterns».

Replication: what if a server goes down

Each partition is stored not on a single server but on several. One of them is the leader, the rest are replicas. Producers and consumers work only with the leader. The replicas catch up to the leader and form the ISR (In-Sync Replicas). If the leader goes down, one of the replicas becomes the new leader.

The producer specifies how strict an acknowledgement it needs (acks):

`acks`	What it means	Risk
`0`	Fire and forget	Can be lost
`1`	The leader received it	Loss if the leader fails before replication
`all`	All in-sync replicas received it	Reliable

For business events — always acks=all.

Message retention

Kafka is a log, and it has a configurable retention period:

By time (retention.ms) — for example, 7 days.
By size (retention.bytes) — for example, 1 TB per partition.
Compaction (cleanup.policy=compact) — only the last message for each key is kept. Suitable for storing current state: stock levels, current prices, user profiles.

# Compacted topic — current stock levels
inventory-current:
  cleanup.policy: compact

# Ordinary event topic — 7 days
order-events:
  cleanup.policy: delete
  retention.ms: 604800000

When you don't need Kafka

Kafka works well for high-throughput event streams. But it is not for everything:

You need a response to a request — Kafka is one-directional, request-response through it is clunky. HTTP is better.
Few events — Kafka is optimized for high load; for a handful of events a day it is overkill.
You need complex rule-based routing — RabbitMQ with routing rules is a better fit.

In short

A topic is a named event log. A partition is its physical part, ordered and immutable.
The message key determines the partition: one key → one partition → ordering guaranteed.
The offset is the consumer's position. Commit the offset after processing, not before.
A consumer group splits partitions among consumers. Several groups read the same topic independently.
The standard choice: at-least-once + manual commit + an idempotent consumer.
For "database + Kafka" atomicity — the Transactional Outbox pattern.
Replication protects against data loss when a server fails. For reliability — acks=all.
Retention is configured by time or size; compaction keeps only the latest version for each key.