AMQP vs Kafka: which broker to choose

← Back to the section

At some point an application stops communicating only over HTTP. The number of services grows, and you need to pass messages between them — so that the sender doesn't wait for a reply and doesn't crash if the receiver is temporarily unavailable. This is where a message broker comes in.

The two most popular ones are RabbitMQ (which implements the AMQP protocol) and Apache Kafka. At first glance they do the same thing — accept and hand out messages. But they are built in fundamentally different ways.

The main difference: a task or a log entry

Here is the key difference, from which everything else follows.

RabbitMQ treats a message as a task. It arrives — someone picks it up — completes it — the message disappears. Like a sticky note with an errand on the fridge: read it, do it, take it off.

Kafka treats a message as a log entry. It arrives — it gets written down — it stays as long as configured. Any reading service can, at any moment, start reading from the beginning or from any position. Like an accounting ledger: every operation is recorded forever, and you can re-read the history.

This difference determines which tool fits your problem.

When you need RabbitMQ

Imagine an online store. A user placed an order, and you need to send them an email. It makes sense to put a task in a queue: "send an email to user #12345." One of the worker processes will take the task, send the email, and "take the sticky note off." The others don't need the task — it has been done.

RabbitMQ was built for exactly this scenario: distributing tasks among workers. The broker itself pushes messages to the consumer and manages ordering.

Typical cases for RabbitMQ:

• Sending emails and notifications
• Processing images or documents in the background
• Calling a service without waiting for a reply (asynchronous RPC)
• Broadcasting one event to dozens of services (for example, "config updated")
• Tasks with a deadline: "if not processed within 5 minutes — discard"

When you need Kafka

Now a different scenario. The same online store records everything that happens: "order created," "payment processed," "goods shipped." A month later the analytics team wants to look at all events from this period. Another month later an ML service launches — it also needs the entire history.

If you use RabbitMQ, the history is already deleted. The messages disappeared after processing.

Kafka keeps all records. A new service can start reading from the very beginning and "catch up" to the present moment. That is exactly why Kafka is called an event log.

Kafka consumers pull messages themselves at their own pace. If one service falls behind — the log waits for it.

Typical cases for Kafka:

• An event log with the ability to re-read history
• High load (hundreds of thousands of events per second)
• Several independent teams reading one data stream
• Real-time streaming analytics
• Collecting events from databases (CDC via Debezium)
• Event Sourcing — when events are primary and state is secondary

Seven comparison criteria

1. The nature of the message

Start with the question: what are you sending?

A command — an instruction to do something. "Send an email," "process a file." Once done, it is no longer needed. → RabbitMQ.

An event — a fact that something happened. "Order created," "user registered." It may be needed by many services, now and in the future. → Kafka.

2. Message order

In Kafka you can specify a key (for example, userId) — and all events for that user will go into one partition, strictly in order. At the same time, different users are processed in parallel. RabbitMQ has nothing like this "out of the box."

3. Throughput

Rough guidelines on a single node:

Kafka was built for huge volumes: sequential writes to disk, batch processing. For IoT telemetry, clickstreams, log collection — Kafka is the only realistic option. RabbitMQ works comfortably at thousands of messages per second.

4. History retention

RabbitMQ: a message disappears after it has been taken and acknowledged. There is no way to re-read it.

Kafka: messages are kept for a configurable time (7 days by default, and it can be longer). A new service that appears a week later will read everything from the very beginning.

5. Routing flexibility

RabbitMQ manages routing at the broker level. The sender publishes to an exchange, and the broker distributes messages to queues according to rules. You can dynamically change who receives what — the sender doesn't know about it.

Kafka is built more simply: the sender chooses a topic, and readers subscribe to the topic. Adding a new reader is just creating a new consumer group, with no changes on the sender's side.

6. Delivery model

RabbitMQ (push): the broker delivers messages to the consumer itself. If the consumer can't keep up — the broker applies backpressure.

Kafka (pull): the consumer requests the next batch itself, at its own pace. If it falls behind — the log waits for it. The broker is not overwhelmed by slow consumers.

In practice: Kafka handles uneven load better. A spike of writes to the log doesn't disturb slow readers.

7. Operational complexity

Kafka is a serious operational investment. Set it up correctly once — and it runs for years. But if the team is small and there's no experienced administrator, RabbitMQ will be more predictable.

When to use both together

This is not "let's take both just in case," but a specific architecture.

A service receives commands via RabbitMQ (process an image, send an email). After completion, it publishes the fact to Kafka (image processed, email sent). Analytics, monitoring, and ML services read Kafka and know nothing about the command queue.

HTTP → RabbitMQ (command) → Service → Kafka (event)
                                          ↓
                               analytics, monitoring, ML

Commands are atomic and disappear after processing. Events accumulate for history.

Common mistakes when choosing

"Let's take RabbitMQ, then move to Kafka later" — if the data is needed as an event log, start with Kafka right away. The switch means rewriting logic, not swapping a client.

"Kafka, because big companies use it" — big companies process tens of millions of events per second. At a few thousand events per day, RabbitMQ is simpler and cheaper to run.

"We'll build a task queue on top of Kafka" — technically possible, but Kafka is not optimized for short-lived tasks. Distributing work tasks is RabbitMQ's job.

"We'll store user requests in Kafka like in a database" — Kafka is not a database. If you need arbitrary search over the content — that's an event store plus PostgreSQL or similar, not raw Kafka.

In short

• RabbitMQ = a task queue. A message disappears after processing. The broker pushes to the consumer.
• Kafka = an event log. Messages are stored and can be read again. The consumer pulls at its own pace.
• Choose RabbitMQ for distributing tasks among workers, asynchronous RPC, and flexible routing.
• Choose Kafka for storing event history, high load, and streaming analysis.
• Ordering events by key is built into Kafka; in RabbitMQ it takes extra effort.
• Kafka is operationally more complex — it's justified when you truly need its capabilities.
• Both together is not "just in case," but a specific pattern: RabbitMQ for commands, Kafka for events.

What to read next

• The AMQP protocol — how the RabbitMQ delivery model works.
• Messaging patterns via AMQP — work queue, pub/sub, RPC, idempotency.
• Distributed patterns — Saga, Outbox, Idempotent Consumer (applicable to both brokers).