← Back to the section

At some point an application stops communicating only over HTTP. Services multiply, and you need to pass messages between them - so that the sender does not wait for a reply and does not crash when the receiver is temporarily unavailable. This is where a message broker comes in.

The most popular ones: NATS, RabbitMQ (an implementation of the AMQP protocol) and Apache Kafka. At first glance they do the same thing - accept and hand out messages. But they are built in fundamentally different ways.

The easiest way to see the difference is to follow the same stream of events through them.

The main difference: a task or a log entry

Here is the key difference from which everything else follows.

RabbitMQ treats a message as a task. It arrives - someone picks it up - completes it - the message disappears. Like a sticky note with an errand on the fridge: read it, do it, take it down.

Kafka treats a message as a log entry. It arrives - it gets written down - it stays for as long as configured. Any reading service can start reading from the beginning or from any position at any moment. Like an accounting ledger: every operation is recorded for good, and the history can be re-read.

This difference determines which tool fits your problem.

When you need RabbitMQ

Imagine an online store. A user has placed an order, and you need to send them an email. It makes sense to put a task in a queue: "send an email to user #12345." One of the worker processes takes the task, sends the email and "takes the sticky note down." The others do not need the task - it is done.

RabbitMQ was built for exactly this scenario: distributing tasks among workers. The broker itself pushes messages to the consumer and manages the order.

Typical cases for RabbitMQ:

• Sending emails and notifications
• Processing images or documents in the background
• Calling a service without waiting for a reply (asynchronous RPC)
• Broadcasting one event to dozens of services (for example, "config updated")
• Tasks with a deadline: "if not processed within 5 minutes - discard"

When you need Kafka

Now a different scenario. The same online store records everything that happens: "order created," "payment captured," "goods shipped." A month later the analytics team wants to look at all the events from that period. Another month later an ML service launches - it also needs the entire history.

With RabbitMQ the history is already gone. The messages disappeared after processing.

Kafka keeps every record. A new service can start reading from the very beginning and "catch up" to the present moment. That is exactly why Kafka is called an event log.

Kafka consumers pull messages themselves at their own pace. If one service falls behind - the log waits for it.

Typical cases for Kafka:

• An event log with the ability to re-read history
• High load (hundreds of thousands of events per second)
• Several independent teams reading one data stream
• Real-time streaming analytics
• Capturing events from databases (CDC via Debezium)
• Event Sourcing - when events are primary and state is derived

When you need NATS

NATS is a lightweight subject-based messaging bus. A single binary under ten megabytes, millions of messages per second, microsecond latency. It does not store events (in core mode) and knows nothing about exchange types, but it starts in a second.

A subject is a dot-separated string: orders.new.eu. The wildcard * replaces one word, > replaces all the rest. Subscribe to orders.> and you get everything about orders, with no exchanges or bindings to configure.

People choose NATS when:

• they need a light internal bus for microservices: a "notify everyone" event without complex routing;
• edge and IoT: a device running NATS weighs less than the JVM needed for a Kafka client;
• request-reply: a built-in ask-and-answer pattern in one line, without creating reply queues;
• speed matters more than guarantees: telemetry, metrics, logs - losing a few messages is not a disaster.

For persistence there is JetStream - a layer that stores events in streams and supports acks, like consumer groups in Kafka. But if you need a full-blown log with weeks of retention and the Connect/Streams ecosystem - take Kafka.

Seven comparison criteria

1. The nature of the message

Start with the question: what are you sending?

A command is an instruction to do something. "Send an email," "process a file." Once executed, it is no longer needed. → RabbitMQ.

An event is a fact that something happened. "Order created," "user registered." It may be needed by many services, now and in the future. → Kafka.

2. Message order

In RabbitMQ order breaks not inside the queue but on the way out of it. The queue hands out messages in order, but two consumers of one queue receive them in turn and process them in parallel - the second may finish before the first. And if the first message is rejected and returns to the queue, it lands behind the messages that were handed out after it.

	RabbitMQ	Kafka
Order guarantee	Within a single queue	Within a single partition
Parallel processing while preserving order	Possible, but with add-ons on top of the model	Natural via the partition key

In Kafka you can specify a key (for example, userId) - and all events for that user go into one partition, strictly in order, while different users are processed in parallel. In RabbitMQ the same thing can be assembled, but from separate parts: a consistent-hash exchange spreads messages across queues by the hash of the key, single active consumer mode keeps one reader per queue at a time, and streams offer super streams with partitioning. All of this works - it is just not the built-in model but add-ons you have to know about and enable.

3. Throughput

Rough guidelines for a single node:

	Throughput
RabbitMQ (reliable mode)	30-50K messages/sec
Kafka (reliable mode)	hundreds of thousands, up to about a million messages/sec at peak

These numbers are an order of magnitude, not a promise. They depend heavily on the conditions: message size, batching on the sender side, the number of partitions, and how many replicas must acknowledge a write. Kafka's upper bound comes from small messages, large batches and many partitions; with bigger messages and acks from all replicas it drops several times over.

Kafka was built for huge volumes: sequential disk writes, batch processing. For IoT telemetry, clickstreams and log collection, Kafka is the only realistic option. RabbitMQ works comfortably at thousands of messages per second.

4. History retention

RabbitMQ: in a regular queue a message disappears once it has been taken and acknowledged - there is no re-reading it. The exception is streams, introduced in version 3.9: they are structured like a log and allow reading the same data many times, but that is a separate type, not how queues behave.

Kafka: messages are kept for a configurable time (7 days by default, and it can be longer). A service that appears a week later reads everything from the very beginning.

Task	Better
Restart processing over old data	Kafka
A new service must receive the history	Kafka
Audit and storage of all events	Kafka
Deliver and forget	RabbitMQ

5. Routing flexibility

RabbitMQ manages routing at the broker level. The sender publishes to an exchange, and the broker distributes messages to queues according to rules. Who receives what can be changed dynamically - the sender knows nothing about it. The analytics department wants all payment events: an administrator creates a new queue and binds it to the same exchange with the key payment.*, while the payment service's code does not change and the service is not even restarted.

Kafka moves routing from the broker into the code. The sender picks a topic and a key - the key and the partitioner decide which partition the record lands in; a reader subscribes to a topic or to a whole pattern of topics at once. Adding a new reader is easy: a new consumer group, no changes to the sender. But changing the rules themselves - "these events now go to a separate stream" - means creating new topics and shipping code on both sides, whereas in RabbitMQ it is a binding edit on the broker.

6. Delivery model

RabbitMQ (push): the broker delivers messages to the consumer itself. If the consumer cannot keep up - the broker applies backpressure.

Kafka (pull): the consumer requests the next batch itself, at its own pace. If it falls behind - the log waits for it. The broker is not overwhelmed by slow consumers.

In practice: Kafka handles uneven load better. A spike of writes to the log does not disturb slow readers.

7. Operational complexity

	RabbitMQ	Kafka
Minimal production cluster	3 nodes	3 nodes (broker and controller roles can be combined)
Monitoring	Built-in UI + Prometheus	JMX + specialized tools
Learning curve for the team	Medium	High
Cross-datacenter replication	Relatively simple	Requires understanding MirrorMaker 2

The cross-datacenter replication row deserves a caveat. RabbitMQ's Federation and Shovel are indeed simpler than MirrorMaker 2, but they move only the messages themselves: consumer positions do not appear on the other side, and ordering is not guaranteed during the transfer. That is enough for "we switched to the standby cluster and moved on," but not for "continue exactly where we left off."

Kafka is a serious operational investment. Set it up correctly once and it runs for years. But if the team is small and there is no experienced administrator, RabbitMQ will be more predictable.

What only RabbitMQ can do

The seven criteria above compare the general models, but the choice is often decided by four capabilities that Kafka does not have at all. If you need at least one of them, the "log versus queue" debate can be skipped.

Priorities. A queue can be given x-max-priority, and a high-priority message jumps ahead of the accumulated low-priority ones. In Kafka priorities cannot exist: a partition is an ordered log read strictly in sequence. The workaround is a separate topic for the "urgent" ones plus separate consumers - so there are no priorities, just two pipes you divide your attention between yourself.

Delayed delivery. "Retry in thirty seconds," "remind in an hour" is done in RabbitMQ with a TTL queue plus a dead-letter exchange or the delayed-message plugin. In Kafka a message is available immediately and sits until its retention expires; it cannot be deferred - the consumer would have to pause reading the whole partition, and that would stall everything else.

A per-message lifetime. In RabbitMQ each message can carry its own TTL: "processing this request makes no sense later than a minute from now." In Kafka retention is set for the topic as a whole and means not "stale" but "delete from disk."

Routing of rejected messages. A RabbitMQ consumer answers for each message individually: acknowledge it, return it to the queue, or send it to a dead-letter exchange. In Kafka the consumer moves its offset, and "skip this one message and set it aside" is impossible at the broker level - a dead-letter queue is built by hand, with a separate topic and your own publishing code.

A fifth, less visible difference grows from the same root: in RabbitMQ work is considered taken in hand. A message is handed to a consumer, stays unacknowledged, and returns to someone else if that consumer crashes. In Kafka a crashed consumer leaves its offset in place, and after a rebalance the whole block is re-read - these are different models of responsibility, and for task-commands the first is usually more convenient.

What only Kafka can do

The other side is just as concrete.

Producer transactions and reading only committed data. Kafka can do what RabbitMQ has no equivalent of at all: several writes to different topics plus a consumer offset shift committed as a single transaction. The sender opens a transaction, writes, commits; a consumer with isolation.level=read_committed sees nothing from an uncommitted transaction. This is the foundation of "read, compute, write" processing without duplicates inside Kafka - what its documentation calls exactly-once. An important caveat: the guarantee works inside Kafka, and a write to an external database is not part of the transaction, so the receiver's idempotency does not go away.

Re-reading history. A consumer offset is just a number, and it can be moved back: a new service reads a month of events and builds its own state, and after a bug in a handler you can re-read yesterday. In RabbitMQ a read and acknowledged message is gone; to re-read it you need your own copy in a database. (RabbitMQ does have streams, where reading from the beginning is possible - a deliberate step toward the Kafka model.)

Log compaction. A topic with cleanup.policy=compact keeps the latest value for each key and discards the earlier ones. The result is a log that is at the same time a snapshot of the current state: a new consumer that reads it end to end gets the up-to-date picture. This is what tables in stream processing and the distribution of reference data across services rest on; RabbitMQ has no equivalent.

Order and parallelism together. The key determines the partition, and per-key order survives any number of consumers. In RabbitMQ the same is achieved either with a single active consumer and no parallelism, or with the hash-exchange plugin and one queue per key - more on that in the section below.

Many independent readers of one stream. Adding another consumer group in Kafka costs nothing: the data is not copied, and each group keeps its own offset. In RabbitMQ every new reader is a new queue where the exchange deposits a copy of the message - that is disk space and another queue to watch.

What it costs to operate

Operational complexity is one of the criteria above, and it deserves to be made concrete: what you will have to run, who is on call, and what is harder to fix at night.

What the installation consists of. A minimal fault-tolerant RabbitMQ is a three-node cluster with durable queues; the nodes need fast disks and a bit of memory, and the broker itself keeps a queue index and some messages in memory. A minimal Kafka is three brokers (in recent versions without ZooKeeper - the brokers themselves take the controller role), and the main resource here is disk: it must fit the entire retention period multiplied by the number of replicas. A rough estimate: one hundred gigabytes of events per day with week-long retention and three replicas requires about two terabytes per cluster - and that is before any headroom for spikes.

Who is on call. RabbitMQ can be maintained by whoever maintains the application: few settings, a dozen metrics, and troubleshooting almost always comes down to "the queue is growing - the consumer cannot keep up." Kafka demands someone who understands partitions, offsets, rebalances and retention; not necessarily a dedicated role, but definitely someone who has read more about it than the quick-start guide. The difference is especially visible in a team of four developers with no dedicated ops.

What breaks and how it is treated. Typical RabbitMQ incidents are a queue that grew and ate the node's memory (treated with length limits and backpressure) and a mismatch of queue properties during a rollout (treated by deleting and recreating the queue, that is, a window of consumer downtime). Typical Kafka incidents are consumer lag you cannot close before retention expires (the data leaves the disk), an endless group rebalance caused by slow processing, and disks running out of space, after which the broker halts. At night the former are fixed faster: "restart the consumer" versus "figure out why the group cannot agree."

What a migration costs. Switching brokers is not swapping a library: the guarantees, the ordering, the retry mechanics and the responsibility model for what has been read all change. That is why the cost of a wrong choice is measured not in money for nodes but in rewritten handlers - and why you should choose by data model, not by popularity.

The practical takeaway for a small team: if the tasks are addressed and few, three RabbitMQ nodes are cheaper to maintain. As soon as several independent readers of one stream, re-reading of history or tens of thousands of events per second appear, Kafka pays for its complexity.

When to use both together

This is not "let's take both just in case" but a specific architecture.

A service receives commands via RabbitMQ (process an image, send an email). On completion it publishes the fact to Kafka (image processed, email sent). Analytics, monitoring and ML services read Kafka and know nothing about the command queue.

Commands are atomic and disappear after processing. Events accumulate as history.

Two caveats about this scheme that people usually learn later.

A bridge between the brokers, if you really need one. Sometimes data does not need to be produced in both but poured from one into the other: hand Kafka events to an external contractor who only speaks AMQP, or collect into Kafka a stream from an old RabbitMQ system. There are two ready-made ways. Kafka Connect with a RabbitMQ connector (source or sink) is a separate process that reads from one and writes to the other, with its own offset management. On the RabbitMQ side the built-in Shovel and Federation play a similar role, but they move messages between AMQP brokers, not into Kafka. A hand-rolled copying service of ten lines looks simpler, and that is a trap: it must handle retries, idempotency, observability and backpressure - everything already written in the ready-made connectors.

Two models - two disciplines. Running both brokers means two sets of metrics and dashboards, two ways of troubleshooting, two ways of retrying and two different developer habits. In a small team this usually costs more than the benefit of each tool matching its task exactly. Introduce the second broker when the first one demonstrably fails at a specific job, not up front "for architectural cleanliness" - and at that moment it helps to write down in the project rules what travels through which broker, otherwise in a year the same traffic will ride through both.

Common mistakes when choosing

"We'll take RabbitMQ and move to Kafka later" - if the data is needed as an event log, start with Kafka right away. The switch means rewriting logic, not swapping a client.

"Kafka, because big companies use it" - big companies process tens of millions of events per second. At a few thousand events a day, RabbitMQ is simpler and cheaper to maintain.

"We'll build a task queue on top of Kafka" - technically possible, but Kafka is not optimized for short-lived tasks. Distributing work tasks is RabbitMQ's job.

"We'll store user requests in Kafka like in a database" - Kafka is not a database. If you need arbitrary search over the content, that means an event store plus PostgreSQL or similar, not raw Kafka.

Deeper: ordering in RabbitMQ: single active consumer, consistent-hash and super streams

"RabbitMQ has no ordering out of the box" is true for a queue with several consumers: they pick it apart in a jumble, and events of one order are processed in parallel. In practice ordering is covered by three stock tools.

Single active consumer. The queue argument x-single-active-consumer: true lets a single consumer work while the rest stay connected and wait; when the active one dies, the next takes over from the same spot. Order is preserved, failover exists, parallelism does not: one queue, one thread. Good for queues with a modest flow where order matters more than speed.

Consistent-hash exchange. The rabbitmq_consistent_hash_exchange plugin provides an exchange that picks a queue by the hash of the routing key: all messages with order-42 land in one of N queues, and if each of those N queues has a single active consumer, you get exactly the Kafka model: N partitions, order within a key, parallelism across keys. The number of queues is chosen up front, like the number of partitions, and changed with a re-shard; a queue's weight is set by its binding key.

Super streams. RabbitMQ streams (the stream queue type, a log with offsets) have partitioning out of the box: a super stream is a set of stream-partitions with key-based routing, consumers each read their own partition with a single active consumer per partition, and a reconnect hands the partition to someone else. This is the answer to "we need Kafka partitions but we have RabbitMQ": the same ordering, the same replay by offset, its own protocol and client (rabbitmq-stream-client), without AMQP.

The rule of thumb: order is rarely needed and the flows are small - single active consumer; both order and per-key parallelism over regular queues are needed - consistent-hash with one queue per partition; a re-readable log is needed - super streams or already Kafka, as in the table above.

Deeper: no broker at all: a job queue in PostgreSQL

The "RabbitMQ or Kafka" fork skips a third answer that is right for many services: neither. If the tasks are background work, the flow is tens per second, and the consumer is the same service, the queue is made a table in the database you already have.

A table jobs(id, type, payload, run_at, attempts, status), a handler that loops over batches: SELECT ... WHERE status = 'NEW' AND run_at <= now() ORDER BY run_at FOR UPDATE SKIP LOCKED LIMIT 10, processes them and marks them done in the same transaction. SKIP LOCKED skips rows held by a neighboring instance, so there can be many consumers without duplicates; run_at gives delayed delivery for free; attempts with a growing run_at is retries with a pause; finished rows are cleaned up by age or moved to an archive table. The locking mechanics are covered in the section on PostgreSQL locks, and FOR UPDATE modes in the Java stack in the article on lock modes.

What you gain over a broker: the task is enqueued in the same transaction as the business change, and the outbox problem disappears because the queue is the table; observability with a plain SELECT; not a single new system to operate. What you lose: at thousands of tasks per second the table becomes hot, with bloat from updates and index contention; polling with a pause adds up to a second of latency unless you wake it up with LISTEN/NOTIFY; consumers from other services would have to reach into someone else's database, which must not happen - and then the queue turns into an integration point, which is exactly what a broker is for.

The boundary runs roughly like this: tasks of a single service, up to hundreds per second, with a need for deferred start and transactionality - a table; events for other services, a flow of thousands, re-reading of history - a broker. The former is often how systems start, and a later move to a broker is just a sender rollout, provided the handlers were idempotent from day one.

In short

• NATS = a subject-based bus. Light, fast, subject-based addressing; core NATS stores nothing. RabbitMQ = a task queue. A message disappears after processing; the broker pushes to the consumer. Kafka = an event log. Messages are stored and can be read again; the consumer pulls at its own pace.
• Choose NATS for a light internal bus, the edge and request-reply. Choose RabbitMQ for distributing tasks among workers, asynchronous RPC and flexible routing. Choose Kafka for event history retention, high load and streaming analytics.
• Ordering by key is built into Kafka; in RabbitMQ it takes extra effort. Ordering in RabbitMQ is covered by single active consumer (one thread), consistent-hash exchange with one queue per partition (per-key order and parallelism) and super streams (a partitioned log).
• Kafka is operationally heavier - justified when you truly need its capabilities. Operations differ more than the protocols do: Kafka needs disk for the entire retention and a person who understands partitions and rebalances, and at night "restart the consumer" is cheaper than figuring out why a group failed to agree.
• Both together is not "just in case" but a specific pattern: RabbitMQ for commands, Kafka for events.
• The third answer to the fork: a job queue as a table with FOR UPDATE SKIP LOCKED for background tasks of a single service up to hundreds per second; a broker is for events between services and re-reading history.
• Only RabbitMQ has: priorities, delayed delivery, a per-message TTL and routing of rejected messages; in Kafka a dead-letter queue is built by hand with a separate topic.
• Only Kafka has: producer transactions with read_committed, re-reading history by offset, log compaction and per-key order combined with parallelism.
• Moving data between brokers is better done with a ready-made connector than a hand-rolled ten-line service, and remember that two brokers mean two sets of metrics and two troubleshooting disciplines.

What to read next

• NATS - subject-based addressing, queue groups, request-reply and JetStream.
• The AMQP protocol - how the RabbitMQ delivery model works.
• Messaging patterns: Java, Go, Node, Python - work queue, pub/sub, RPC, idempotency.
• Distributed patterns - Saga, Outbox, Idempotent Consumer (applicable to any of these brokers): Java, Go, Node, Python.