RabbitMQ in production: clustering, queue types and monitoring

← Back to the section

A single RabbitMQ node works fine on a laptop. In production things get harder: you need to survive a server restart, avoid losing messages, and monitor the broker's health. Let's look at how RabbitMQ is set up in a real environment.

Why you need a cluster

Imagine your server running RabbitMQ reboots. Every message waiting to be processed is gone. Every application that was writing to a queue gets a connection error. Until the server comes back up, the system is down.

A cluster solves this: several RabbitMQ nodes work together and know about each other. If one node goes down, the rest keep accepting and delivering messages.

How a cluster works. All cluster nodes store the same metadata: definitions of exchanges, queues, bindings, and users. This metadata is synchronized automatically — through Raft (since version 3.10) or Mnesia (in older versions).

A client can connect to any node in the cluster. If the queue it needs lives on another node, the current node forwards the request itself.

A few important consequences:

• At least 3 nodes for reliable operation. With two nodes, losing one puts the other into read-only mode and the cluster stops accepting new messages.
• A network split between nodes (split brain) is a serious problem. RabbitMQ supports several behavior strategies (pause-minority, autoheal, ignore), but none of them comes without consequences. Choosing the right queue type reduces the risk.
• The queue itself physically lives on one node (Classic) or is replicated across several (Quorum, Streams). This is the main architectural decision.

Three queue types

Classic Queue

The oldest type, present since the first versions of RabbitMQ. By default the queue is not replicated — it lives on one node. If that node goes down, the queue is unavailable until it comes back.

Classic Queues used to be "mirrorable" to other nodes through the ha-mode policy. This mechanism was deprecated in version 3.8 and completely removed in 4.0. It cannot be used in new code.

When a Classic Queue is appropriate today:

• Temporary queues: RPC replies, notifications without guarantees, cache invalidation.
• Situations where losing a message is acceptable: metrics, non-critical events.
• Not suitable for: orders, payments, any business events.

Quorum Queue

Introduced in version 3.8, became the standard in 3.10. This is a queue that is replicated across several nodes through the Raft algorithm: one node is the leader, the rest are followers. A message is considered stored only once a majority of replicas have received it.

channel.queue_declare(
    queue="orders",
    durable=True,
    arguments={"x-queue-type": "quorum"}
)

Properties:

• High availability. In a typical configuration (3 nodes), losing one has no effect on operation. Losing two makes the queue unavailable until a node returns.
• Durable storage. Every message is replicated to a majority of nodes before the sender is acknowledged.
• More expensive than Classic. More I/O operations, more memory. For small volumes it is overkill; for critical data it is a must.
• Limitations. No priority queues, no exclusive queues, no per-message TTL.

Quorum Queue is the default choice for anything that must not be lost.

Streams

Introduced in version 3.9. This is a message log — an append-only log, replicated through Raft, with the ability to read from any offset.

channel.queue_declare(
    queue="events-log",
    durable=True,
    arguments={
        "x-queue-type": "stream",
        "x-max-length-bytes": 10_000_000_000  # 10 GB of storage
    }
)

A good fit when:

• You need to replay history: a new service wants to read events from the past week.
• The same messages are read by several independent consumers without copying.
• You need high throughput (millions of messages per second).

Not suitable for: routing by headers and patterns, RPC scenarios, dead letter routing.

If the task is purely log-based and you need Kafka-level performance, Kafka is probably the better choice. Streams are useful when you already have RabbitMQ infrastructure and don't want to add yet another system.

Message storage and memory

A persistent message in a Classic or Quorum Queue is written to disk. But at the same time it is also kept in memory — for fast delivery to the consumer.

Classic Queues had a special lazy queue mode: messages are written to disk right away and don't take up memory. It was used when a queue could accumulate millions of messages. With the arrival of Quorum Queue this mode is no longer needed — there the behavior is controlled through the x-max-in-memory-length parameter.

For new projects: Quorum Queue with x-max-in-memory-length tuned for the load.

Cross-region replication

RabbitMQ clustering is not meant for nodes in different regions. Raft requires low latency within the cluster. Connecting across regions with latency above 10 ms leads to instability and cluster splits.

For geographically distributed systems, use separate clusters in each region and one of two tools for moving messages between them:

Federation — an exchange-to-exchange or queue-to-queue link between clusters. Messages are published in one cluster and asynchronously copied to another. The clusters are independent of each other.

Shovel — a simple "pump": it reads from a source queue and publishes to a target exchange. Handy for one-off transfers and point-to-point scenarios.

If the primary cluster fails completely: applications switch to the backup, and at most the replication lag is lost (usually a few seconds).

Backpressure and flow control

When a queue grows faster than it is processed, RabbitMQ engages backpressure:

1. Blocking the sender — the broker pauses publishing if a memory or disk limit is reached (the vm_memory_high_watermark, disk_free_limit parameters).
2. A credit system between cluster nodes — a delay inside the cluster.
3. Prefetch on the consumer side — limiting the number of unacknowledged messages.

When blocking kicks in, publishing is throttled synchronously. A service that used to "fire and forget" starts getting delays or timeouts. This is correct broker behavior, but the application must be able to handle it: pass the pressure further up the chain, throttle the incoming flow, use a fallback path.

Monitoring

RabbitMQ exports metrics in Prometheus format. The key indicators:

The main thing is to watch the ratio between messages_ready and consumer throughput. This is the equivalent of Kafka consumer lag: if the queue only grows, the system isn't keeping up.

Performance ballpark

Rough numbers for a single modern server (8 cores, 32 GB of memory, NVMe):

• Quorum Queue with persistent storage: 30,000–50,000 messages per second.
• Classic Queue without persistent storage: 100,000+ messages per second.
• Optimal message size: 1–10 KB. Messages larger than 100 KB noticeably reduce performance; anything over 1 MB is better stored in object storage, passing only a reference.
• Number of queues: thousands is normal, tens of thousands is a load on metadata, hundreds of thousands is risky.

A baseline reliability strategy

For a critical service on RabbitMQ:

1. A cluster of three or more nodes in one region, Quorum Queue for business messages.
2. Federation or Shovel to a backup cluster in another region for important queues.
3. A configuration backup via rabbitmqctl export_definitions, kept in version control.
4. A tested recovery procedure — not "we'll figure it out when it goes down".

In short

• A RabbitMQ cluster = several nodes with shared metadata. At least 3 nodes for reliable operation.
• Classic Queue — no replication, for temporary and non-critical messages. Mirroring was removed in version 4.0.
• Quorum Queue — the standard for business data; replication through Raft, a message is acknowledged only once a majority of nodes have received it.
• Streams — a log with replay capability; suitable for high throughput and several independent readers.
• Clustering doesn't work across regions with high latency — Federation and Shovel exist for that.
• On overflow the broker blocks publishing — the application must be able to handle it.
• The key metric is messages_ready growing without dropping: the queue is piling up.

What to read next

• The AMQP protocol — the delivery model, exchanges, queues, bindings.
• Spring AMQP — how to write client code in Java.
• Patterns over AMQP — work queue, pub/sub, RPC.
• AMQP versus Kafka — when to choose which.