Create RabbitMQ cluster to replicate configuration across multiple nodes.
Preliminary information
I will create three node setup using nodes: rabbit (192.168.50.201), reindeer (192.168.50.202) and raccoon (192.168.50.203).
RabbitMQ uses a cookie to determine whether nodes can talk to each other. I will use string JFKZVCBYEISEQILVZMSD. It is an alphanumeric string up to 255 characters. Keep it secret.
Install RabbitMQ server on every node
Install RabbitMQ message broker on every node.
$ sudo apt update $ sudo apt install gnupg2 apt-transport-https curl $ curl https://dl.bintray.com/rabbitmq/Keys/rabbitmq-release-signing-key.asc | sudo apt-key add - $ echo "deb https://dl.bintray.com/rabbitmq/debian stretch main" | sudo tee /etc/apt/sources.list.d/rabbitmq.list $ echo "deb https://dl.bintray.com/rabbitmq-erlang/debian buster erlang-22.x" | sudo tee -a /etc/apt/sources.list.d/rabbitmq.list $ apt update $ apt install rabbitmq-server
Use rabbitmq-diagnostics to display RabbitMQ version.
$ sudo rabbitmq-diagnostics server_version Asking node rabbit@buster for its RabbitMQ version... 3.7.18
Setup first RabbitMQ node
Setup first RabbitMQ node – rabbit.
Define Erlang cookie.
$ sudo systemctl stop rabbitmq-server $ echo "JFKZVCBYEISEQILVZMSD" | sudo tee /var/lib/rabbitmq/.erlang.cookie $ sudo systemctl start rabbitmq-server
Enable management plugin.
$ rabbitmq-plugins enable rabbitmq_management
Define admin user with password password.
.
$ sudo rabbitmqctl add_user admin password $ sudo rabbitmqctl set_user_tags admin administrator $ sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
Delete guest user.
$ sudo rabbitmqctl delete_user guest
Setup additional RabbitMQ nodes
Setup additional RabbitMQ nodes – reindeer and raccoon.
Define Erlang cookie.
$ sudo systemctl stop rabbitmq-server $ echo "JFKZVCBYEISEQILVZMSD" | sudo tee /var/lib/rabbitmq/.erlang.cookie $ sudo systemctl start rabbitmq-server
Enable management plugin.
$ rabbitmq-plugins enable rabbitmq_management
Join cluster by specifying existing member.
$ sudo rabbitmqctl stop_app $ sudo rabbitmqctl join_cluster rabbit@rabbit $ sudo rabbitmqctl start_app
Verify cluster status
Verify cluster status.
$ sudo rabbitmqctl cluster_status Cluster status of node rabbit@rabbit ... Basics Cluster name: rabbit@rabbit Disk Nodes rabbit@rabbit rabbit@raccoon rabbit@reindeer Running Nodes rabbit@rabbit rabbit@raccoon rabbit@reindeer Versions rabbit@rabbit: RabbitMQ 3.8.0 on Erlang 22.1.1 rabbit@raccoon: RabbitMQ 3.8.0 on Erlang 22.1.1 rabbit@reindeer: RabbitMQ 3.8.0 on Erlang 22.1.1 Alarms (none) Network Partitions (none) Listeners Node: rabbit@rabbit, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@rabbit, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Node: rabbit@rabbit, interface: [::], port: 15672, protocol: http, purpose: HTTP API Node: rabbit@raccoon, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@raccoon, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Node: rabbit@raccoon, interface: [::], port: 15672, protocol: http, purpose: HTTP API Node: rabbit@reindeer, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Node: rabbit@reindeer, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 Feature flags Flag: drop_unroutable_metric, state: enabled Flag: empty_basic_get_metric, state: enabled Flag: implicit_default_bindings, state: enabled Flag: quorum_queue, state: enabled Flag: virtual_host_metadata, state: enabled
Alternatively, list running nodes.
$ sudo rabbitmqctl cluster_status --formatter=json | jq -r .running_nodes[] rabbit@raccoon rabbit@rabbit rabbit@reindeer
Example
This is an example to illustrate how RabbitMQ cluster behaves in relation to declared queues in case of node downtime.
Get rabbitmqadmin utility.
$ curl -o rabbitmqadmin http://127.0.0.1:15672/cli/rabbitmqadmin
Ensure that executable bit is set.
$ chmod +x rabbitmqadmin
Declare cluster_example vhost.
$ ./rabbitmqadmin --username admin --password password declare vhost name=cluster_example
Declare cluster_example user with password password, grant permissions to cluster_example vhost and set management.
$ sudo rabbitmqctl add_user cluster_example password $ sudo rabbitmqctl set_permissions cluster_example --vhost cluster_example ".*" ".*" ".*" $ sudo rabbitmqctl set_user_tags cluster_example management
Declare sample exchange.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare exchange name=messages type=fanout
Declare durable queue.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare queue name=durable_messages durable=true
Declare not durable queue.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare queue name=not_durable_messages durable=false
Declare bindings.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare binding source=messages destination=durable_messages
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example declare binding source=messages destination=not_durable_messages
Install jq utility.
$ sudo apt install jq
Export configuration.
$ ./rabbitmqadmin --username admin --password password export config.json
$ cat config.json | jq .
{
"rabbit_version": "3.8.0",
"users": [
{
"name": "cluster_example",
"password_hash": "cQb0oesCsqP6KJOjMaM36xWE9rcWef5r33peMF/5tYGGTPlR",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "management"
},
{
"name": "admin",
"password_hash": "ZtNWgL2TdCqLMJx3jOpJVsHDsw/yiIPHF/srfFcqNkIpYzdr",
"hashing_algorithm": "rabbit_password_hashing_sha256",
"tags": "administrator"
}
],
"vhosts": [
{
"name": "cluster_example"
},
{
"name": "/"
}
],
"permissions": [
{
"user": "cluster_example",
"vhost": "cluster_example",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "admin",
"vhost": "/",
"configure": ".*",
"write": ".*",
"read": ".*"
},
{
"user": "admin",
"vhost": "cluster_example",
"configure": ".*",
"write": ".*",
"read": ".*"
}
],
"topic_permissions": [],
"parameters": [],
"global_parameters": [
{
"name": "cluster_name",
"value": "rabbit@rabbit"
}
],
"policies": [],
"queues": [
{
"name": "durable_messages",
"vhost": "cluster_example",
"durable": true,
"auto_delete": false,
"arguments": {}
},
{
"name": "not_durable_messages",
"vhost": "cluster_example",
"durable": false,
"auto_delete": false,
"arguments": {}
}
],
"exchanges": [
{
"name": "messages",
"vhost": "cluster_example",
"type": "fanout",
"durable": true,
"auto_delete": false,
"internal": false,
"arguments": {}
}
],
"bindings": [
{
"source": "messages",
"vhost": "cluster_example",
"destination": "durable_messages",
"destination_type": "queue",
"routing_key": "",
"arguments": {}
},
{
"source": "messages",
"vhost": "cluster_example",
"destination": "not_durable_messages",
"destination_type": "queue",
"routing_key": "",
"arguments": {}
}
]
}
Publish sample messages.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example publish exchange=messages routing_key= payload="sample message a"
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example publish exchange=messages routing_key= payload="sample message b"
Inspect queues, notice that each queue is created on a single node.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages +-----------------+----------------------+---------------+----------+ | vhost | name | node | messages | +-----------------+----------------------+---------------+----------+ | cluster_example | durable_messages | rabbit@rabbit | 2 | | cluster_example | not_durable_messages | rabbit@rabbit | 2 | +-----------------+----------------------+---------------+----------+
Get sample message from queue, but do not requeue it.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example get queue=durable_messages ackmode=ack_requeue_false +-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+ | routing_key | exchange | message_count | payload | payload_bytes | payload_encoding | properties | redelivered | +-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+ | | messages | 1 | sample message a | 16 | string | | False | +-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+
Get sample message from queue and requeue it.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example get queue=durable_messages +-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+ | routing_key | exchange | message_count | payload | payload_bytes | payload_encoding | properties | redelivered | +-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+ | | messages | 1 | sample message b | 16 | string | | True | +-------------+----------+---------------+------------------+---------------+------------------+------------+-------------+
Inspect queues.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages +-----------------+----------------------+---------------+----------+ | vhost | name | node | messages | +-----------------+----------------------+---------------+----------+ | cluster_example | durable_messages | rabbit@rabbit | 1 | | cluster_example | not_durable_messages | rabbit@rabbit | 2 | +-----------------+----------------------+---------------+----------+
Stop the RabbitMQ node on which the above-mentioned queues reside.
$ sudo systemctl stop rabbitmq-server
Inspect queues.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages +-----------------+------------------+---------------+----------+ | vhost | name | node | messages | +-----------------+------------------+---------------+----------+ | cluster_example | durable_messages | rabbit@rabbit | | +-----------------+------------------+---------------+----------+
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example get queue=durable_messages *** Not found: /api/queues/cluster_example/durable_messages/get
Start the RabbitMQ node on which the above-mentioned queues reside.
$ sudo systemctl start rabbitmq-server
Inspect queues.
$ ./rabbitmqadmin --username cluster_example --password password --vhost cluster_example list queues vhost name node messages +-----------------+------------------+---------------+----------+ | vhost | name | node | messages | +-----------------+------------------+---------------+----------+ | cluster_example | durable_messages | rabbit@rabbit | 0 | +-----------------+------------------+---------------+----------+
To sum up, use queue mirroring to ensure that queue contents are also protected against single node failures.