Skip to main content

RabbitMQ

RabbitMQ is an open-source message broker implementing AMQP. In NDIP it carries:

  • Galaxy ↔ Pulsar job messages (setup, status and kill messages per Pulsar manager);
  • Galaxy's Celery tasks and internal control.* messages between Galaxy processes;
  • Remote Data Broker requests (queues such as galaxy/galaxy-test, sns-ndip).

Deployment​

RabbitMQ is a Bitnami subchart of the Galaxy Helm chart (StatefulSet galaxy-rabbitmq in namespace ndip), so the Galaxy jobs (galaxy-test, galaxy-k8s-prod) deploy it. Settings are under rabbitmq: in calvera's k8s/galaxy/values-<env>.yml. The password and Erlang cookie are in secrets-<env>.yml.

Prod (speedy)Test (mickey)
Brokers3, hard anti-affinity, pause_minority, PDB maxUnavailable: 11
Default queue typequorum (survives losing one broker)classic
Galaxy connects togalaxy-rabbitmq:5672 (plain, in-cluster)galaxy-rabbitmq:5672
Pulsars/RDBs connect toamqps://ndip.ornl.gov:5672 → VM proxy → NodePort 30671 (TLS, cert galaxy-tls)10.64.197.29:30672 (plain AMQP NodePort on a single mickey node)
StorageLonghorn PVC per brokershared-nvme

User ndip, vhost /.

note

Test Pulsars point at one mickey node, so maintenance on that node stops test jobs. A request for an F5 TCP virtual server for AMQP (test, and later prod) has been drafted. With the F5, the idle timeout must be at least 3600 s, because Pulsar uses a 580 s AMQP heartbeat.

Useful commands​

kubectl -n ndip get pods -l app.kubernetes.io/name=rabbitmq -o wide
kubectl -n ndip exec galaxy-rabbitmq-0 -- rabbitmqctl cluster_status
kubectl -n ndip exec galaxy-rabbitmq-0 -- rabbitmqctl list_queues name messages consumers
kubectl -n ndip exec galaxy-rabbitmq-0 -- rabbitmqctl list_policies -p /
kubectl -n ndip logs galaxy-rabbitmq-0

On prod, cross-node rabbitmqctl commands need the extra NetworkPolicy ports (35672–35682) that values-prod.yml opens. Without them they time out.

Orphaned control queues​

Every Galaxy process declares durable control.<server>@<pod> queues. Each rollout creates new pod names, and the old queues stay behind with messages piling up, which grows memory use on the brokers. On the clusters, a RabbitMQ policy named control-expiry deletes unused control queues after 1 hour. It is not part of the chart, so check it after rebuilding RabbitMQ or resetting the vhost:

kubectl -n ndip exec galaxy-rabbitmq-0 -- rabbitmqctl set_policy -p / --apply-to queues \
  control-expiry '^control\.' '{"expires":3600000}'

The policy is applied by hand (not yet in the chart), on both clusters:

kubectl -n ndip exec galaxy-rabbitmq-0 -c rabbitmq -- rabbitmqctl set_policy control-expiry '^control\.' '{"expires":3600000}' --apply-to queues

expires only applies to queues with no consumers, so the live Galaxy pod's control queues are never removed.

To clean up by hand, list queues named control.* with no consumers and delete them (rabbitmqctl delete_queue <name>), or wait for the policy.

Resetting the vhost​

This drops every queued message, including pending Pulsar jobs. Use it only when stale messages must not be delivered, for example after restoring a database copy whose jobs were already dispatched elsewhere. Scale Galaxy to 0 first:

kubectl -n ndip scale deploy/galaxy --replicas=0
kubectl -n ndip exec galaxy-rabbitmq-0 -- rabbitmqctl delete_vhost /
kubectl -n ndip exec galaxy-rabbitmq-0 -- rabbitmqctl add_vhost /
kubectl -n ndip exec galaxy-rabbitmq-0 -- rabbitmqctl set_permissions -p / ndip '.*' '.*' '.*'
# re-apply the control-expiry policy, then scale Galaxy back to 1

For restart loops and other failures, see the troubleshooting guide.

Legacy: VM deployment​

On the VM stack, RabbitMQ ran on the Galaxy VM (Ansible role ansible-role-rabbitmq, part of the ndip job), with the SSL certificate from the host vars. On the old prod VM it is now stopped and disabled, because port 5672 there belongs to the VM proxy. galaxy.ornl.gov still uses its Ansible-managed RabbitMQ.