Pulsar

Pulsar is a service developed by the Galaxy team for running Galaxy jobs on remote servers. We maintain a fork with changes necessary for NDIP. Pulsar talks to Galaxy through RabbitMQ, reads and writes data through Rucio and the Remote Data Broker, and runs each job in a Docker container.

To add an instance, see Adding a Pulsar instance.

Deployment

Pulsar did not move to Kubernetes. The compute hosts are still deployed with Ansible from calvera.

note

For information about worker nodes infrastructure please refer to the infrastructure section

Pulsar instances

Host	Prod job (main only)	Test / dev jobs	Galaxy destination
ndip-node01.sns.gov	pulsars-sns-prod	pulsars-sns-test, pulsars-sns-dev	pulsar_sns_node01
sns-ndip-gpu.sns.gov	pulsars-prod	pulsars-test, pulsars-dev	pulsar_sns_gpu_node01
imaginex-1.ornl.gov, imaginex-2.ornl.gov	pulsars-imaginex-prod	pulsars-imaginex-test	pulsar_imaginex1/2
ORC pulsar-cpu, pulsar-large-cpu (galaxy.ornl.gov)	pulsars-common-galaxy		n/a (galaxy.ornl.gov)
OLCF Frontier (Slate/Marble)	Helm chart, manually		pulsar_marble_*
NERSC Perlmutter (Spin)	Helm chart, manually		pulsar_spin_*

One machine, several instances. Each SNS/imaginex host appears in ansible/inventories/pulsar-hosts under several aliases (for example pulsar-sns-node01, -test and -dev are all ndip-node01.sns.gov). The groups pulsar_prod, pulsar_test and pulsar_dev give each alias its own systemd unit (pulsar, pulsar_test, pulsar_dev), its own directories (/opt/pulsar, /opt/pulsar_test, ...) and its own endpoints. A "test" run still runs host-level roles (Docker, packages) on a production machine.

	prod instances	test instances
Galaxy	https://ndip.ornl.gov (speedy)	https://ndip-test.ornl.gov (mickey)
AMQP (rabbitmq_url)	ndip.ornl.gov:5672 (TLS, through the VM proxy)	10.64.197.29:30672 (mickey node)
Rucio (rucio_url)	https://ndip.ornl.gov/rucio	https://ndip-test.ornl.gov/rucio
Scope / RSE suffix	galaxy / none	galaxy_test / _TEST
Logs	/var/log/pulsar.log	/var/log/pulsar_test.log

Deploy Pulsar from main

Prod Pulsar jobs only run from main, and main carries the current prod settings (Rucio on speedy). Don't deploy Pulsars from k8s-env-split: its ansible/ tree is older (it still points prod at the Rucio VM and would downgrade Pulsar).

note

All deployed Pulsar instances can be accessed via ssh (user cloud, or xv6 on ndip-node01). The SSH key is the PULSAR_SSH_KEY CI/CD variable. The xv6 password for ndip-node01 is XV6_PASS.

Deploying

Restarts kill running jobs

Ansible restarts the Pulsar systemd unit directly. Pulsar does not exit while its job containers are alive, so systemd waits out TimeoutStopSec (90s) and then SIGKILLs Pulsar and every running job container. In Galaxy these jobs look like they died for no reason. On 2026-09-04 a deploy on ndip-node01 killed about 28 jobs this way.

Before deploying to a host:

1. Check what is running: docker ps on the host, or the jobs admin page filtered by destination.
2. Stop new jobs reaching the host. There is no per-host drain switch. Either lock job dispatch in Galaxy (admin → Jobs → Job Lock, which affects every destination), or rely on the router skipping any host named (label host) in a firing env: prod alert (see job routing).
3. Wait for running jobs to finish, then run the job. Prefer quiet hours.
4. Check systemctl status pulsar (or pulsar_test), the log, and a test job.

Configuration

Configuration is in Ansible variable files: group_vars/pulsar_all_servers/pulsarservers.yml (shared, including RDB and Rucio client settings), group_vars/pulsar_prod, pulsar_test, pulsar_dev, and pulsar_common_galaxy for galaxy.ornl.gov. The role is ornl.tuning.pulsar (templates for the RDB and OIDC setup, tasks to run and monitor Pulsar).

Log shipping from the Pulsar hosts is off (fluentd_install: false). Filebeat for some hosts is deployed by the filebeat pulsar job.

Manual deployment (Slate, Spin)

Both Slate and Spin deployments are performed manually with Helm, because of two-factor authentication:

• Slate (NCCS - Frontier)
• Spin (NERSC - Perlmutter)