Rucio

Rucio is a service developed by CERN that enables centralized management of large volumes of data backed by many heterogeneous storage backends. NDIP uses it so that jobs can access data in different places transparently for the user. Galaxy stores every dataset in Rucio through our Rucio object store plugin.

Deployment

Instance	Where	URL	CI job
NDIP prod	speedy, Deployment rucio-server, DB rucio-pg	https://ndip.ornl.gov/rucio	rucio-k8s-prod
NDIP test	mickey, Deployment rucio-server, DB rucio-pg	https://ndip-test.ornl.gov/rucio	rucio-test
galaxy.ornl.gov	Rucio VM 10.64.199.233 (Docker Compose + PostgreSQL)	http://10.64.199.233	rucio-prod (Ansible)

Only the Rucio server (API) and its database are deployed on the clusters. There are no Rucio daemons: no reaper and no abacus.

The Kubernetes job (k8s/rucio/deploy.sh) runs these steps:

1. helm upgrade --install rucio-server (chart ndip/rucio-server from charts/rucio, values values-server-<env>.yml, image savannah.ornl.gov/ndip/rucio-protocols/rucio);
2. applies the /rucio Ingress (rucio-ingress.yaml);
3. runs the rucio-db-init Job. It fails harmlessly if the database is already initialized;
4. runs prepare_rses.py with rses-<env>.yml inside the server pod. This adds missing scopes and RSEs, reconciles RSE parameters and reapplies protocols. It exits before changing anything if an RSE's deterministic flag doesn't match. That stop is deliberate; see below.

The database connection string (config.database.default) and the root (bootstrap) credentials are in k8s/rucio/secrets-<env>.yml.

Prod Pulsars use rucio_url: https://ndip.ornl.gov/rucio (ansible/group_vars/pulsar_prod on main). Test Pulsars use https://ndip-test.ornl.gov/rucio.

Password Rotation

If the Rucio root account password is leaked, then there is a script (k8s/rucio/rotate_password.sh) and corresponding CI job that can rotate the password in Kubernetes. You will need to replace the current password in all Ansible vault secrets and add a CI variable named RUCIO_OLD_BOOTSTRAP_PASSWORD with the current password before running the CI job. For legacy VMs, you can run the accompanying Python script (k8s/rucio/rotate_bootstrap_password.py) directly on the VM.

Once the password is rotated, then you will need to redeploy Galaxy, all Pulsar servers (for NDIP, do not forget the IMAGINE-X Pulsar nodes), and the Rucio server. Once done, you should immediately run system and tool tests to confirm that the rotated password is properly applied to all services.

Names: RSEs and scope

	Prod	Test
Scope	galaxy	galaxy_test
RSE names	<FACILITY>	<FACILITY>_TEST
Galaxy's upload RSE	GALAXY (files on the galaxy-data PVC; old CEPH data copied in)	GALAXY_TEST

Prod Pulsar upload targets: the SNS hosts write to SNS_GPFS, the imaginex hosts to CEPH (CEPH is still live for them). Neither RSE nor scope is stored per dataset in the Galaxy database. Galaxy reads both from object_store_conf.xml (k8s/galaxy/values-objectstore-<env>.yml), and the DID name is sha256(rel_path).

The determinism rule

An RSE is either deterministic (Rucio computes the path from scope and name) or not (the path is given at registration). Galaxy/Pulsar uploads need deterministic RSEs (GALAXY*, SNS_GPFS*). Pulsar register_only ingest and scripts that call add_replicas with an explicit PFN need non-deterministic ones (the facility RSEs). Never flip the flag to make a migration work: Rucio refuses to change it on an RSE that holds replicas, and flipping it breaks the other write path. rses-<env>.yml states deterministic: explicitly for every RSE.

Other things to know:

• Protocol priority 0 means "first choice", not "disabled". Rucio sorts priorities ascending and only null excludes a protocol. To deprioritise a protocol, give it a high number.
• No reaper. Deleting replicas in the catalog never deletes bytes. Old files must be removed by hand.
• No abacus. RSE usage counters always report 0. Verify migrations with the catalog and the filesystem, never with usage counters.
• Most RSEs set skip_upload_stat: true, so a replica can be registered before its bytes exist. A replica without bytes is an expected failure mode, not catalog corruption.
• The scope is part of a deterministic path (<prefix>/<scope>/<h1>/<h2>/<name>), so renaming a scope relocates every file.

Tools

Reusable scripts in calvera k8s/rucio/: audit_rses.py, verify_rucio.py, prepare_rses.py (run by deploy), and register_galaxy_replicas.py (registered the copied CEPH files on GALAXY during the prod migration).

To use the Rucio CLI against a cluster, exec into the server pod, which has the client configured for localhost:

kubectl -n ndip exec -it deploy/rucio-server -c rucio-server -- bash
rucio whoami          # may need the [client] section that deploy.sh appends to rucio.cfg
rucio-admin rse list

Health check: curl https://ndip.ornl.gov/rucio/ping. Prometheus probes this URL.

Legacy: Rucio VM

The Rucio VM (10.64.199.233) served NDIP prod until 2026-10-06. It now serves only galaxy.ornl.gov and its Pulsars (pulsar_common_galaxy, rucio_url_prod). Its catalog has diverged from speedy's since the migration.

It is deployed by the rucio_playbook.yml Ansible playbook (role ornl.rucio, Docker Compose with PostgreSQL), and its RSEs are in ansible/group_vars/rucio_server/rses.yml.