Introduction
This section describes the infrastructure behind NDIP: storage, network and compute. The diagram below shows the original VM-based layout and is kept for reference. It predates the move of the core services to Kubernetes.
NDIP consists of two primary components:
- Core services: Galaxy and the services it depends on. These run in namespace
ndipon two RKE2 Kubernetes clusters managed through Rancher. mickey runs test (ndip-test.ornl.gov) and speedy runs production (ndip.ornl.gov). See Kubernetes clusters. - Worker nodes: the compute resources where user jobs run, through Pulsar:
- Neutron Science analysis cluster (
sns-ndip-gpu.sns.gov,ndip-node01.sns.gov) - imaginex (
imaginex-1.ornl.gov,imaginex-2.ornl.gov) - OLCF Frontier, through the Slate Kubernetes cluster
- NERSC Perlmutter, through the Spin Kubernetes cluster
- Neutron Science analysis cluster (
note
For the services themselves, see the services section.
How traffic reaches NDIP
| Name | Path today |
|---|---|
ndip-test.ornl.gov | DNS → F5 VIP → mickey's ingress-nginx |
ndip.ornl.gov (also calvera.ornl.gov, nova.ornl.gov) | DNS → old prod VM 128.219.184.101 → nginx stream SNI passthrough → speedy's ingress-nginx (443) |
Prod AMQP ndip.ornl.gov:5672 (Pulsar ↔ RabbitMQ) | old prod VM → speedy NodePort 30671 (TLS) |
| Test AMQP | Pulsars connect directly to a single mickey node on NodePort 30672 |
The prod path through the VM is temporary, until the F5 onboarding and DNS cutover for speedy are done. See Temporary VM proxy.
Currently in use
| Name | Host | Managed by |
|---|---|---|
| NDIP production | speedy cluster (ndip.ornl.gov), namespace ndip | Rancher (cluster); calvera k8s/ CI jobs (workloads) |
| NDIP test | mickey cluster (ndip-test.ornl.gov), namespace ndip | Rancher (cluster); calvera k8s/ CI jobs (workloads) |
| Old prod VM, now a proxy only | 128.219.184.101 (ndip.ornl.gov in DNS) | Manual; see VM proxy. Also hosts Globus and the galaxy-mcp server |
| Pulsar "SNS GPU" node | sns-ndip-gpu.sns.gov | Ansible (pulsar-hosts); OS by support@linux.sns.gov |
| Pulsar "SNS CPU" node | ndip-node01.sns.gov | Ansible; OS by support@linux.sns.gov |
| Pulsar imaginex nodes | imaginex-1.ornl.gov, imaginex-2.ornl.gov | Ansible |
| OLCF/Slate/Marble | https://console-openshift-console.apps.marble.ccs.ornl.gov | Helm, manually (NCCS) |
| NERSC/Spin | https://rancher2.spin.nersc.gov | Helm, manually (NERSC) |
| Monitoring cluster (Prometheus, Alertmanager, Push Gateway, Blackbox) | 10.64.193.81 | calvera k8s/monitoring/helm (CI job prometheus) |
| Documentation server | 10.64.196.237 (docs-vm) | Ansible (CI job docs) |
| Ingress proxy + Kafka VM | 10.64.193.19 | Ansible (CI job ingress-proxy) |
Rucio VM, for galaxy.ornl.gov only | 10.64.199.233 | Ansible (CI job rucio-prod) |
galaxy.ornl.gov (separate Galaxy instance) and its Pulsars pulsar-cpu, pulsar-large-cpu | ORC VMs | Ansible (CI jobs galaxy-common, pulsars-common-galaxy) |
Retired or legacy
| Name | Status |
|---|---|
ndip-test VM (ndip-test.ornl.gov Galaxy) | Retired. DNS points at mickey |
| Prod VM Galaxy, RabbitMQ and Rucio-for-ndip | Stopped since the 2026-10-06 migration. VM Postgres data is kept for rollback during the soak period |
Rucio test VM (10.64.192.94) | Not used by NDIP test any more; test Rucio runs on mickey. |
| ORC Pulsar "test"/"export" VMs | Removed from the inventory |
In the following sections, we provide details about infrastructure components and their deployment where applicable.