Kubernetes Clusters

NDIP's core services run on two RKE2 clusters managed through Rancher. Both carry the same stack in namespace ndip. They differ only in sizing and in the per-environment settings in calvera's k8s/env/<env>.env.

	mickey	speedy
Environment	test (DEPLOY_ENV=test)	production (DEPLOY_ENV=prod)
Host	ndip-test.ornl.gov	ndip.ornl.gov
Front door	F5 VIP	old prod VM, SNI passthrough (VM proxy); F5 pending
Rucio scope / RSE suffix	galaxy_test / _TEST	galaxy / none
Galaxy upload RSE	GALAXY_TEST	GALAXY
CNPG instances per DB	1 (galaxy-pg pinned to mickey-wn01)	2 (primary + async standby)
DB backups	off (PG_BACKUPS=false)	nightly pg_dump + Longhorn snapshots
RabbitMQ	1 broker, plain AMQP NodePort 30672	3 brokers, quorum queues, TLS NodePort 30671
Persistent storage	shared-nvme (RWX), Longhorn on the 47 GB root disk	longhorn on a separate ~983 GB disk per worker
CI kubeconfig variable	KUBE_CONFIG	KUBE_CONFIG_PROD

Speedy worker nodes: 10.64.193.103, 10.64.197.44, 10.64.201.29. The ingress controller is RKE2's bundled rke2-ingress-nginx (IngressClass nginx), configured with RKE2's HelmChart/HelmChartConfig mechanism.

Access

• Rancher UI: cluster status, workloads, shells, logs, and the per-cluster kubeconfig download.
• kubectl: download a kubeconfig from Rancher. The CI jobs use base64-encoded kubeconfigs from the calvera CI/CD variables KUBE_CONFIG (mickey) and KUBE_CONFIG_PROD (speedy). The CI containers add a hosts entry piper01.sns.gov → 10.64.194.88 to reach the API server; you may need the same entry locally.

export KUBECONFIG=~/.kube/speedy.yaml      # or mickey
kubectl -n ndip get pods -o wide

danger

Use clusters.postgresql.cnpg.io (not cluster) with kubectl. On these clusters the short name cluster resolves to Rancher's own clusters.management.cattle.io.

What runs where

Workload	Kind / name	Namespace	Notes
Galaxy	Deployment galaxy (containers galaxy, gx-it-proxy; init containers tools-sync, it-map)	ndip	Helm release galaxy (chart charts/galaxy)
RabbitMQ	StatefulSet galaxy-rabbitmq	ndip	Bitnami subchart of the Galaxy chart
PostgreSQL	CNPG clusters galaxy-pg, rucio-pg	ndip	Apps connect to <cluster>-rw
CNPG operator	Deployment (2 replicas)	cnpg-system	Cluster-wide, one per cluster
Rucio server	Deployment rucio-server	ndip	Helm release rucio-server; Ingress /rucio
Remote Data Broker	Helm release rdb-galaxy	ndip	Serves the galaxy-data PVC; Ingress /v0.1
NOVA dashboard	Helm release nova-dashboard	ndip	/nova on the Galaxy Ingress
ONCat proxy	Helm release oncat-proxy	ndip	Ingress /oncat
kube-state-metrics	Helm release	monitoring	/kube-state-metrics/metrics, scraped by the external Prometheus
Extra routes (prod only)	Ingresses docs, nova-tutorial, nova-sans, mcp	ndip	k8s/galaxy/extra-routes-prod.yaml
ELK (test only)	Elasticsearch, Kibana (/logs), Filebeat	ndip	Pinned to mickey-wn01
Rancher Logging / Loki		cattle-monitoring-system	Ships all pod logs to Loki; see logging

Galaxy's PVC galaxy-data (RWX) holds the GALAXY/GALAXY_TEST RSE files, the object store cache, job working directories and the interactive tools map. Galaxy's Kubernetes job pods and rdb-galaxy mount it too.

Limits and known constraints

• LimitRange on speedy. The ndip namespace on speedy caps each container at 2 CPU / 4 Gi. A pod that asks for more is rejected at admission, so a Galaxy redeploy with a higher limit fails rather than running. Galaxy currently runs with a 4 Gi limit on prod and 8 Gi on test. k8s/galaxy/values-prod.yml asks for 8Gi, which this LimitRange rejects: prod stays at 4Gi until the admins raise it (and the ndip quota, 32Gi, which ELK and an 8Gi Galaxy together would exceed).
• Node root disks are 50 GB. containerd's image store lives on the root disk. The Galaxy image is about 3 GB compressed and about 8 GB unpacked, so one pull can push a node into kubelet DiskPressure, which evicts pods. On mickey, Longhorn also lives on the root disk, and evictions on 2026-10-05 took the whole test stack down. Requests to the cluster admins (move containerd to the large disk on speedy, a persistent volume for Rancher's Prometheus, discard_unpacked_layers, bigger disks on mickey) are tracked separately.
• Galaxy rollouts overlap. The Galaxy Deployment uses RollingUpdate, so for a short time during each redeploy two Galaxy pods run against the same database and RabbitMQ.
• The Galaxy image tag floats. Both clusters run savannah.ornl.gov/ndip/galaxy:k8 with pullPolicy: Always. Any pod restart on either cluster picks up whatever image was last tagged :k8 (see Managing the Galaxy codebase).
• Images from ghcr.io. CNPG images are published only on ghcr.io. If a cluster can't pull them, set the CI variable CNPG_REGISTRY to a mirror prefix.