Deploying NDIP

All deployments are run by hand as GitLab CI jobs in the calvera repository. Every job is when: manual: open Build > Pipelines, start a pipeline on the branch you want to deploy, and click Run on the job.

Branches

The deployed Kubernetes configuration currently lives on calvera branch k8s-env-split. The Kubernetes jobs (*-test and *-k8s-prod) can run from any branch until the VM cutover is finished. After that, prod deploys move back to main only and the jobs are renamed *-prod. The Ansible prod jobs for the Pulsars run only from main.

Kubernetes jobs

Each Kubernetes job decrypts k8s/<component>/secrets-<env>.yml with the vault password, then runs k8s/<component>/deploy.sh in the KUBECTL_IMAGE container with DEPLOY_ENV=test|prod. k8s/lib/env.sh loads k8s/env/<env>.env and fills the @NAME@ placeholders (host, Rucio scope, RSE suffix, ...) in the values files. Values files are named values-<env>.yml.

Component	test → mickey	prod → speedy	Notes
Whole stack	ndip-full-stack-test	ndip-k8s-full-stack-prod	k8s/deploy-all.sh: CNPG operator, Postgres, Rucio, oncat-proxy, nova-dashboard, Galaxy (+ ELK on test). Does not deploy RDB, ingress-proxy or kube-state-metrics
CNPG operator	cnpg-operator-test	cnpg-operator-prod	Cluster-wide, namespace cnpg-system. Run before Postgres
PostgreSQL	postgres-test	postgres-k8s-prod	CNPG clusters, secrets, backups (prod)
Rucio	rucio-test	rucio-k8s-prod	Server, /rucio Ingress, db-init Job, scopes and RSEs (prepare_rses.py)
Galaxy	galaxy-test	galaxy-k8s-prod	Chart, prod extra routes, then rollout restart
Remote Data Broker	rdb-test	rdb-k8s-prod	rdb-galaxy + the /v0.1 Ingress
NOVA dashboard	nova-dashboard-test	nova-dashboard-k8s-prod
ONCat proxy	oncat-proxy-test	oncat-proxy-k8s-prod
kube-state-metrics	kube-state-metrics-test	kube-state-metrics-k8s-prod	Needs CI variable KSM_SCRAPE_PASSWORD
Ingress proxy	ingress-proxy-test	ingress-proxy-k8s-prod	The live ingress proxy still runs on its VM (job ingress-proxy)
ELK	elk-test		Test only

Order for a fresh cluster: cnpg-operator-* → postgres-* → rucio-* → galaxy-* → the rest. The full-stack job follows this order.

Jobs that still use Ansible

Job	Target	Branch
pulsars-sns-prod, pulsars-prod, pulsars-imaginex-prod	prod Pulsars on ndip-node01, sns-ndip-gpu, imaginex-1/2	main only
pulsars-sns-test, pulsars-test, pulsars-imaginex-test, *-dev	test/dev Pulsar instances on the same machines	any
galaxy-common, pulsars-common-galaxy, rucio-prod	galaxy.ornl.gov, its Pulsar VMs, and its Rucio VM (10.64.199.233)	any
docs	docs server 10.64.196.237	any
ingress-proxy	ingress-proxy/Kafka VM 10.64.193.19	any
prometheus, prometheus push gateway	monitoring cluster (Helm, not Ansible)	any
filebeat pulsar, globus, ai-services	Pulsar hosts / prod VM / AI VM	any

Do not run

ndip, nova-dashboard-prod and oncat-proxy-prod deploy to the old prod VM, which is now the VM proxy. Running them rewrites the VM's nginx and restarts its Galaxy, which breaks production. globus also targets that VM: check what it changes before running it.

Pulsar deploys kill running jobs

Ansible restarts the Pulsar systemd unit directly. Pulsar does not exit while its job containers are alive, so systemd waits out TimeoutStopSec (90s) and then kills Pulsar and every running job. Drain the host first (see Pulsar). Because one machine carries prod, test and dev aliases, a "test" run still executes host-level roles (Docker etc.) on a production machine.

Deploying Galaxy to production

Galaxy redeploys are rolling updates: the new pod starts before the old one stops, so the web UI has little or no downtime. Jobs already handed to Pulsar keep running, and their messages wait in RabbitMQ. Uploads run as local jobs inside the Galaxy pod and are lost when the old pod stops.

1. Announce the change in the #ndip-nova Slack channel.
2. For risky changes (schema migrations, job config), lock job dispatch on the jobs admin page and wait for running jobs. The lock does not survive a Galaxy restart, so set it again after the rollout if you still need it.
3. Run galaxy-k8s-prod on the deployed branch. It runs helm upgrade, applies extra-routes-prod.yaml, restarts the Deployment and waits up to 10 minutes for the rollout.
4. Check: the job log ends with Done., kubectl -n ndip get pods -l app.kubernetes.io/name=galaxy shows one ready pod, https://ndip.ornl.gov loads, and a test job routes as expected (kubectl -n ndip logs deploy/galaxy -c galaxy | grep 'Mapped job to destination id').
5. Announce completion.

Tools

Galaxy's tools-sync init container clones galaxy-tools every time the pod starts. Prod uses branch main with no prototypes. Test uses dev plus the prototype branch (toolsSync in values-<env>.yml). To roll out tool changes, merge them to the right galaxy-tools branch and restart Galaxy, either by rerunning the Galaxy job or with kubectl -n ndip rollout restart deployment/galaxy. The GALAXY_TOOLS_VERSION CI variable only affects the Ansible (VM) jobs. tool_conf.xml is k8s/galaxy/values-toolconf-<env>.yml.

Secrets

Secrets are ansible-vault encrypted files committed next to each component: k8s/<component>/secrets-test.yml and k8s/<component>/secrets-prod.yml (secrets.yml.sample lists the keys). CI decrypts them into secrets-decrypted.yml at deploy time, and that file is never committed. The vault password is the calvera CI/CD variable CALVERA_VAULT_PASSWORD_FILE. Ansible secrets for the VMs and Pulsars are under ansible/host_vars/*/secrets.yml and ansible/group_vars/*/, edited with the scripts in ansible/scripts/.

# write the vault password to a local file outside git
ansible-vault edit k8s/galaxy/secrets-prod.yml --vault-password-file ~/ndip-vault-password.txt

Things to know:

• Postgres role passwords are reconciled. Changing galaxyDbPassword/rucioDbPassword in k8s/postgres/secrets-<env>.yml and running the Postgres job really changes the password. Update Galaxy's galaxy.secrets.databaseUrl and Rucio's config.database.default in the same window. Percent-encode the password in connection strings if it contains @ : / ? # [ ] %.
• Helm replaces lists. Lists that need secrets (for example ingress.extraHosts with certificates) are set entirely in the secrets file, which is applied last.

Other CI/CD variables used by the Kubernetes jobs: KUBE_CONFIG / KUBE_CONFIG_PROD (base64 kubeconfigs), NDIP_DOCKER_USER / NDIP_DOCKER_PASSWORD (savannah.ornl.gov pull secret), *_CHART_PROJECT_ID (Helm registries), KSM_SCRAPE_PASSWORD, and optionally CNPG_REGISTRY. Check before a prod change that KUBE_CONFIG_PROD is not about to expire.

Helm charts

NDIP's charts live in their own repositories under charts/ (galaxy, rucio, remote-data-broker, nova-dashboard, oncat-proxy, ingress-proxy, pulsar, minio).

1. Change the chart and bump version in Chart.yaml.
2. Push. The helm-file-test job runs helm package . (the only automated check; there is no lint or template test).
3. On main, run the manual helm-file-build job. It publishes the package to the project's GitLab Helm registry, in the channel named after the branch.
4. calvera's deploy.sh scripts use .../packages/helm/main with no version pin, so the next deploy of that component, on either cluster, picks up the newest chart in the main channel.

Keep charts environment-agnostic: hostnames and other per-environment values belong in calvera's k8s/*/values-*.yml or k8s/env/*.env.

Container images

• Galaxy: built from the Galaxy fork (branch k8s-support), see Managing the Galaxy codebase. Both clusters pull the floating tag :k8.
• Everything else (Rucio, RDB, nova-dashboard, oncat-proxy, gx-it-proxy) is pinned by tag in the chart defaults or in calvera's values-<env>.yml. To upgrade, change the tag and run the component's job.