Temporary VM Proxy (ndip.ornl.gov)

Temporary

This setup bridges the gap until the F5 load balancer onboarding and the DNS cutover of ndip.ornl.gov to the speedy cluster are done. Remove it as described in When the F5 takes over.

Since the production migration (2026-10-06), DNS for ndip.ornl.gov still points at the old production VM (128.219.184.101, Ubuntu 22.04). The VM's nginx forwards that traffic to speedy at the TCP level, using a stream block with ssl_preread. TLS passes through untouched: speedy's ingress and RabbitMQ terminate it with the same certificates, so the proxy itself needs no certificates.

Restoring the proxy

The proxy is reproducible with Ansible from calvera branch vm-proxy (it only lives on that branch; it never merges to dev or main). Use it if the VM's nginx config is lost or the VM is rebuilt:

• CI job vm-proxy (manual) on that branch, or by hand from ansible/: ansible-playbook -i inventories/vm-proxy-hosts vm_proxy_playbook.yml --private-key <key> (add --check --diff to preview).
• Settings in ansible/group_vars/vm_proxy/proxy.yml: speedy node IPs, ports, the Globus domain. Templates and a README in ansible/templates/vm_proxy/. The inventory addresses the VM by IP, so it keeps working after DNS moves.
• The playbook installs nginx from nginx.org if missing, stops and disables the retired RabbitMQ, keeps the retired Galaxy sites out of sites-enabled, writes nginx.conf, ndip-stream.conf, the port 80 redirect and the local-only Globus site, validates with nginx -t, restarts nginx and checks that 443, 5672 and 80 listen.
• It doesn't set up Globus itself or galaxy-mcp. If Globus isn't set up, it skips the Globus site and says so.
• Re-running Globus's own nginx step puts Globus back on 443 and breaks the proxy; rerun the vm-proxy job afterwards.

What goes where

VM port	Traffic	Goes to
443, SNI fedf59.6fbd.gaccess.io	Globus	VM's own nginx, moved to 127.0.0.1:4443
443, any other SNI	ndip.ornl.gov (Galaxy, /rucio, /v0.1, /nova, /oncat, /docs, /mcp), calvera.ornl.gov, nova.ornl.gov	speedy nodes :443
5672	Pulsar amqps	speedy nodes :30671 (RabbitMQ TLS NodePort)
80	plain HTTP	VM nginx redirects to https://ndip.ornl.gov
8002	galaxy-mcp	Local service; speedy's /mcp Ingress proxies back to 128.219.184.101:8002

The configuration is in /etc/nginx/ndip-stream.conf, included at the top level of /etc/nginx/nginx.conf (outside http). Its upstreams are the three speedy nodes (10.64.193.103, 10.64.197.44, 10.64.201.29) on 443 and 30671. proxy_timeout is 1h for long downloads and long-lived AMQP connections. The pre-cutover nginx tree is saved at /root/nginx-before-cutover.

What still runs on the VM

Service	State
nginx (stream proxy, port-80 redirect, Globus on 127.0.0.1:4443)	running
galaxy-mcp (/mcp, port 8002, upstream galaxyproject/galaxy-mcp v1.8.0, GALAXY_URL=https://ndip.ornl.gov)	running. Clients connect() with their own Galaxy API key. Its systemd unit file was lost; to be moved to speedy
Globus (fedf59.6fbd.gaccess.io)	served by the local nginx, but gcs_manager has been dead since 2026-09-08 (API returns 503)
Galaxy, RabbitMQ (disabled), Rucio-for-ndip	stopped
PostgreSQL data	kept, stopped, for rollback during the soak period

Rules

• Don't run the VM Ansible jobs (ndip, nova-dashboard-prod, oncat-proxy-prod). The Galaxy playbook rewrites nginx and restarts the VM's services. (galaxy-common targets galaxy.ornl.gov, a different host.)
• Never run sudo galaxyctl ... on the VM. As root, gravity switches to systemd mode and starts a second Galaxy on the same database. This happened on 2026-10-06. VM Galaxy is controlled only with sudo systemctl start|stop galaxy.service.
• Keep RabbitMQ disabled (systemctl disable). If it starts at boot it takes port 5672, and nginx then fails to start.
• Speedy's ingress sees every client as the VM's address. Real client IPs are not available until the F5 is in place, so source-IP allowlists on speedy Ingresses won't work yet.

Checks

sudo ss -ltnp | grep -E ':(443|4443|5672|80) '      # nginx on all four
curl -s https://ndip.ornl.gov/api/version            # answered by speedy
curl -sI https://calvera.ornl.gov/ | grep -i ^location
echo | openssl s_client -connect ndip.ornl.gov:5672 -servername ndip.ornl.gov 2>/dev/null | grep 'Verify return code'   # 0 (ok)

Prometheus also probes https://ndip.ornl.gov/rucio/ping and TCP ndip.ornl.gov:5672, so a broken proxy raises alerts.

Rollback to the VM

Only for a deliberate rollback of production to the VM:

sudo rm -rf /etc/nginx && sudo cp -a /root/nginx-before-cutover /etc/nginx
sudo nginx -t && sudo systemctl restart nginx
sudo systemctl enable --now rabbitmq-server
sudo systemctl start galaxy.service     # never `galaxyctl`

Then restart the VM's Rucio and repoint the prod Pulsars (rucio_url) at it. Anything created on speedy since the cutover is not in the VM database.

When the F5 takes over

1. Point DNS for ndip.ornl.gov, calvera.ornl.gov and nova.ornl.gov at the F5 and wait out the TTL.
2. On the VM, remove include /etc/nginx/ndip-stream.conf; from nginx.conf, and move Globus back to 443 (or off the VM).
3. Move galaxy-mcp to speedy and update the mcp Ingress in k8s/galaxy/extra-routes-prod.yaml.
4. Prometheus target labels and the AMQP probe keep working unchanged. The cluster side doesn't change.

The SNI proxy must never coexist with a Galaxy prod Ingress on the same ingress controller. That is not the case here: the proxy runs on the VM, outside the cluster.