Skip to main content

Known Issues

Second Galaxy started on the old prod VM​

On October 6th, 2026, during the prod migration, sudo galaxyctl ... was run on the old prod VM. As root, gravity switched to its systemd mode and started a second Galaxy (galaxy.target, galaxy-*@ units) on the same database as the supervisord-managed one.

How to Identify​

  1. systemctl list-units 'galaxy*' shows galaxy-*@ units next to galaxy.service.
  2. Duplicate gunicorn/handler processes: ps -eo user,pid,ppid,lstart,args | grep -E "[g]unicorn|[g]alaxy-main".

How to Resolve​

  1. Stop the systemd-mode units (sudo systemctl stop galaxy.target).
  2. Only ever control VM Galaxy with sudo systemctl start|stop galaxy.service. VM Galaxy is stopped now anyway; see VM proxy.

mickey test stack down from node disk pressure​

On October 5th, 2026, mickey's workers repeatedly crossed kubelet's eviction threshold on their 47 GB root disks, which they share with images, Longhorn and logs. About 300 evictions followed, including Longhorn's instance-manager and CSI pods, which detached every Longhorn volume. The single replica of the test Galaxy database volume was marked failed and could not be salvaged while the node was below Longhorn's 25% free floor.

How to Identify​

  1. Many Evicted pods across namespaces; nodes report DiskPressure.
  2. galaxy-pg not ready; Longhorn volumes detached or faulted.

How to Resolve​

  1. Free space by hand (stale Longhorn volumes, unused images, logs), then let Longhorn salvage the replica.
  2. See DiskPressure and the requests on the Kubernetes clusters page.

Test jobs stopped during mickey node maintenance​

On September 21st, 2026, maintenance on the one mickey node that test Pulsars use for AMQP (NodePort 30672) stopped all test jobs. Until an F5 TCP virtual server exists for AMQP, test Pulsars depend on that single node.

Pulsar deploy killed running jobs​

On September 4th, 2026, a Pulsar deploy on ndip-node01 hung for 90 s and then systemd SIGKILLed Pulsar and about 28 running job containers. These jobs showed up in Galaxy as failures with no clear cause. This happens on every Pulsar restart while jobs are running: drain the host first (see Pulsar).

Jobs failed after stop due to resource orchestration bug​

On July 27th, 2026, tools were able to run but would always fail postprocessing due to issues with the SNS-HFIR mounts during an SNS patching window.

How to Identify​

  1. Run any tool using compute nodes using SNS-HFIR mounts for storage of tool outputs.
  2. Confirm that the tool ran but ultimately failed in Galaxy. Usually, this contains the text Remote job server indicated a problem running or monitoring this job.
  3. ssh into the compute node, look at /var/log/pulsar{_test}.log, and look for write failures to /SNS or /HFIR.

How to Resolve​

  1. Email linux@support.sns.gov for help fixing the mounts. If during SNS patching, it's still helpful to email to let them know so that they double-check the mounts after patching.

Monitoring failed to recover properly from ORC outage​

On July 20th, 2026, the Prometheus monitoring was down after an ORC maintenance period ended. The machine was online and deployment pipelines ran, but the Kubernetes service was not responding properly to requests.

How to Identify​

  1. Saw that http://10.64.193.81:32001/alerts was not responding to requests via a web browser.
  2. Ran the prometheus deployment pipeline to no effect.

How to Resolve​

  1. Hard rebooted the machine in OpenStack.
  2. Deleted all existing K8s pods with kubectl --kubeconfig ~/.kube/config-prometheus delete pods --all -n monitoring where config-prometheus contained the K8S_CONFIG deployment variable.
  3. Re-ran the prometheus pipeline to start all services with a clean slate.

SNS-HFIR experiment data mounts were lost on reboot​

On July 17th, 2026, both of our SNS Pulsar nodes were restarted while attempting to bring services back online after the end of an ORC maintenance period. Upon restart, the /HFIR and /SNS mounts were not mounted properly.

How to Identify​

  1. Normally, Slack alerts would've automatically alerted us to this, but monitoring was still down due to ORC maintenance.
  2. Logged into each Pulsar node after reboot and saw that /HFIR and /SNS symlinks were pointing to invalid paths.

How to Resolve​

  1. Run sudo mount -o remount -a.
  2. Email linux@support.sns.gov immediately if the above command fails to recover the data mounts.

System shut down by Linux Support​

Linux support shut down one of our Pulsar nodes without advance notice due to an upcoming power outage on July 9th, 2026.

How to Identify​

  1. Slack sent an alert that metrics were unavailable on the target machine.
  2. I attempted to login to the Pulsar node but DNS was failing to resolve on the machine.

How to Resolve​

  1. Emailed linux@support.sns.gov asking them to look into why the machine was down.

System power cycled unexpectedly​

On July 9th, 2026, one of our Pulsar nodes that wasn't meant to shut down during the power outage was power cycled unintentionally by the outage. Upon restoration of power, the machine superficially was operating but RDB was to process download requests.

How to Identify​

  1. Inspected the pulsar and rdb services via systemctl and confirmed they were not in an error state.
  2. Checked the logs for both services in /var/log. In /var/log/rdb.log, I noticed that there were many entries referencing file download requests that were never responded to.
  3. Restarted the rdb service via systemctl to no effect.
  4. Checked the rdb config and noticed it depended on /SNS/NDIP and /ceph mounts to operate.
  5. Checked both mounts, and noticed that /ceph mount was broken.

How to Resolve​

  1. Reboot the machine.
  2. Rerun the Pulsar deployment pipeline if there are still issues.