Skip to main content

Known Issues

Jobs failed after stop due to resource orchestration bug​

On July 27th, 2026, tools were able to run but would always fail postprocessing due to issues with the SNS-HFIR mounts during an SNS patching window.

How to Identify​

  1. Run any tool using compute nodes using SNS-HFIR mounts for storage of tool outputs.
  2. Confirm that the tool ran but ultimately failed in Galaxy. Usually, this contains the text Remote job server indicated a problem running or monitoring this job.
  3. ssh into the compute node, look at /var/log/pulsar{_test}.log, and look for write failures to /SNS or /HFIR.

How to Resolve​

  1. Email linux@support.sns.gov for help fixing the mounts. If during SNS patching, it's still helpful to email to let them know so that they double-check the mounts after patching.

Monitoring failed to recover properly from ORC outage​

On July 20th, 2026, the Prometheus monitoring was down after an ORC maintenance period ended. The machine was online and deployment pipelines ran, but the Kubernetes service was not responding properly to requests.

How to Identify​

  1. Saw that http://10.64.193.81:32001/alerts was not responding to requests via a web browser.
  2. Ran the prometheus deployment pipeline to no effect.

How to Resolve​

  1. Hard rebooted the machine in OpenStack.
  2. Deleted all existing K8s pods with kubectl --kubeconfig ~/.kube/config-prometheus delete pods --all -n monitoring where config-prometheus contained the K8S_CONFIG deployment variable.
  3. Re-ran the prometheus pipeline to start all services with a clean slate.

SNS-HFIR experiment data mounts were lost on reboot​

On July 17th, 2026, both of our SNS Pulsar nodes were restarted while attempting to bring services back online after the end of an ORC maintenance period. Upon restart, the /HFIR and /SNS mounts were not mounted properly.

How to Identify​

  1. Normally, Slack alerts would've automatically alerted us to this, but monitoring was still down due to ORC maintenance.
  2. Logged into each Pulsar node after reboot and saw that /HFIR and /SNS symlinks were pointing to invalid paths.

How to Resolve​

  1. Run sudo mount -o remount -a.
  2. Email linux@support.sns.gov immediately if the above command fails to recover the data mounts.

System shut down by Linux Support​

Linux support shut down one of our Pulsar nodes without advance notice due to an upcoming power outage on July 9th, 2026.

How to Identify​

  1. Slack sent an alert that metrics were unavailable on the target machine.
  2. I attempted to login to the Pulsar node but DNS was failing to resolve on the machine.

How to Resolve​

  1. Emailed linux@support.sns.gov asking them to look into why the machine was down.

System power cycled unexpectedly​

On July 9th, 2026, one of our Pulsar nodes that wasn't meant to shut down during the power outage was power cycled unintentionally by the outage. Upon restoration of power, the machine superficially was operating but RDB was to process download requests.

How to Identify​

  1. Inspected the pulsar and rdb services via systemctl and confirmed they were not in an error state.
  2. Checked the logs for both services in /var/log. In /var/log/rdb.log, I noticed that there were many entries referencing file download requests that were never responded to.
  3. Restarted the rdb service via systemctl to no effect.
  4. Checked the rdb config and noticed it depended on /SNS/NDIP and /ceph mounts to operate.
  5. Checked both mounts, and noticed that /ceph mount was broken.

How to Resolve​

  1. Reboot the machine.
  2. Rerun the Pulsar deployment pipeline if there are still issues.