Skip to main content

Known Issues

1. System shut down by Linux Support​

Linux support shut down one of our Pulsar nodes without advance notice due to an upcoming power outage on July 9th, 2026.

How to Identify​

  1. Slack sent an alert that metrics were unavailable on the target machine.
  2. I attempted to login to the Pulsar node but DNS was failing to resolve on the machine.

How to Resolve​

  1. Emailed linux@support.sns.gov asking them to look into why the machine was down.

2. System power cycled unexpectedly​

On July 9th, 2026, one of our Pulsar nodes that wasn't meant to shut down during the power outage was power cycled unintentionally by the outage. Upon restoration of power, the machine superficially was operating but RDB was to process download requests.

How to Identify​

  1. Inspected the pulsar and rdb services via systemctl and confirmed they were not in an error state.
  2. Checked the logs for both services in /var/log. In /var/log/rdb.log, I noticed that there were many entries referencing file download requests that were never responded to.
  3. Restarted the rdb service via systemctl to no effect.
  4. Checked the rdb config and noticed it depended on /SNS/NDIP and /ceph mounts to operate.
  5. Checked both mounts, and noticed that /ceph mount was broken.

How to Resolve​

  1. Reboot the machine.
  2. Rerun the Pulsar deployment pipeline if there are still issues.