Skip to main content

Troubleshooting NDIP Platform

Identifying Problems​

Slack Channels​

The Prometheus stack monitors Kubernetes workloads on mickey and speedy, system services and file systems on the Pulsar hosts, and URL endpoints. Alarms are reported to our Slack channels (ndip-status-prod and ndip-status-test). Each alarm includes a description of the problem and the host where it occurred. See Monitoring.

Automatic GitLab Tests​

  • System Tests: system tests run a set of tools on all compute resources every two hours. Failures are reported in the Slack channel, including the name of the failing test and a link to the corresponding GitLab pipeline.
  • Nightly Tests: nightly tests for our tools are also conducted. GitLab e-mails failures to the pipeline owner (currently Sergey).

Manual Tests​

You can quickly run a tool manually from ndip.ornl.gov/ndip-test.ornl.gov to verify that it works as expected. This is especially useful for interactive tools, since they cannot be tested effectively with automated methods.

Resolving Problems​

The resolution steps depend on the type of problem. Below are some common scenarios and solutions.

note

Most issues arise from infrastructure problems, such as unavailability, restarts, disk space issues, or network problems. Some issues resolve on their own when the infrastructure recovers, while others require manual intervention.

Access

Kubernetes (mickey / speedy)​

First look​

kubectl -n ndip get pods -o wide                       # anything not Running/Ready, restarts, which node
kubectl -n ndip get events --sort-by=.lastTimestamp | tail -30
kubectl -n ndip describe pod <pod>                     # Events: OOMKilled, Evicted, FailedScheduling, image pulls
kubectl -n ndip logs <pod> -c <container> [--previous]
kubectl get nodes                                      # NotReady, pressure conditions
kubectl -n ndip get clusters.postgresql.cnpg.io        # database health

Galaxy logs go to stderr only: kubectl -n ndip logs deploy/galaxy -c galaxy. See Logging.

Galaxy pod restarting or OOMKilled​

kubectl -n ndip describe pod <galaxy-pod> shows Last State: Terminated, Reason: OOMKilled. Uploads (upload1, __DATA_FETCH__) run as local jobs inside the Galaxy container (2 local workers), so a burst of large uploads is the usual cause. Check kubectl -n ndip logs deploy/galaxy -c galaxy --previous for what was running.

  • The memory limit is set in k8s/galaxy/values-<env>.yml (resources.limits.memory).
  • On speedy, the namespace LimitRange caps a container at 2 CPU / 4 Gi. A deploy with a higher limit is rejected: the new ReplicaSet never creates a pod, and kubectl -n ndip get events shows maximum memory usage per Container is 4Gi. Either keep the limit at 4 Gi or ask the cluster admins to raise the LimitRange.
  • gx-it-proxy has its own, smaller limit (gxItProxy.resources).

Pods evicted, node DiskPressure​

Node root disks are 50 GB, and containerd stores images there. A Galaxy image pull (about 3 GB compressed, 8 GB unpacked) can push a node over kubelet's eviction threshold. Kubelet then evicts pods, which can include Longhorn's own pods, and that detaches volumes.

kubectl describe node <node> | grep -A5 Conditions     # DiskPressure=True
kubectl get pods -A --field-selector=status.phase=Failed | grep Evicted

What to do:

  1. Free space on the node: delete Evicted pods; ask the cluster admins to prune unused images (crictl rmi --prune from a node shell); remove stale Longhorn volumes of deleted PVCs.
  2. Watch Longhorn volumes come back (Longhorn UI, or kubectl -n longhorn-system get volumes.longhorn.io). A volume whose only replica failed may need salvaging once the node is above Longhorn's 25% minimum-free floor.
  3. On mickey, check galaxy-pg (single instance, single replica) first.

Background and the requests to the cluster admins are on the Kubernetes clusters page.

Database (CNPG)​

See Database recovery.

RabbitMQ​

  • Brokers in a restart loop. Check kubectl -n ndip logs galaxy-rabbitmq-<n> --previous and describe pod. Common causes:
    • memory limit reached because of orphaned control.* queues (see below), so the broker is OOMKilled or hits its memory alarm;
    • on prod, a broker that can't rejoin the cluster after all three were down together (pause_minority keeps a lone node paused, and the logs show it waiting for peers). Bring all pods back (podManagementPolicy: Parallel starts them together). Only if one node's data is lost, remove it from the cluster with rabbitmqctl forget_cluster_node from a healthy broker and delete its PVC.
  • Orphaned control queues. Every Galaxy rollout leaves durable control.* queues behind. Check rabbitmqctl list_queues name messages consumers | grep ^control, and make sure the control-expiry policy exists (rabbitmqctl list_policies -p /). See RabbitMQ.
  • Pulsars not consuming. Check rabbitmqctl list_queues name consumers. A Pulsar manager's queues with 0 consumers means that Pulsar is down or can't connect. Prod Pulsars connect through the VM proxy (ndip.ornl.gov:5672 → speedy 30671). Test Pulsars connect to one mickey node (10.64.197.29:30672), so they lose AMQP when that node is down.

Jobs stay in new / queued​

The job router defers jobs instead of failing them.

  1. kubectl -n ndip logs deploy/galaxy -c galaxy | grep -E 'cannot get destination status|no resources available|Mapped job'.
  2. cannot get destination status: Galaxy can't reach Prometheus (http://10.64.193.81:32001/api/v1/alerts). Fix the monitoring cluster first.
  3. no resources available: either every allowed host is busy, or a firing alert has taken the hosts out. Open the Prometheus alerts and look for alerts with env: prod and a host label of a Pulsar host (sns-ndip-gpu.sns.gov, ndip-node01.sns.gov, imaginex-1/2.ornl.gov). A warning is enough, for example disk usage over the warning threshold. Fix the cause. Silencing the alert in Alertmanager only stops the notifications: Prometheus's /api/v1/alerts still lists the alert, and the router still skips the host.
  4. Check the job lock on the jobs admin page.

Website down or errors​

  • Prod: check the VM proxy first, then speedy's ingress and the Galaxy pod.
  • 502/503 from the ingress: Galaxy is not ready (starting, or crashlooping). kubectl -n ndip get pods.
  • During a Galaxy redeploy, two pods run briefly. That's expected.

Pulsar hosts and VMs​

Disk Full​

We have automated clean-up processes, but not for all cases. To address disk space issues, log in to the affected node and identify the issue.

Useful commands:

df -h                  # Check mounted folders and their capacity  
du -h -d 1 somefolder  # Identify large folders/files and clean them up if possible  
docker system df       # Docker images/containers on Pulsar hosts

A disk warning on a Pulsar host also takes it out of job rotation (see above). If the issue involves CEPH, you can increase the CEPH share size "on the fly" using:

Service Down​

  1. Log in to the node.
  2. Check the status of the service, review logs, and restart if necessary. 
systemctl status pulsar          # or pulsar_test, rdb, node_exporter
tail -n 200 /var/log/pulsar.log  # or pulsar_test.log, rdb.log
systemctl restart pulsar         # kills running jobs, see the Pulsar page

User Can't Access Experiment Data​

If a user reports that an NDIP/NOVA tool cannot access experiment data they should have access to:

  1. Verify with the user that they are added to the proper IPTS for accessing experiment data.
  2. Log in to the node where the tool is running and run id {{3-character user ID}}.
  3. If id returns user information, check the user's group permissions. If group permissions are missing for the relevant data, have the user e-mail Linux Support.
  4. If id reports that the user is not found, e-mail Linux Support and tell them the user's account is missing from the LDAP system (include the node the tool is running on).

No Access to Node​

  1. For ORC VMs, log in to OpenStack and check the node's status. If it is "stopped", start it again.
  2. For SNS hosts, e-mail support@linux.sns.gov.
  3. For mickey/speedy nodes, check the node in Rancher and contact the cluster admins.

No Access to a Filesystem​

  1. Log in to the node.
  2. Check whether the filesystem is mounted. If not, remount it:
df -h  
umount path_to_folder  
mount -a
  • SNS/HFIR filesystems: check for permission issues.
  • CEPH filesystems: review error messages. It is generally safe to delete CEPH mounts used for Docker, but not data mounts (ceph_share_test, ceph_share).

Galaxy Test Failures​

  1. Check for infrastructure alerts in Slack.
  2. On the cluster:
    • check the Galaxy logs (kubectl -n ndip logs deploy/galaxy -c galaxy) and the job's destination (Mapped job to destination id);
    • check that galaxy, galaxy-rabbitmq-*, galaxy-pg-*, rucio-server-* and rdb-galaxy-* pods are ready;
    • restart Galaxy if needed: kubectl -n ndip rollout restart deployment/galaxy.
  3. Log in to the Pulsar node where the failing test ran:
    • check the logs in /var/log/pulsar.log or /var/log/pulsar_test.log;
    • verify the services (pulsar/pulsar_test, rdb);
    • confirm that the filesystems are mounted correctly.

Authorization Problems​

We support two OIDC providers, AzureAD and PingFed. If either has problems, logins to Galaxy and token updates might fail.

  • For PingFed - create a support ticket here using the "SSO Request" option.
  • For AzureAD - try to login here and check if everything looks ok (expired certificates, redirect URIs match your expectations, etc.). Greg Watson, Andrew Ayres, Sergey Yakubov, John Duggan, and Greg Cage all should have access.

Job Launch Problems​

The following error messages when launching jobs have been previously encountered:

  • "Could not authenticate job" - Pulsar failed to get a valid access token for the user. This usually means there is an issue with one of the above OIDC providers that should be investigated. Logging out and back in will resolve the issue temporarily for a user.
  • "You have to remove (or rename) that container to be able to reuse that name." - Multiple job handlers tried to run the same job. On Kubernetes, check that only one Galaxy pod is running once a rollout has finished (kubectl -n ndip get pods -l app.kubernetes.io/name=galaxy). On the old VM, it meant stale Galaxy processes survived a restart (ps -eo user,pid,ppid,lstart,args | grep -E "[g]unicorn|[g]alaxy-main"). A second Galaxy on the VM is also what sudo galaxyctl starts, so never run it there.
  • Jobs that die for no reason during a deploy: a Pulsar restart SIGKILLs running job containers. See Pulsar.