Overview

danger

NDIP job submission uses this monitoring to decide whether compute nodes can receive jobs. Galaxy's job router reads the firing alerts from Prometheus and skips any Pulsar host named (label host) in a firing alert with env: prod, or with the cluster's own env. Even a warning takes a host out of rotation. If Galaxy cannot reach Prometheus at all, every job stays in the new state waiting for healthy compute nodes.

We use a set of services to monitor NDIP and raise alerts when there are problems: Kubernetes workloads, system services, Docker, disk space, web endpoints, system tests, tool tests, and so on.

Service	Host	Configuration
Prometheus stack (Prometheus, Alertmanager, Push Gateway, Blackbox)	monitoring cluster 10.64.193.81	calvera k8s/monitoring/helm/, CI job prometheus
kube-state-metrics	mickey and speedy, namespace monitoring	calvera k8s/kube-state-metrics/, CI jobs kube-state-metrics-test / -k8s-prod
Node Exporters	Pulsar hosts and remaining VMs	Ansible role node_exporter
Slack	ndip-status-prod, ndip-status-test	k8s/monitoring/helm/prometheus-slack.yaml
Logging	Loki on both clusters, ELK in progress	Rancher Logging, calvera k8s/elk/

Further details about each service are provided in the corresponding subsections.

What is monitored

Metric	Source
Kubernetes Deployments/StatefulSets/DaemonSets below desired replicas, CNPG cluster degraded, node not ready, pods pending or crashlooping	kube-state-metrics on mickey and speedy (scrape job ndip-k8s)
Systemd services, disk space, mounts on Pulsar hosts and VMs	Node Exporter
Docker response time & number of running containers	Push Gateway, Docker metrics
Web endpoints: https://ndip(-test).ornl.gov, /nova, /oncat, /rucio/ping, /docs/, login providers	Black Box (HTTP)
Prod AMQP ndip.ornl.gov:5672 (through the VM proxy)	Black Box (tcp_connect)
System tests, tool tests, interactive tool tests	Push Gateway

The Kubernetes rules are in the k8s-capacity group of prometheus-rules.yaml:

Alert	Fires when
deployment / statefulset under capacity	ready replicas < desired (critical when none are ready)
daemonset under capacity	scheduled pods not ready
postgres cluster degraded	kube_cnpg_cluster_status_ready_instances < spec instances
postgres cluster metrics missing	CNPG metrics absent (the degraded rule could never fire)
pod stuck pending / pod crashlooping	a pod is pending or in CrashLoopBackOff
node not ready	a node's Ready condition is false

Take a look at the alert rules in calvera (k8s/monitoring/helm/prometheus-rules.yaml) for details.

Which branch is deployed

The ndip-k8s scrape jobs, the /rucio/ping and AMQP probes, and the k8s-capacity rules are on calvera branch k8s-capacity-alerts, not dev. The prometheus job deploys whatever branch it runs from, so running it from a branch without these changes silently removes them. Merge first, or deploy from that branch.

kube-state-metrics

One kube-state-metrics per cluster (namespace monitoring) exports only what the rules use (metricAllowlist in k8s/kube-state-metrics/values.yml), plus two CNPG gauges. Batch Jobs are deliberately not collected, because Galaxy creates one per Kubernetes tool run. It is exposed at https://<host>/kube-state-metrics/metrics with basic auth (user prometheus, password from the CI variable KSM_SCRAPE_PASSWORD, which both the KSM jobs and the prometheus job read).

• speedy is scraped as https://ndip.ornl.gov/kube-state-metrics/metrics (through the VM proxy).
• mickey is scraped through a node IP (10.64.193.253:443, TLS server name ndip-test.ornl.gov), because the F5 mangles the Host: ...:443 header that Prometheus sends.

To rotate the password, change KSM_SCRAPE_PASSWORD and rerun both KSM jobs and the prometheus job.