Skip to main content

Logging

Pod logs (kubectl)​

Every NDIP service on mickey and speedy logs to stdout/stderr. The quickest way to read logs is kubectl (or the Rancher UI):

kubectl -n ndip logs deploy/galaxy -c galaxy --since=1h          # Galaxy (gunicorn, Celery, job handlers)
kubectl -n ndip logs deploy/galaxy -c gx-it-proxy                # interactive tools proxy
kubectl -n ndip logs deploy/galaxy -c galaxy --previous          # the container before its last restart (after an OOM kill)
kubectl -n ndip logs deploy/galaxy -c tools-sync                 # tool clone at pod start
kubectl -n ndip logs deploy/rucio-server -c rucio-server
kubectl -n ndip logs galaxy-rabbitmq-0
kubectl -n ndip logs galaxy-pg-1                                 # Postgres (JSON lines, from CNPG)
kubectl -n ndip logs deploy/rdb-galaxy

kubectl logs only covers the current and the previous container. After a pod is deleted or rescheduled, its logs are gone from the node.

Useful Galaxy greps:

  • Mapped job to destination id: where the job router sent a job;
  • cannot get destination status / no resources available: jobs deferred by the router (Prometheus unreachable, or no healthy host with free resources).

Rancher Logging and Loki​

Both clusters run Rancher Logging, which ships all pod logs to Loki in namespace cattle-monitoring-system. Current limitations:

  • Grafana has no Loki data source yet, so the logs can't be browsed there (Rancher's Grafana only has a dashboard sidecar, so the data source has to be added in the rancher-monitoring app values, grafana.additionalDataSources).
  • mickey's Loki is down.

Loki holds history that has already rotated off the nodes (and is older than ELK on prod). Query it with a port-forward, for example to save Galaxy's log for a time window:

kubectl -n cattle-monitoring-system port-forward svc/loki 3100 &
curl -sG localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={namespace="ndip",container="galaxy"}' \
  --data-urlencode 'start=2026-10-07T03:14:00Z' --data-urlencode 'end=2026-10-07T03:20:00Z' \
  --data-urlencode 'limit=5000' --data-urlencode 'direction=forward' | jq -r '.data.result[].values[][1]'

Each line is a JSON record; the log text is in its message field. Results are capped at 5000 lines per request, so page through longer windows by moving start past the last timestamp.

ELK​

calvera's k8s/elk/ deploys Elasticsearch, Kibana and Filebeat into namespace ndip on both clusters. Deploy with CI job elk-test (mickey) or elk-k8s-prod (speedy); k8s/deploy-all.sh also runs it in both environments.

test (mickey)prod (speedy)
Elasticsearch storagehostPath PV on mickey-wn01 (elasticsearch-pv.yaml, class manual)50Gi Longhorn volume, no node pin
Valueselasticsearch-values.yaml + elasticsearch-values-test.yamlelasticsearch-values.yaml + elasticsearch-values-prod.yaml
Kibanahttps://ndip-test.ornl.gov/logs, via the Galaxy chart's kibana Ingress (no auth)https://ndip.ornl.gov/logs, kibana-ingress.yaml, basic auth (user ndip)
  • Filebeat (filebeat-k8s.yaml, image elastic/filebeat:8.12.1, same version as Elasticsearch and Kibana) is a DaemonSet on the worker nodes. It tails the containers of every pod in namespace ndip and writes to daily data streams ndip-k8s-YYYY.MM.dd, tagged service.name: k8s and environment: test|prod, with the Kubernetes fields (kubernetes.namespace, kubernetes.pod.name, kubernetes.container.name). Pulsar hosts ship their own logs with the filebeat pulsar Ansible job (ndip-pulsar-*); they still point at mickey's Elasticsearch.
  • Retention: es-retention-cronjob.yaml deletes ndip-* data streams older than 30 days, daily at 03:00.
  • Kibana password (prod): set the masked CI variable KIBANA_PASSWORD and run elk-k8s-prod; the job hashes it into Secret kibana-basic-auth. Without the variable the existing Secret is kept.
  • Elasticsearch is also exposed on NodePort 31750 (no auth, internal network) for the Pulsar-host Filebeats.
  • The files in k8s/elk/ are rendered per environment by k8s/lib/env.sh (@NDIP_HOST@, @ENV_NAME@).
  • Quota: ELK adds about 3.8Gi of memory limits on speedy, where the ndip quota is 32Gi.
  • Unauthenticated routes: Kibana's Service is a NodePort (31550) and Elasticsearch's too (31750). Both are reachable without the password from the internal network. Test's /logs has no auth.

Kibana views​

The ELK deploy imports a standard set of data views and saved searches from k8s/elk/kibana-objects.ndjson (Kibana saved objects API, overwrite=true). They have fixed IDs, so every deploy restores them without touching objects people made by hand. The deploy sets "NDIP pods (k8s)" as the default data view only if none is set, and sets Kibana's defaultColumns to kubernetes.container.name and message, so Discover shows timestamp, source and message instead of the whole document. The saved searches use the same columns (host.name and message for Pulsar). Add any other field from the field list on the left when you need it.

data viewssaved searches (Discover → Open)
NDIP pods (k8s) ndip-k8s-*, Pulsar hosts ndip-pulsar-*, Galaxy VM (legacy) ndip-galaxy-*Galaxy, Galaxy errors, Galaxy job routing, Rucio server, RabbitMQ, Postgres (CNPG), RDB, All pods: errors, Pulsar, Pulsar errors

To change them: edit them in Kibana, export them (Stack Management → Saved objects → Export), and replace the matching lines in kibana-objects.ndjson, or edit the file directly. To load them without a full deploy:

kubectl -n ndip exec -i deploy/ndip-kibana -- curl -s -X POST 'localhost:5601/logs/api/saved_objects/_import?overwrite=true' \
  -H 'kbn-xsrf: true' -F 'file=@-;filename=kibana-objects.ndjson' < k8s/elk/kibana-objects.ndjson

Troubleshooting​

  • Nothing arrives, Filebeat logs Cannot index event (status=400): dropping event!: Elasticsearch is rejecting every event, almost always a field whose type clashes with the index template. Filebeat loads its ECS template for ndip-k8s-*, so custom fields must fit it: service is an object (use service.name, not service). Check the mapping with GET ndip-k8s-*/_mapping/field/<field>. Rejected events are dropped, not retried.
  • Count documents and see what's shipping:kubectl -n ndip exec ndip-elasticsearch-master-0 -c elasticsearch -- curl -s localhost:9200/ndip-k8s-*/_count
  • Filebeat pods in ImagePullBackOff: the image must be elastic/filebeat:<full version>; there is no beats/filebeat repository and no short 8.12 tag.

Backfilling​

Filebeat records how far it has read in each container log file (its registry, in /var/lib/filebeat-data on each node) and moves past events Elasticsearch rejected. To ship logs again:

  1. Delete the data streams that would otherwise be duplicated: curl -X DELETE localhost:9200/_data_stream/ndip-k8s-* (from the Elasticsearch pod).
  2. Give Filebeat a fresh registry: set filebeat.registry.path in the ConfigMap to a new directory (e.g. /usr/share/filebeat/data/registry-v2) and kubectl -n ndip rollout restart ds/filebeat. It then re-reads every container log file still on the nodes from the start.

This only reaches as far back as the kubelet keeps container logs on the nodes (a few rotated files per container: hours for Galaxy, longer for quiet pods; nothing for deleted pods). Older history is in Loki (above).

Pulsar hosts and VMs​

  • Pulsar: /var/log/pulsar.log (prod) and /var/log/pulsar_test.log (test); RDB in /var/log/rdb.log. Fluentd is off on the Pulsar hosts (fluentd_install: false). Filebeat is deployed to some hosts by the filebeat pulsar job.
  • Old prod VM (proxy): nginx logs in /var/log/nginx/. The stream proxy logs only errors by default.