Kubernetes Administration

How to get onto the clusters and do day-to-day work there. For what runs where, storage and limits, see Kubernetes clusters.

The clusters

	mickey	speedy
Environment	test, https://ndip-test.ornl.gov	prod, https://ndip.ornl.gov
Rancher cluster ID	c-m-dgjrkls6	c-m-fxxf8rr4
CI kubeconfig variable	KUBE_CONFIG	KUBE_CONFIG_PROD

Both are RKE2 clusters (Kubernetes 1.34) managed by Rancher on piper. Everything NDIP runs is in namespace ndip, apart from the CNPG operator (cnpg-system), kube-state-metrics (monitoring) and Rancher's monitoring and Loki (cattle-monitoring-system). mickey is shared with other projects (JupyterHub, INTERSECT, GitLab runners and others), so leave namespaces you don't own alone.

Rancher (piper)

Rancher is at https://piper01.sns.gov/.

Open a cluster from the Rancher home page to get:

• workloads, pods, events and node status;
• a pod's logs and a shell in it (the pod's ⋮ menu → View Logs / Execute Shell);
• Download KubeConfig (the file icon in the top bar) for kubectl.

kubectl

1. In Rancher, open the cluster and click Download KubeConfig.
2. Save it as ~/.kube/mickey.yaml or ~/.kube/speedy.yaml and chmod 600 it. It holds a token that acts as you.
3. Point kubectl at it:

export KUBECONFIG=~/.kube/mickey.yaml       # or speedy.yaml
kubectl -n ndip get pods -o wide

Keep one cluster per shell rather than merging both files into one KUBECONFIG, so a command meant for test can't land on prod. Each file's context is named after its cluster.

The token in a downloaded kubeconfig expires after 30 days. Once it has, every command fails with Unauthorized or system:unauthenticated: download a new kubeconfig. To see when yours expires:

kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > /tmp/piper-ca.pem
TOK=$(kubectl config view --raw -o jsonpath='{.users[0].user.token}')
curl -s --cacert /tmp/piper-ca.pem -H "Authorization: Bearer $TOK" \
  "https://piper01.sns.gov/v3/tokens/${TOK%%:*}" | jq -r .expiresAt

Everyday commands

kubectl -n ndip get pods                                  # status, restarts, age
kubectl -n ndip describe pod <pod>                        # why it isn't starting: events at the bottom
kubectl -n ndip get events --sort-by=.lastTimestamp       # recent events in the namespace
kubectl -n ndip logs deploy/galaxy -c galaxy --since=1h   # see Logging for more
kubectl -n ndip exec -it deploy/galaxy -c galaxy -- bash  # shell in a container
kubectl -n ndip rollout restart deploy/galaxy             # restart a Deployment
kubectl top nodes                                         # CPU and memory per node
kubectl get nodes                                         # Ready / NotReady / DiskPressure

Helm releases in ndip: galaxy, rucio-server, rdb-galaxy, nova-dashboard, oncat-proxy, ndip-elasticsearch, ndip-kibana. To see what was deployed when, and with which values:

helm -n ndip history galaxy
helm -n ndip get values galaxy --revision <n>

Deploy through the CI jobs (Deploying NDIP), not with helm upgrade or kubectl edit by hand: the next CI deploy overwrites hand changes, and the deployment repo stays the record of what is running. Prod jobs only run from main.

CI kubeconfigs

The Kubernetes deploy jobs in the deployment repo (code.ornl.gov/ndip/deployments/calvera) read a base64-encoded kubeconfig from the CI/CD variable KUBE_CONFIG (mickey) or KUBE_CONFIG_PROD (speedy). They decode it into the deploy container and add a hosts entry piper01.sns.gov → 10.64.194.88 so the runners can reach Rancher.

The token in it expires like any other: when it does, every Kubernetes job for that cluster fails at its first kubectl or helm call with Unauthorized. The token also belongs to whoever downloaded the kubeconfig, so CI deploys act as that person and break if their Rancher account goes away.

To rotate a CI kubeconfig:

1. Download a fresh kubeconfig for the cluster from Rancher (see kubectl) and check it works: KUBECONFIG=speedy.yaml kubectl -n ndip get pods.
2. Encode it on one line: base64 < speedy.yaml | tr -d '\n' | pbcopy (macOS; on Linux drop pbcopy and copy the output).
3. In the deployment repo, Settings → CI/CD → Variables, edit KUBE_CONFIG_PROD (or KUBE_CONFIG), paste the new value and save. Leave the variable's flags as they are.
4. Delete the downloaded file. The next Kubernetes job for that cluster uses the new token.

A downloaded kubeconfig lasts 30 days, but Rancher allows tokens of up to 90 days. To rotate less often, create an API key in Rancher (avatar menu → Account & API Keys → Create API Key, no scope, expiry 90 days) and put its bearer token in place of the token: value in the downloaded kubeconfig before encoding it.

Things that catch people out

• kubectl get cluster lists Rancher's clusters, not CNPG's: use clusters.postgresql.cnpg.io.
• speedy's ndip namespace caps each container at 2 CPU / 4 Gi and the namespace at 32 Gi of memory; pods asking for more are rejected at admission. See Limits and known constraints.
• Worker root disks are small, and a full one evicts pods (DiskPressure). Same section.