Skip to main content

Dynamic Tool Destinations

Galaxy needs to know where (on which resources) to run tools. This is defined in job_conf.yml, rendered from calvera's k8s/galaxy/values-jobconf-<env>.yml. There you can:

  • Manually define a destination for a specific tool (the tools: list), or
  • Use dynamic destinations that apply rules to determine the final destination.

The list of available destinations is here.

By default, tools use the dynamic_resource_orchestration destination (execution.default). Its rule and data are both in k8s/galaxy/values-resource-orchestration.yml:

  • resourceOrchestration becomes resource_orchestration.yaml (tools, machines, destinations);
  • resourceOrchestrationRules is the Python rule. The chart mounts it into lib/galaxy/jobs/rules/.
danger

Don't add a static tools: mapping for a tool that should be orchestrated. A static mapping in job_conf takes precedence over execution.default, so the rule is silently bypassed. Only uploads (upload1, __DATA_FETCH__), the SNS register/export tools and a few HPC tools are mapped statically.

How the rule decides​

For each job, the rule:

  1. Sends internal tools (__*, upload1, tools under lib/galaxy/tools) to local.
  2. Reads the firing alerts from Prometheus (http://10.64.193.81:32001/api/v1/alerts). Any alert with label env: prod, or the cluster's own env, and a host label naming a Pulsar host removes that host's destinations. Hosts whose names don't resolve are removed too. If Prometheus can't be reached, the job waits.
  3. Subtracts the CPU, memory and GPU of queued and running jobs from each machine.
  4. Picks the first allowed destination with enough free resources and matching features. If none fits, it raises JobNotReadyException and Galaxy retries later. The job stays new/queued.

Verify routing from the Galaxy log rather than from whether a job succeeds:

kubectl -n ndip logs deploy/galaxy -c galaxy | grep 'Mapped job to destination id'

The rest of this page describes the three sections of resourceOrchestration.

Tools​

  • Defines resource requirements such as CPU, memory, and GPU.
  • Restricts where a tool may run with allowed_destinations_and_groups (destinations or groups such as shared, imaginex). The default entry applies to tools that aren't listed.
  • May require features (features:) that only some destinations provide.

Machines​

  • Defines the physical nodes available for jobs.
  • Each machine specifies the number of CPU cores, memory (GB) and optional GPUs.

Destinations​

  • Maps each orchestrated destination to a machine, with optional groups and features.
  • Destination names must match destinations defined in values-jobconf-<env>.yml.

Examples​

Adding a tool to Galaxy​

You don't have to add a new tool to the job configuration. By default, Galaxy uses dynamic_resource_orchestration with the default tool entry (1 CPU, 1 GB, pulsar_sns_node01 or pulsar_sns_gpu_node01).

To give a tool other resources or restrict where it runs, add it to the tools section of resourceOrchestration:

      example_tool_needing_a_gpu:
        gpu: 0.1
      example_tool_for_imaginex_only:
        allowed_destinations_and_groups:
        - imaginex

Adding a new machine​

Add it to machines, add a matching destination to destinations, and define that destination (and its Pulsar runner) in both values-jobconf-*.yml files:

    machines:
      sns_gpu_node01:
        cpu: 64
        memory: 1024
        gpu: 4
    destinations:
      pulsar_sns_gpu_node01:
        machine: sns_gpu_node01
        groups:
        - shared

To include the machine in the alert-based health check, add its hostname to destination_map in resourceOrchestrationRules, and make sure Prometheus scrapes it with a matching host label (see Adding a Pulsar instance).

Deploy with galaxy-test, then galaxy-k8s-prod.

Legacy

On the VM stack, these files were ansible/files/galaxy/config/resource_orchestration.yaml, ansible/files/galaxy/dynamic_job_rules/ and ansible/templates/galaxy/config/job_conf.yml.j2. They still exist in calvera for galaxy.ornl.gov.