Skip to content

Follow a cluster job

This is the compute line. ExaMLOps runs on the schedulers HPC centres already operate — Slurm, Flux, or none at all in mock mode — through one scheduler-neutral adapter. A fleet layer on top decides which clusters the platform may use at all, which one a job should go to, and what each job cost.

  1. Discover what a cluster offers. exa hpc detect login.example.org probes the scheduler and GPUs over SSH, read-only, and prints a suggested configuration. exa hpc nodes --host login.example.org --save --cluster cluster-a stores that cluster's node inventory for placement.
  2. Register it — as PENDING. exa hpc connect records the connection and a fingerprint of the SSH client key given with --key. A new cluster starts PENDING: nothing can be scheduled on it yet.
  3. A sysadmin approves it. exa hpc approve cluster-a (or the dashboard's Facility console) makes it ACTIVE and records who approved it. Re-probing later never revokes that approval.
  4. Placement picks the best approved cluster. Among ACTIVE clusters that can fit the request, the default score prefers the most idle GPUs, then idle nodes. The scoring function is a swappable provider.
  5. Preflight checks it. exa hpc preflight cluster-a --gpus 2 runs discovery checks over SSH and exits 1 on any failure — useful as a CI gate before a long run.
  6. The pipeline runs on that cluster. exa pipeline run --model JPCP --dataset PM100Dataset --cluster auto asks placement for a cluster, refuses anything that is not ACTIVE, and hands the flow the scheduler and SSH settings.
  7. The adapter submits and waits. sbatch on Slurm or flux batch on Flux, then polling every 10 seconds. Every scheduler command is capped at 30 seconds; the wait gives up after 24 hours or five unknown states in a row.
  8. The job is recorded. Job id, scheduler, flow run, model, dataset, nodes, GPUs and CPUs, then state, start, end and exit code. The MLflow run is tagged with the job id, linking the model version to its compute.
  9. GPU-hours become cost. exa hpc capacity reports utilisation and GPU-hours; exa models cost jpcp --record reads the scheduler's accounting for each version and prices it.
  10. …and carbon. exa finops carbon record converts GPU- and CPU-hours to energy and CO₂e with a swappable provider — grid intensity, data-centre PUE and GPU power — and records which provider produced the estimate.
  11. The fleet is monitored. exa hpc prometheus-sd --out platform/infra/docker-compose/targets/fleet.json writes Prometheus targets — node and GPU exporters from the saved node inventories, vLLM from the LLM endpoint registry — into the directory Prometheus reads, and Prometheus re-reads it every 30 seconds.

Mock mode

With EXAMLOPS_HPC_SCHEDULER=mock (the default) training runs inline on the machine that runs the flow, so no job record is written and exa models cost reports illustrative placeholder figures. Set slurm or flux to schedule real jobs.

Scheduler adapters at a glance

Mock Slurm Flux
Submit Trains inline sbatch flux batch
Poll Completes immediately squeue, then sacct flux jobs / event log
Transport Local Local or SSH (host-key checked) Local or SSH (host-key checked)
Job record None hpc_jobs hpc_jobs
Select with EXAMLOPS_HPC_SCHEDULER=mock …=slurm …=flux

Try it

exa hpc detect login.example.org
exa hpc connect login.example.org --name cluster-a --user me --key ~/.ssh/id_ed25519
exa hpc clusters
exa hpc approve cluster-a
exa hpc place --gpus 2
exa hpc preflight cluster-a --gpus 2
exa pipeline run --model JPCP --dataset PM100Dataset --cluster auto
exa hpc jobs
exa hpc capacity

Read more