ExaMLOps automates the loop from drift to retraining, but some decisions stay with people.
Each one passes through a gate and releases one part of the system, and the decision is
recorded — in the hash-chained audit trail or, for approvals made in the dashboard, in the
control plane's approval table. This tour walks through them; the catalogue below lists every
human action the platform has.
A developer changes a model. CI works out which models the commit touched and files one pending approval per model with the control plane. Nothing trains yet.
An operator approves or rejects it. From the CLI (exa approvals approve JPCP) or the dashboard's Approvals page. Either can carry a reason; the dashboard asks for one on rejection.
Approval starts the training flow. The control plane dispatches the training deployment for that model. Approvals nobody answers expire after 72 hours, so a stale change cannot be approved weeks later.
Promotion waits for a metric.exa pipeline promote jpcp --if-rmse-lt 5.0 moves the alias only if the condition holds. Evaluation and judge-calibration gates can block it too; --force overrides a failing gate and is recorded.
Traffic moves in steps.exa serve traffic JPCP --production 90 --canary 10 sets the weights the model router reads, so a new version takes a slice of traffic before it takes all of it.
The autopilot acts only when switched on. Its kill-switch is off by default. exa autopilot enable lets it act and exa autopilot disable stops it; each exa autopilot run — from your own scheduler — runs one drift → retrain → promote cycle.
Autonomy levels decide whether it acts or asks. Each behaviour is AUTONOMOUS, REVIEW or DISABLED. Under REVIEW, or when a policy rule says require_approval, the autopilot does not act: it writes a human_approval_required event to the audit trail and an operator follows up with the normal commands.
A cluster is approved before any job lands on it.exa hpc connect registers a cluster as PENDING. Placement refuses it until a sysadmin runs exa hpc approve.
Agents ask before they write. Skipper pauses every mutating tool and asks; through its web UI and exa ask, the approval is single-use and expires after 10 minutes by default. The MCP server offers write tools only when started with --allow-writes, and the dashboard copilot runs a read-only agent that only proposes commands.
Decisions are recorded. Commands such as exa approvals, exa pipeline promote, exa hpc approve and the autopilot write to the hash-chained audit trail: who, what and why, each event carrying the hash of the one before it, so an edited record breaks the chain. Approvals made from the dashboard or Skipper are recorded in the control plane's approval table — the calling credential and the time — and approvals also as an outbox event; the signed-in person is not recorded there. exa audit verify walks the chain; with an external anchor configured, exa audit verify-worm checks it against that too.
What is not wired yet
The approval queue guards model changes from CI. Retrains started by drift — through
exa drift trigger, the bus bridge's error-rate tracker or the autopilot — go to the control
plane directly, governed by cooldowns, the control plane's rate limit, the corruption veto
and, for the autopilot, the autonomy levels above — not by the queue. Routing them through the queue is on the roadmap.
Read everything, run read-only commands in the CLI console, acknowledge alerts, ask the copilot
Dashboard admin
Signed in with the admin password
Every governed action: approve, promote, trigger retrains, change traffic, manage secrets, connections and projects
CLI operator
Whoever runs exa; recorded as EXAMLOPS_ACTOR or the login user
Anything the configured credentials allow; the control plane checks its token's read/write scope
Project owner, editor, viewer
Relations on a project, checked when multi-tenancy is on
Owner ⊇ editor ⊇ viewer; relations on a project apply to everything inside it
Sysadmin
By convention, the operator responsible for the HPC fleet (the dashboard requires the admin role)
Registers, approves and rejects clusters
Skipper user
Anyone chatting with the agent
Approves or denies each write the agent proposes
The dashboard's CLI console sorts every command into four
tiers: read (any signed-in user), admin, destructive (admin, and the command path must be typed
back to confirm) and CLI-only (never run from a browser).