Skip to content

Follow the signals

This is the signal line. Two kinds of signal leave the platform. Telemetry tells you how it is running: metrics, alerts, logs and traces. Evidence tells you what it did and who decided: the hash-chained audit trail, its signed checkpoints, and the events other systems subscribe to.

  1. Prometheus scrapes the serving, control and bridge services. Every 15 seconds: Ray Serve (requests, latency, loaded versions), the control plane (retrains, Prefect retries, circuit-breaker opens, approvals), the bus bridge, and fleet exporters. Metrics are kept for 7 days.
  2. Rules turn metrics into alerts. 31 rules — for example Ray Serve errors above 5% for 10 minutes (warning) or 20% for 5 (critical), p99 latency, no models loaded, a bridge down, SLO burn. Alertmanager groups them and pages on critical ones.
  3. Logs flow without code changes. Promtail tails the output of every container in the ExaMLOps compose project through the Docker API and pushes it to Loki, labelled by compose service and container. Logs are kept for 7 days.
  4. Traces go to Tempo — when you turn them on. The control plane and dashboard are auto-instrumented and the inference pipeline adds its own spans. Tracing is off until OTEL_SDK_DISABLED=false. The inference pipeline then samples 5% of traces; the auto-instrumented services keep every trace unless OTEL_TRACES_SAMPLER says otherwise.
  5. Grafana reads all three. Seven provisioned dashboards — overview, online metrics, control plane, drift, approvals, logs and bus — and it can be embedded in the ExaMLOps dashboard.
  6. Every governed action is appended to the audit trail. Retrains, approvals, promotions, cluster admissions, agent writes and each inference served over the bus. Every event's hash covers the one before it, so a changed record breaks the chain.
  7. Checkpoints are signed and anchored outside the database. exa audit checkpoint signs the chain head and — when EXAMLOPS_AUDIT_WORM_PATH is set — appends it to an append-only anchor file; exa audit verify-worm checks the chain against that anchor.
  8. Events leave through an outbox. The control plane writes an event in the same transaction as the change it describes. A relay inside the control plane publishes them every second — to the log by default, or to Redis Streams — at least once, with stable ids.

Drift is watched by the drift trigger, not by an alert rule

None of the 31 alert rules looks at the drift z-score. Drift is measured from the bridge's snapshots by exa drift status and acted on by exa drift trigger or the autopilot — see Follow a retrain.

Where each signal lives

Signal Store Kept for Read it with
Metrics Prometheus 7 days Grafana, dashboard consoles
Alerts Alertmanager until resolved paging and chat receivers, dashboard Alerts
Logs Loki 7 days Grafana, dashboard service log tail
Traces Tempo 48 hours Grafana (Tempo data source)
Audit trail audit_events in the platform datastore not touched by exa data retention-prune exa audit, exa audit verify
Checkpoints audit_checkpoints, plus the anchor file when EXAMLOPS_AUDIT_WORM_PATH is set permanent exa audit verify-worm
Events event_outbox kept; published rows are stamped, failed ones stay as evidence exa events stats, exa events relay

Try it

make monitoring-up          # Prometheus, Alertmanager, Grafana, Loki, Promtail, Tempo
make alerts-check           # validate the alert rules with promtool
exa audit --last 7d --model JPCP
exa audit verify
exa audit checkpoint && exa audit verify-worm
exa events stats

Read more