Skip to content

Follow a prediction

A site service needs a prediction from one model. This is the data line: the request crosses the message bus, is validated and routed inside the inference pipeline, is answered by a model alias on Ray Serve, and comes back the same way. The bridge then records the inference, and that record is what drift detection reads later.

  1. A client sends a job to one model's address. Every model has its own UUID on the bus, set as seanerbus_uuid in its YAML. In request/reply mode — the one the stack runs — the bus routes by that UUID, not by the model name.
  2. The bridge picks the handler for that model. At start-up it registered one handler per model UUID. The handler fixes the model, picks the alias — Production unless the job asks for another — and builds the features from the model's input schema.
  3. The job enters the inference pipeline. The bridge posts it to /infer-pipeline/infer on Ray Serve over one shared HTTP client: 5 s to connect, 10 s to read, no retry at this hop.
  4. The ingress validates the payload. An embedding and a node count are required. A malformed request gets a 422 here — a schema error, which is not treated as a model failure.
  5. The feature transformer checks the embedding. It batches up to 32 requests or 50 ms to validate them together, confirms each embedding has 384 values, and keeps job and user IDs as metadata. After validation, every request is routed on its own.
  6. The model router applies the traffic split. It reads the model's Production / Canary weights from the traffic rules, cached for 30 seconds. The split applies to traffic addressed to the model's default alias (Production), which is what bus jobs send: when a split names two or more aliases, the router picks one by weighted random choice. A request pinned to any other alias, such as Staging, gets that alias. Rules are matched case-insensitively, so a split set for JPCP applies to jpcp.
  7. Ray Serve answers from its hot set. The alias resolves to a model already in memory, loaded from MLflow at start-up. Prediction runs on a pool of 4 workers with a 30 second limit.
  8. The answer returns the same way. Model name, alias, version, MLflow run and prediction travel back to the client. On any error the reply carries an error message and a prediction of 0.0, so clients must check the message.
  9. The bridge records the inference. In one background-thread hop, off its event loop: a drift snapshot of the prediction, the embedding's norm, mean and standard deviation, and an "inference served" event on the hash-chained audit trail.
  10. Repeated failures ask for a retrain. The bridge tracks each model's failure rate over its last 50 requests. At 0.5 or above it asks the control plane to retrain, then waits out a 300 second cooldown.
  11. Metrics and reloads run alongside. Ray Serve exports request counts and latency by model and alias. Every 60 seconds it asks MLflow whether an alias moved, and reloads that model without dropping traffic.
  12. Direct HTTP calls skip the bus. exa predict, the dashboard and the agent call Ray Serve's API on port 18001. They get the same models, but their requests write no drift snapshots — drift detection sees bus traffic only.

Limits and timeouts on the way

Hop Limit Where it is set
Bridge → pipeline 5 s connect, 10 s read, no retry EXAMLOPS_HTTP_*
Feature transformer batches of up to 32 requests or 50 ms; embedding of exactly 384 values inference pipeline
Model router traffic rules cached 30 s; 2 retries on transport errors TRAFFIC_RULES_TTL_SECONDS, INFERENCE_ROUTE_RETRIES
Ray Serve predict 4 workers, 30 s hard limit (504 on timeout) RAY_PREDICT_WORKERS, RAY_PREDICT_TIMEOUT
Pinned versions the 8 most recent raw versions stay loaded RAY_VERSION_CACHE_SIZE
Alias reload MLflow polled every 60 s; 0 turns polling off RAY_RELOAD_POLL_SECONDS
Bridge retrain trigger failure rate ≥ 0.5 over 50 requests, 300 s cooldown bridge

What errors mean

Reply Meaning Counts toward the retrain trigger?
422 validation error The payload does not match the model's schema No
5xx marked inference_failed, or an empty prediction The model failed while predicting Yes
Any other error Model not found, pipeline unreachable, transport failure No

Try it

exa serve check                                   # Ray Serve health and the loaded hot set
exa serve infer-check                             # one end-to-end request through the pipeline
exa serve traffic JPCP --production 90 --canary 10
exa drift status                                  # prediction drift, from the bridge's snapshots
exa drift input status                            # input-embedding drift

Read more