Skip to content

Follow an LLM request

Large language models run on the same platform as the rest of ExaMLOps, with one extra piece in front of them: the model gateway. A request through it can be checked against a virtual key and its budget, is scanned by a guardrail, can be answered from a cache, and is routed to a model server and costed. The model servers themselves are vLLM processes that exa serve llm starts on Compose or a Slurm allocation, or registers when someone else runs them; for Kubernetes it prepares and validates a KServe manifest that you apply.

The gateway is a library that runs inside the calling process. There is no separate gateway service to deploy; whatever calls it — an exa command or your own code — applies the same rules.

A request through the gateway

  1. Four callers use the gateway. exa gateway chat, the answer step of exa rag query, the judge in exa serve challenger judge, and Python code that builds a GatewayClient. Only callers that pass a virtual key get the key and budget check; the RAG and judge calls do not pass one. The Skipper agent calls its own configured model backend directly — Ollama when no hosted backend is set — and does not go through the gateway.
  2. The virtual key is checked first. Before any model is called: an unknown or revoked key, a model missing from the key's allow-list, or spend already at the budget each stop the request with a typed error. exa gateway key issue prints a key once; only its hash is stored, and issuing and revoking are audited. A request with no key skips this check.
  3. A caller can name a registry prompt. With prompt_ref="support-bot@prod" the gateway prepends that prompt version as a system message and records the version on the trace, so changing the prompt is a label move, not a redeploy. A reference that does not resolve is an error. The template is trusted, versioned text, so the guardrail scans only the caller's own messages.
  4. The request is scanned. In the default monitor mode the guardrail records what it finds in each message's text — prompt injection, personal data, secrets — and changes nothing. That includes the text parts of a message sent as a list of content parts, such as an image with a question; image parts are left to the media guard. In enforce mode it blocks injection and redacts personal data and secrets. A blocked request never reaches the cache or a model. Set the mode with EXAMLOPS_GUARDRAIL_MODE.
  5. A close match can be answered from the cache. When the cache is on, a request whose last message embeds at least 0.85 similar to a stored one — for the same model and the tenant the cache was set up for, stored within the last hour — gets the stored answer without a model call. The system prompt, earlier turns and the caller's sampling settings are not part of the match, and a request containing content parts never reads or writes the cache. The cache lives in the calling process, so a single exa gateway chat --cache call starts empty; a long-running caller benefits.
  6. The router picks a model. Each route name maps to backends in priority order, and an error moves to the next. The default table holds an echo route named default and one route for every endpoint registered with exa serve llm start, under the endpoint's name. No echo stands in for a down endpoint: the request fails and says which endpoint failed.
  7. Images are checked, then the server answers. Image parts are validated before the request leaves — the per-prompt image limit, the domain allow-list, the local-path root, and a size cap (20 MiB by default) on inline and local images — and a rejected image is never retried elsewhere. The vLLM server receives the chat request under the name of the weights it serves.
  8. The call is costed. Cost comes from the llm_cost provider when an operator has selected one, else the built-in rate table, else a flat rate. Each call writes a gateway_calls row and adds its cost to the key's spend, which the next budget check reads. With tracing on, a GenAI span carries the model, tenant, tokens and a rate-table cost estimate to Tempo.
  9. The answer is scanned on the way out. In enforce mode toxic output is blocked and personal data and secrets are redacted. The cost was recorded first on purpose: the tokens were spent whatever the guardrail decides, so the bill matches the provider's.
  10. A schema, if asked for, is enforced. With response_schema the gateway finds the JSON in the reply, validates it, and repairs it once by default — locally, by dropping unknown fields, coercing types and filling missing required fields with empty values, not by asking the model again. If it still does not validate, the caller gets StructuredOutputError. This validates after generation; it does not constrain the model while it generates. A cache hit is validated the same way; a stored answer that does not fit the schema counts as a miss, and the model is asked.
  11. The answer is stored and returned. A blocked or schema-invalid answer is never cached. The caller receives the text, the backend that answered — endpoint:qwen, echo or cache — the token counts and the cost.

What can stop a request

Stage Stops the request when Error
Virtual key The key is unknown or revoked · the model is not on its allow-list · spend has reached the budget KeyInvalid · ModelNotAllowed · BudgetExceeded
Registry prompt The named prompt or label does not exist the registry's lookup error
Guardrail, inbound enforce mode finds prompt injection, or the scanner itself fails GuardrailBlocked
Media guard An image breaks the endpoint's limit, allow-list, path root or size cap MediaNotAllowed
Router Every backend for the route failed, or no route has that name AllBackendsFailed
Guardrail, outbound enforce mode finds toxic output GuardrailBlocked
Structured output The reply cannot be made to fit the schema StructuredOutputError

exa gateway chat prints the error name and reason. exa rag query does not surface them: its answer step reports "(no answer)" instead.

Run a model server

  1. An operator starts an endpoint. exa serve llm start qwen previews with --dry-run, asks for confirmation and writes an audit event. The model's engine: block, overridden by flags such as --tp and --max-model-len, is rendered into vllm serve flags by one function that the Compose service, the Slurm job and the KServe manifest all use. A vision model must set --max-images.
  2. One of four launchers takes it. external (the default) registers a server someone else runs. compose starts the GPU vllm service. kserve builds a Kubernetes manifest and checks it with a server-side dry run, but never applies it — you apply it with kubectl. slurm and flux submit a batch job through the scheduler the launcher is named after; the job script uses Slurm's srun and scontrol, so today the server only starts under Slurm.
  3. A vLLM server comes up. On a Slurm allocation, Apptainer runs the vLLM image; across several nodes a Ray cluster forms, and --tp and --pp set tensor parallelism inside a node and pipeline parallelism across nodes. The job is recorded in the HPC job table as a serving job, so exa hpc jobs lists it.
  4. The endpoint is recorded. The registry keeps its address, state, launcher, job id, project, modality and engine block. An external endpoint is READY at once; Compose and HPC endpoints start as STARTING while the model loads, and a KServe endpoint starts as PENDING.
  5. An HPC job publishes its own address. The job writes its URL to an endpoint file as soon as its head node is known; starting and stopping the endpoint remove any file a previous job left. exa serve llm health, status and chat read that file, or fetch it over SSH, and record the address; a gateway request picks it up only when the file is visible on the machine it runs on. The work directory must be on a filesystem the compute nodes share with the login node.
  6. A health probe marks it ready. exa serve llm health qwen probes /health, lists the models the server serves, and records READY or FAILED; it exits 1 when the endpoint is not ready, so it works as a deploy gate.
  7. The endpoint is now a gateway route and a metrics target. The gateway routes to it by name, with keys, budgets, guardrails, caching and cost applied. Prometheus scrapes the Compose server directly. exa hpc prometheus-sd writes a target for every other endpoint that is READY or STARTING with a recorded, non-loopback address — re-run it when that changes. Alerts cover a down endpoint, a nearly full KV cache, a queue backlog and slow first tokens.

Launchers at a glance

External Compose Slurm / Flux KServe
Starts Nothing — registers a URL The vllm service (GPU profile) A batch job: Apptainer, Ray across nodes (the script needs Slurm's srun) Nothing — builds and dry-runs an LLMInferenceService
Address The --base-url you give http://localhost:18011 Published by the job, read on first use EXAMLOPS_KSERVE_GATEWAY_URL
State after start READY STARTING STARTING PENDING
exa serve llm stop Marks it STOPPED; nothing is killed docker compose stop vllm Cancels the job (scancel / flux cancel) Refuses: delete it with kubectl
Works without a GPU Yes No No Manifest only

Answer from your documents

  1. Documents are chunked. exa rag ingest runbooks --docs runbooks.jsonl reads one JSON object per line, each with an id and its text, and splits the text into 40-word windows that overlap by 10 words, so a phrase of up to ten words cut at a boundary is whole in the next window.
  2. Chunks are embedded and stored. By default the embedding is a deterministic 64-dimension token hash, so ingest needs no model and no service. Chunks go into the vector store — tables in the platform datastore by default, or pgvector — and the knowledge base records its source revision, its encoder and its chunk count.
  3. A question is embedded the same way. exa rag query runbooks --question "How do I drain a node?" embeds the question with the same function used at ingest, so the question and the chunks are compared in one space. The pipeline's encoder is checked against the one stamped on the knowledge base, and a mismatch is refused rather than scored.
  4. The nearest chunks are retrieved. The store returns up to three times as many candidates as the answer will use, for the reranker to choose from.
  5. They are reranked. Candidates are ordered by how many of the question's words they contain, and the top k (5 by default) are kept. With tracing on, a retriever span records the ids and scores of the chunks kept.
  6. Retrieved text is treated as untrusted. A chunk that says "ignore previous instructions" is an attack on the model, not a fact about your system. Such phrases are replaced with [redacted-instruction], the chunk stays in the context as data, and the answer is flagged.
  7. The prompt goes through the gateway. The chunks are numbered into a prompt that asks for an answer from the context alone, with chunk citations, and sent to the gateway's default route — the request path above, without a virtual key. The echo route answers unless a model endpoint is registered as default — and an echo returns the whole assembled prompt as the answer. If the gateway call fails for any reason, the answer reads "(no answer)" rather than showing the error. The citations list the k chunks retrieved, whether or not the answer used them.

Not built yet

The gateway validates structured output after generation; constraining generation to a schema is on the roadmap. The semantic cache and the default RAG embedding are in-process fallbacks — the cache does not persist between processes, and the token-hash embedding matches shared words rather than meaning. The SGLang engine is a stub until a GPU host exists, and no real GPU has run this path yet: it is tested against a stub vLLM server over real sockets. The HPC job script is Slurm-only, and KServe endpoints are never applied for you.

Try it

# Register a server someone else runs, check it, and route through the gateway
exa serve llm start qwen --base-url http://gpu01:8000 --hf-model Qwen/Qwen3-8B
exa serve llm health qwen
KEY=$(exa --json gateway key issue --tenant acme --project chat --model qwen --budget 20 \
      | jq -r .virtual_key)
exa gateway chat qwen --message "Summarise this alert" --key "$KEY"
exa gateway key list

# Or launch one on a cluster (preview first)
exa serve llm start qwen --launcher slurm --nodes 2 --gpus 4 --tp 4 --pp 2 \
    --hf-model Qwen/Qwen3-8B --dry-run

# Retrieval-augmented answers
exa rag ingest runbooks --docs runbooks.jsonl
exa rag query runbooks --question "How do I drain a node?"
exa guardrails stats

Read more