Installation & Running AOBench¶

This page is the single, canonical guide to installing AOBench and running it. There are three supported paths — pick the one that matches how you want to use the benchmark:

Paths B and C are complementary to A, not replacements: the Compose stack runs the services around the benchmark, while you still drive runs with the CLI or REST/MCP surfaces.

Requirements¶

• Python ≥ 3.10 (3.12 is used in the Docker image and CI, and is recommended).
• uv — the project's package manager/runner (recommended). Plain pip also works.
• Docker with Compose v2.20+ — only for paths B and C. Check with docker compose version.

AOBench evaluates agents against deterministic environment snapshots with mock HPC tools, so no live cluster, SLURM install, or GPU is required.

Path A — Python package (recommended)¶

AOBench is a standard Python package that installs an aobench console script.

With uv (recommended)¶

git clone https://github.com/MSKazemi/aobench.git
cd aobench

# Install the package with the dev group + all optional extras
uv sync --all-extras

With pip¶

git clone https://github.com/MSKazemi/aobench.git
cd aobench

# Editable install with the dev extras
pip install -e ".[dev]"

Optional extras¶

Install only the surfaces/adapters you need. Extras compose (list several together):

Verify the install¶

aobench --help                     # CLI is on PATH
aobench validate benchmark         # validates every task spec + environment bundle

Or via the Makefile: make install (creates .venv and installs everything).

First run¶

# Zero-tool baseline — no API key required
aobench run task --task JOB_USR_001 --env env_01 --adapter direct_qa

# A real adapter (needs a key)
export OPENAI_API_KEY=sk-…
aobench run all --adapter openai:gpt-4o --split dev
aobench report json data/runs/<run_id>
aobench clear run data/runs/<run_id>

See the CLI command reference for every subcommand, and Serve the Benchmark to drive it over REST/MCP.

Path B — Docker CLI image¶

The repository ships a Dockerfile that builds a slim image whose entrypoint is the aobench CLI. Use it for a reproducible, host-independent run.

# Build the image
docker build -t aobench:latest .

# Run any aobench subcommand — arguments after the image name are passed to aobench
docker run --rm aobench:latest --help
docker run --rm aobench:latest validate benchmark
docker run --rm aobench:latest run task \
  --task JOB_USR_001 --env env_01 --adapter direct_qa

To persist run artifacts to the host and pass API keys:

docker run --rm \
  -e OPENAI_API_KEY \
  -v "$PWD/data:/app/data" \
  aobench:latest run all --adapter openai:gpt-4o --split dev

A one-shot build + entrypoint smoke test is wired into the Makefile:

make repro-docker

Path C — Docker Compose stack (Langfuse + leaderboard)¶

The root compose.yml brings up the service stack around the benchmark — Langfuse for observability and the leaderboard HTTP API — via Compose's include: (requires Compose v2.20+).

# Start the full stack (Langfuse UI + leaderboard API)
make stack-up

# Stream logs / stop (volumes are preserved on stop)
make stack-logs
make stack-down

make stack-up runs langfuse-setup first, which writes Langfuse keys into .env idempotently. To run only one service:

make langfuse-up          # Langfuse only  → http://localhost:3000
make leaderboard-serve    # leaderboard API only (needs the `leaderboard` extra)

Once Langfuse is up, export traces from any run with the --langfuse flag (e.g. make run-langfuse). See the Langfuse integration guide for the full stack layout and troubleshooting.

Environment variables¶

Copy the example file and fill in what you need:

cp .env.example .env

Key variables (see .env.example for the full list):

Where to go next¶

• Serve the Benchmark — run the engine over REST & MCP.
• Programmatic access — call the BenchmarkService façade directly.
• CLI command reference — every subcommand and flag.
• Adapters & tools — how adapters and mock tools fit together.