Installation & Running AOBench¶

This page is the single, canonical guide to installing AOBench and running it.

git clone https://github.com/MSKazemi/aobench.git && cd aobench && make install
aobench quickstart

There are three supported paths — pick the one that matches how you want to use the benchmark:

Paths B and C are complementary to A, not replacements: the Compose stack runs the services around the benchmark, while you still drive runs with the CLI or REST/MCP surfaces.

Requirements¶

• Python ≥ 3.10 (3.12 is used in the Docker image and CI, and is recommended).
• uv — the project's package manager/runner (recommended). Plain pip also works.
• Docker with Compose v2.20+ — only for paths B and C. Check with docker compose version.

AOBench evaluates agents against deterministic environment snapshots with mock HPC tools, so no live cluster, SLURM install, or GPU is required.

Path A — Python package (recommended)¶

AOBench is a standard Python package that installs an aobench console script.

With uv (recommended)¶

git clone https://github.com/MSKazemi/aobench.git
cd aobench

# Install the package with the dev group + all optional extras
uv sync --all-extras

With pip¶

git clone https://github.com/MSKazemi/aobench.git
cd aobench

# Editable install with the dev extras
pip install -e ".[dev]"

Optional extras¶

Install only the surfaces/adapters you need. Extras compose (list several together):

Choosing an install target¶

Verify the install¶

aobench --version                  # CLI is on PATH
aobench doctor                     # Python, corpus, extras, credentials — with fixes

aobench doctor exits non-zero only when a required check fails. Missing optional extras and absent API keys are reported as warnings, because the direct_qa baseline genuinely needs neither. aobench info --json prints the same picture as a JSON blob — paste that into a bug report.

If the aobench console script is not on your PATH, python -m aobench is equivalent.

First run¶

# The whole thing in one command — no arguments, no API key, no network
aobench quickstart

# The same run, spelled out
aobench run task --task JOB_USR_001 --env env_01 --adapter direct_qa

# A real adapter (needs a key)
export OPENAI_API_KEY=sk-…
aobench run all --adapter openai:gpt-4o --split dev
aobench report json data/runs/<run_id>
aobench clear run data/runs/<run_id>

See the Quickstart for a narrated walkthrough of that first run and how to read the seven-dimension scorecard.

Where the benchmark corpus comes from¶

Every command needs the corpus (task specs, environment snapshots, scoring configs). AOBench finds it automatically, in this order:

1. $AOBENCH_BENCHMARK_ROOT, if set — an explicit override.
2. A benchmark/ directory found by walking up from the current directory, so a source checkout works from any subdirectory.
3. The copy bundled inside the installed package — so a wheel install works from any directory at all, with no checkout present.

Every command also accepts --benchmark /path/to/benchmark to override it for a single invocation. If none of the three resolve, the error names all the locations tried.

See the CLI command reference for every subcommand, and Serve the Benchmark to drive it over REST/MCP.

Path B — Docker CLI image¶

The repository ships a Dockerfile that builds a slim image whose entrypoint is the aobench CLI. Use it for a reproducible, host-independent run.

# Build the image
docker build -t aobench:latest .

# Run any aobench subcommand — arguments after the image name are passed to aobench
docker run --rm aobench:latest --help
docker run --rm aobench:latest validate benchmark
docker run --rm aobench:latest run task \
  --task JOB_USR_001 --env env_01 --adapter direct_qa

To persist run artifacts to the host and pass API keys:

docker run --rm \
  -e OPENAI_API_KEY \
  -v "$PWD/data:/app/data" \
  aobench:latest run all --adapter openai:gpt-4o --split dev

A one-shot build + entrypoint smoke test is wired into the Makefile:

make repro-docker

Path C — Docker Compose stack (Langfuse + leaderboard)¶

The root compose.yml brings up the service stack around the benchmark — Langfuse for observability and the leaderboard HTTP API — via Compose's include: (requires Compose v2.20+).

# Start the full stack (Langfuse UI + leaderboard API)
make stack-up

# Stream logs / stop (volumes are preserved on stop)
make stack-logs
make stack-down

make stack-up runs langfuse-setup first, which writes Langfuse keys into .env idempotently. To run only one service:

make langfuse-up          # Langfuse only  → http://localhost:3000
make leaderboard-serve    # leaderboard API only (needs the `leaderboard` extra)

Once Langfuse is up, export traces from any run with the --langfuse flag (e.g. make run-langfuse). See the Langfuse integration guide for the full stack layout and troubleshooting.

Environment variables¶

Copy the example file and fill in what you need:

cp .env.example .env

Key variables (see .env.example for the full list):

Where to go next¶

• Serve the Benchmark — run the engine over REST & MCP.
• Programmatic access — call the BenchmarkService façade directly.
• CLI command reference — every subcommand and flag.
• Adapters & tools — how adapters and mock tools fit together.