Leaderboard¶

A leaderboard is only as good as the reproducibility of its rows. Every entry here names its AOBench version, split, scoring profile, and exact model snapshot, so anyone can re-derive it. Entries that cannot be re-derived do not go on the board.

Reference baselines¶

These ship with the benchmark and anchor the scale. Reproduce them yourself with the commands shown.

direct_qa calls no tools and answers from the prompt alone. It exists to give the scale a floor: any tool-using agent that does not clearly beat it is not using tools usefully. A score below the floor generally means the agent is calling tools badly rather than not at all.

Community results¶

Independent runs on independent hardware, submitted through #27. Each row is the submitter's own measurement, published as submitted, with the submitter's run_summary.json attached to the linked issue so anyone can re-derive it.

Scoring profile is per task: 42 × alpha1_grounding, 25 × default_hpc_v01. The three runs scored 0.6132 / 0.6192 / 0.6335 — a spread of 0.020, which is what a well-behaved result looks like. Per-dimension means across the three runs:

robustness and workflow are unset in these runs, and that is expected rather than a gap: robustness is only produced by aobench robustness task, which re-runs a single task N times.

Submitting a result¶

1. Run it¶

aobench run all --adapter <your adapter> --split dev
aobench clear run data/runs/<run_id>
aobench report json data/runs/<run_id>

aobench report json writes data/runs/<run_id>/run_summary.json and prints a short human summary to the terminal. It does not emit JSON on stdout, so do not redirect it — attach run_summary.json itself.

Use --split dev. Results on the locked test split are accepted only from maintainers or by prior arrangement, because a public test-split leaderboard is a training target within a year.

2. Check it meets the bar¶

A submission must state:

And it must be re-derivable by someone else: the adapter has to be either one that ships with AOBench, a public MCP server, or a documented endpoint.

The profile is per task, not per run¶

Each task spec names its own aggregate_weight_profile, so a full-split run mixes them: on the current corpus, 60 tasks score under alpha1_grounding and 28 under default_hpc_v01. A dev-split run therefore has no single profile, and that is by design — alpha1_grounding puts weight on grounding and tool use for tasks where gold evidence exists, which is not meaningful for every task.

Write corpus default in that field unless you deliberately forced one profile across every task, in which case say which and note that the number is not comparable to rows that used the corpus default.

Confirm what your own run used with:

jq -r '.tasks[].weight_profile_name' data/runs/<run_id>/run_summary.json | sort | uniq -c

Where to find the per-dimension scores¶

aobench report json writes mean_dimension_scores into the run summary — the run-level mean for each of the seven weighted dimensions. Paste that object straight into the submission:

jq .mean_dimension_scores data/runs/<run_id>/run_summary.json

{
  "outcome": 0.7124, "tool_use": 0.6810, "grounding": 0.5903,
  "governance": 0.9500, "robustness": 0.4417, "efficiency": 0.8302,
  "workflow": 0.6188
}

{"outcome": 0.24, "tool_use": 0.0, "grounding": 0.0, "governance": 1.0,
 "robustness": null, "efficiency": 1.0, "workflow": null}

The same seven appear per task under .tasks[], which is what to look at when one dimension drags the aggregate down and you want to know which tasks did it. Each task row also carries its own weight_profile_name, and .weight_profiles gives the run-level tally, so the weights behind a row can always be recovered from the file alone.

Reporting more than one run¶

The bar is three independent runs, and the form has one score field, so report the mean across runs as the headline number and give the spread alongside it:

Attach every run's run_summary.json, not just the one you took the headline from. The spread is the part that says whether a gap between two agents is real, so a submission with three attached runs is worth more to the board than a tidier single number.

3. Submit¶

Open a leaderboard submission issue with run_summary.json attached and the table above filled in. Submissions are checked for internal consistency and, where possible, spot-replicated before they go on the board.

Reading a leaderboard row honestly¶

Three habits worth having, including with our own numbers:

1. Look at hard fails before the aggregate. A high score with a non-zero hard-fail count is the dangerous profile: competent right up until it oversteps its role.
2. Distrust small gaps. With 67 dev tasks, differences under a couple of points are usually noise. Ask for the confidence interval.
3. Check the profile. An aggregate under a custom profile is not comparable to one under default_hpc_v01, however similar the number looks.

Running your own private leaderboard¶

Nothing here requires our involvement. aobench leaderboard serves the same view over your own runs, which is the right approach for evaluating vendor agents under NDA:

aobench serve rest             # then POST your runs
aobench leaderboard --help

The public board is a convenience, not the product. The reproducible evaluation is the product.

Related: versioning and comparability · reproducing results · evaluate your own agent