Leaderboard¶

A leaderboard is only as good as the reproducibility of its rows. Every entry here names its AOBench version, split, scoring profile, and exact model snapshot, so anyone can re-derive it. Entries that cannot be re-derived do not go on the board.

Reference baselines¶

These ship with the benchmark and anchor the scale. Reproduce them yourself with the commands shown.

Entry	Score	Version	Split	Profile	Reproduce
`direct_qa` (tool-free floor), task `JOB_USR_001`	0.334	0.4.1	—	`default_hpc_v01`	`aobench quickstart`

direct_qa calls no tools and answers from the prompt alone. It exists to give the scale a floor: any tool-using agent that does not clearly beat it is not using tools usefully. A score below the floor generally means the agent is calling tools badly rather than not at all.

Why this table is short

Model rows are added as runs are completed and verified against the submission requirements below. We would rather publish three rows anyone can reproduce than thirty nobody can. If you have run AOBench, your submission is genuinely wanted — including a bad result, which is often the more informative kind.

Submitting a result¶

1. Run it¶

aobench run all --adapter <your adapter> --split dev
aobench clear run data/runs/<run_id>
aobench report json data/runs/<run_id> > my_result.json

Use --split dev. Results on the locked test split are accepted only from maintainers or by prior arrangement, because a public test-split leaderboard is a training target within a year.

2. Check it meets the bar¶

A submission must state:

Field	Example	Why
AOBench version	`0.4.1`	The corpus is part of the version
Split	`dev`	Scores differ by split
Scoring profile	`default_hpc_v01`	Weights change the aggregate
Adapter	`openai`	How the agent was driven
Model snapshot	`gpt-4o-2024-11-20`	Dated, never a moving alias
Judge model	`gpt-4o-2024-11-20` or `n/a`	Rubric-path variance
Runs	`3`	Single runs are not evidence
Hard fails	`0`	Reported separately from the score, always
Cost	`$4.20`	So others can budget a replication

And it must be re-derivable by someone else: the adapter has to be either one that ships with AOBench, a public MCP server, or a documented endpoint.

3. Submit¶

Open a leaderboard submission issue with my_result.json attached and the table above filled in. Submissions are checked for internal consistency and, where possible, spot-replicated before they go on the board.

Reading a leaderboard row honestly¶

Three habits worth having, including with our own numbers:

Look at hard fails before the aggregate. A high score with a non-zero hard-fail count is the dangerous profile: competent right up until it oversteps its role.
Distrust small gaps. With 67 dev tasks, differences under a couple of points are usually noise. Ask for the confidence interval.
Check the profile. An aggregate under a custom profile is not comparable to one under default_hpc_v01, however similar the number looks.

Running your own private leaderboard¶

Nothing here requires our involvement. aobench leaderboard serves the same view over your own runs, which is the right approach for evaluating vendor agents under NDA:

aobench serve rest             # then POST your runs
aobench leaderboard --help

The public board is a convenience, not the product. The reproducible evaluation is the product.