Contributors¶
Thank you. AOBench is a benchmark, which means its value is not in the code — it is in how carefully the 90 tasks, the 29 environments, the scoring rules, and the documentation have been checked by people who were not the person who wrote them. Every fix, every question that exposed an unclear page, every "this crashed for me" is that checking. This page is where those people are named.
Maintainer and advisor¶
-
Mohsen Seyedkazemi Ardebili @MSKazemi Creator & maintainer Design, scoring, corpus, and the research behind them - Andrea Bartolini ORCID Scientific advisor Co-author
Authors for citation purposes are listed in Cite AOBench and in CITATION.cff.
Contributors¶
Listed in the order their first contribution merged. The badge on each avatar marks what they worked on; the line underneath says it in words.
-
abhinav @erensh27 First contributor Friendly CLI errors -
Barshana Chatterjee @Barshana24 Contributor Machine-readable output · coverage matrix · typed CLI -
Leo Zhao @LobsterQBA Contributor Side-by-side comparison · report error messages -
Atiqur Rahman @atiqur-rahman-pro Contributor Lint gate for scripts/ · ICC(A,1) fix -
Dream @TrueFurina Contributor The first-10-minutes path -
lorenzo-benites @lorenzo-benites Contributor Typed reports, leaderboard, judge & scorers -
Akimbo92i @Akimbo92i Contributor The last ten type errors · a test-isolation bug found by running one extra gate -
Enzo @motodriver Contributor Fixed the Langfuse test-isolation bug Akimbo92i found · a leaked coroutine · a scaffold-scoring bug · the test-count fact gate -
Qiu Guanzong @QIU-Guanzong Contributor A type signature the corpus disagreed with · closed the encoding gate's own blind spot · a stale-dimension-count regex gap -
Ziao Yang @yangziao56 Contributor `PERF_RES_002`, a checkpoint-throughput task with its own independent evidence audit -
BillP313 @BillP313 Contributor The README's second demo, recorded from a Windows PowerShell prompt
| Contributor | What they added | |
|---|---|---|
| @erensh27 | An actionable one-line error, with tests, in place of the traceback you used to get from a mistyped --task or --env | #25 |
| @Barshana24 | --json on report json and compare runs, so AOBench's numbers can be piped into jq instead of scraped | #43 |
| @Barshana24 | aobench list coverage — the QCAT × role matrix, its thin and empty cells, and how much of each rests on real Marconi100 data | #47 |
| @LobsterQBA | examples/05_compare_two_adapters.py — two systems side by side with per-dimension deltas, offline | #44 |
| @LobsterQBA | aobench report now names a missing run directory and lists the runs that exist, instead of raising a traceback | #48 |
| @atiqur-rahman-pro | ruff check over scripts/ — the 55 generators behind the catalogs, RBAC pages and paper tables, previously outside every gate | #45 |
| @atiqur-rahman-pro | The rubric reliability gate now computes ICC(A,1), the statistic it had always documented, instead of ICC1 | #46 |
| @Barshana24 | mypy --strict clean across cli/ — narrowed, not silenced — plus a latent None-comparison crash found and filed rather than folded in | #50 |
| @TrueFurina | Your first 10 minutes with AOBench — one unbranched path from git clone to reading a score, the route the five existing pages never drew | #52 |
| @lorenzo-benites | mypy --strict clean across reports/, leaderboard/ and judge/, and a judge that now rejects a non-object JSON reply instead of returning it as a dict | #53, #54, #55 |
| @lorenzo-benites | scorers/ from 26 mypy --strict errors to 1, including the reachable Anthropic content-block bug in the rubric judge | #59 |
| @Akimbo92i | The last ten mypy --strict errors, in the five mock HPC tools every task runs through — tools now carries no budget at all, so it cannot rot back | #64 |
| @motodriver | Fixed the order-dependent Langfuse exporter test failure — one root cause (patch.dict evicting a module it imported) applied at all six call sites | #67 |
| @QIU-Guanzong | MockSlurmTool._load_json's return type now says what the corpus actually contains — an @overload pair instead of one signature that couldn't cover both shapes | #68 |
| @QIU-Guanzong | Brought examples/ inside the encoding gate — the CI-gate example itself could fail with a Windows cp1252 error while the gate it exercises reported a clean tree | #79 |
| @QIU-Guanzong | Closed the check_dimension_counts regex gap that missed "six evaluation dimensions" — a qualifying word between the number and "dimension" the old pattern never allowed for | #86 |
| @mgalore | Caught that review task --json called an untouched scaffold ok: true — the one state the command is guaranteed to run in was the state it green-lit | #77 |
| @userfypp | The first corpus contribution from outside the project — DOCS_USR_002, a PII storage-compliance task grounded in env_21, which took thin cells from 32 of 50 to 31. It exists because they stopped to report the retrieval bug that blocked it instead of reshaping the task around it | #78 |
| @motodriver | A mocked asyncio.run(...) left the real coroutine alive in the mock's call args, leaking a RuntimeWarning into unrelated later tests | #80 |
| @motodriver | Stopped scaffolded/unready tasks from joining scored dev runs, and made an empty expected_tool_calls score as unmeasurable instead of a vacuous perfect 1.0 | #82 |
| @motodriver | check_facts.py now guards the documented test-suite counts too, closing the exact class of drift that let README.md say ~1510 tests long after the real count moved | #83 |
| @yangziao56 | PERF_RES_002 — a researcher task that separates throughput reduction from CPU/thermal figures and treats a reported thermal cause as reported, not established — shipped with its own offline audit script that independently re-derives the gold answer from the real mock-tool output | #81 |
| @BillP313 | docs/assets/demo-windows.gif — the same run-task-then-clear-run demo as the README's original, recorded from a Windows PowerShell prompt with ScreenToGif after reporting that vhs produces no output on Windows | #87 |
This is a young project and that is a short list. It is worth reading anyway, because one of those contributions has already paid for itself: the CLI tests in #25 failed against the maintainer's own overlapping implementation and exposed a ranking bug in it — --task JOB_USR_00 was answering "did you mean JOB_USR_005, JOB_USR_004, JOB_USR_003?" and silently omitting JOB_USR_001. A second person's test found what the author's own tests were structurally unable to see. That is the argument for contributing here, and it is the reason the list being short is a reason to add to it rather than a reason to wait.
Reported and tested¶
A benchmark is checked by people who run it somewhere the maintainer cannot and report what happened. Every person below found a real defect that is now fixed, and each was right on the first telling — no reproduction work was needed on any of them.
-
hari760 @hari760 Reporter The Windows portability class · five defects from one run -
userfypp @userfypp Reporter The truncated-snippet grounding bug
| Reporter | What their report found | |
|---|---|---|
| @hari760 | Ran the dev split on Windows and lost it at task 6 of 67 to a UnicodeEncodeError, with the correct diagnosis attached: text I/O with no encoding= falls back to cp1252. 259 call sites across src/, tests/ and scripts/ had it — so Windows contributors could not run the test suite either. There is a static gate for it now (make encoding-check), because Linux CI can never see this class of bug | #60 |
| @hari760 | Left the per-dimension field blank because nothing emitted it — which exposed that workflow, carrying 0.10 weight, was missing from every task row, from the OTel export and from compare runs --show-dims. The published breakdown could not have reconciled against the aggregate it explained. Also caught that the submission form demanded one scoring profile where the corpus uses two, and that a documented submission command captured prose rather than JSON | #60 |
| @hari760 | Reported the two task IDs that hard failed rather than only the score — and because the same two failed in all three runs, it was diagnosable as construction rather than behaviour. AIOPS_USR_001 and PERF_USR_001 ask about the requester's own job while the snapshot assigns it to someone else, so a correct agent is RBAC hard-failed and the task zeroed: unpassable by any agent. One was already excluded from scoring; the other was not, and had been depressing published aggregates unnoticed | #63 |
| @userfypp | Checked a proposed task against env_21 before writing it and found MockDocsTool._retrieve returned characters 0–500 rather than the match. The tool reported a hit whose snippet contained none of the query terms — so an agent could be scored on grounding against evidence the tool would never surface, silently capping the dimension on any task whose answer lives late in a long document | #26 |
Both of those are scoring- or portability-integrity bugs found from the outside, by people using the benchmark rather than reading it. That is the argument for running AOBench somewhere unusual and saying what broke: it is worth more than the score.
Design and interoperability¶
Reading a codebase carefully enough to find where its model is wrong is a contribution, and it does not require a commit.
-
Sahi @adhabnr-ux Design input Eval-interchange mapping · the hard-constraint question
@adhabnr-ux proposed an EvalPort interchange adapter in discussion #51 with a complete field-by-field mapping of TaskSpec / Trace / BenchmarkResult onto the schema — derived from reading src/aobench/schemas/ rather than the README, and correct in the places that are easy to get wrong. The valuable part is what it could not map: an RBAC hard fail zeroes an entire task, including dimensions the agent handled well, and no per-dimension grader slot can express that. Rather than paper over it, the mismatch was flagged as an open question about how eval interchange formats should model hard constraints at all. That is a genuine gap in the field, not just in one adapter, and it was raised before a line of code was written.
Work in flight¶
Named here so nobody duplicates the effort, and because a claim is worth acknowledging before it lands rather than only afterwards.
| Who | What | Since |
|---|---|---|
| @hoti-code | A LiteLLM adapter — one file, ~100 providers. Settled the litellm:<model> adapter-string format by asking first, which is why it will not need redoing in review | 2026-09-01 |
| @aawhan0 | Verifying that the documented commands actually do what the pages say, from a clean checkout — quickstart.md first. Volunteered for exactly this in August and was let down by us before it could start | 2026-09-10 |
Claiming an issue protects it. Comment to say you are taking something and it is yours; nothing will be merged over you. That promise is written down because it was broken once.
How you get on this wall¶
Every merged contribution earns a place here, whatever its size, and so does a report that turns out to be right. A typo fix in the docs is a real contribution to a project whose documentation is the product — and so is telling us that the benchmark crashed on your machine.
| If you want to… | Start here |
|---|---|
| Fix something small and well-specified | Good first issues — each names the files, the tests, and an honest time estimate |
| Improve a page that confused you | Edit it directly; the pencil icon at the top of every page opens a PR |
| Report a bug or request a feature | Open an issue — a correct report earns a place on this page as surely as a patch does |
| Propose a task or an environment | Contributing guide |
| Ask something | Discussions — questions are welcome and expected |
You do not need HPC access or a cluster. The whole benchmark runs against frozen snapshots on a laptop, and the direct_qa adapter needs no API key.
What recognition means here¶
- Code, docs, tests, corpus, review, and bug reports all count. Reviewing someone else's PR carefully is a contribution, and so is a report that turns out to be right — both get listed.
- Release notes name contributors for the version their change shipped in, and the changelog links the person next to the fix.
- Substantial corpus or methodological contributions may warrant co-authorship on a paper that depends on them. If you think that applies to your work, say so — the awkwardness of asking should not be what decides who gets credit.
- Contributions stay listed even if you later step away.
- You can decline. If you would rather not appear here, say so in the PR and you won't.
What you can expect from us¶
A first response within three working days, even when that response is only "seen, I'll look properly on Friday". If a PR of yours goes quiet for longer, ping it — that is our failure, not rudeness on your part.
We hold ourselves to that publicly because we have already missed it: PR #25 sat for a day while an overlapping implementation was written and merged in parallel, which is the one thing a maintainer most owes a contributor not to do. The rework of that PR preserves the contributor's commits and credit precisely because the mistake was ours.
If you want your work seen¶
This is said once, here, so it never has to be attached as a condition to a review.
AOBench's bottleneck is discovery, not interest. The code gets used and the fixes get merged; almost nobody knows the project exists. That is a problem you can help with more cheaply than by writing any more code, and doing so happens to serve you too:
- This page is public and linkable. Your entry has a stable URL and your own words describing what you did. Put it in a CV, a portfolio, or a profile — it is third-party evidence of your work, which is worth more than a self-description.
- Link your own pull request, not the repository. "Cleared the strict-typing debt in the CLI of an open HPC agent benchmark, and found a latent crash doing it" is a better sentence about you than any link to us. The credit is yours; point at it.
- Release notes name you for the version your change shipped in, and the changelog puts your handle next to the fix.
- A ⭐ is the strongest discovery signal GitHub surfaces, and the project is early enough that each one measurably moves it. If you know someone working on agent evaluation, HPC operations or LLM benchmarking, pointing them here is worth more to this project than another feature.
All of that is entirely optional and always will be. It is never a condition of review, it changes nothing about how your next pull request is read, and nobody will ask you twice.
Acknowledgements¶
- CINECA, for publishing the Marconi100 ExaData release, which is what lets six of AOBench's environments be grounded in real Tier-0 operational data rather than invented. A benchmark of this kind is only as credible as the real data underneath it.
- The authors of BFCL, τ-bench, SWE-bench, and TRAIL, whose evaluation designs AOBench borrows from directly and cites in related work.
The canonical, machine-readable record of everyone listed here is AUTHORS.md in the repository root; this page is its presentation. Author metadata for citation purposes lives in CITATION.cff — see Cite AOBench.