Contributors¶
Thank you. AOBench is a benchmark, which means its value is not in the code — it is in how carefully the 88 tasks, the 29 environments, the scoring rules, and the documentation have been checked by people who were not the person who wrote them. Every fix, every question that exposed an unclear page, every "this crashed for me" is that checking. This page is where those people are named.
Maintainer and advisor¶
-
Mohsen Seyedkazemi Ardebili @MSKazemi Creator & maintainer Design, scoring, corpus, and the research behind them - Andrea Bartolini ORCID Scientific advisor Co-author
Authors for citation purposes are listed in Cite AOBench and in CITATION.cff.
Contributors¶
Listed in the order their first contribution merged. The badge on each avatar marks what they worked on; the line underneath says it in words.
-
abhinav @erensh27 First contributor Friendly CLI errors -
Barshana Chatterjee @Barshana24 Contributor Machine-readable output · coverage matrix -
LeoZhaoo @LobsterQBA Contributor Side-by-side comparison -
Atiqur Rahman @atiqur-rahman-pro Contributor Lint gate for scripts/ · ICC(A,1) fix
| Contributor | What they added | |
|---|---|---|
| @erensh27 | An actionable one-line error, with tests, in place of the traceback you used to get from a mistyped --task or --env | #25 |
| @Barshana24 | --json on report json and compare runs, so AOBench's numbers can be piped into jq instead of scraped | #43 |
| @Barshana24 | aobench list coverage — the QCAT × role matrix, its thin and empty cells, and how much of each rests on real Marconi100 data | #47 |
| @LobsterQBA | examples/05_compare_two_adapters.py — two systems side by side with per-dimension deltas, offline | #44 |
| @atiqur-rahman-pro | ruff check over scripts/ — the 55 generators behind the catalogs, RBAC pages and paper tables, previously outside every gate | #45 |
| @atiqur-rahman-pro | The rubric reliability gate now computes ICC(A,1), the statistic it had always documented, instead of ICC1 | #46 |
This is a young project and that is a short list. It is worth reading anyway, because one of those contributions has already paid for itself: the CLI tests in #25 failed against the maintainer's own overlapping implementation and exposed a ranking bug in it — --task JOB_USR_00 was answering "did you mean JOB_USR_005, JOB_USR_004, JOB_USR_003?" and silently omitting JOB_USR_001. A second person's test found what the author's own tests were structurally unable to see. That is the argument for contributing here, and it is the reason the list being short is a reason to add to it rather than a reason to wait.
How you get on this wall¶
Every merged contribution earns a place here, whatever its size. A typo fix in the docs is a real contribution to a project whose documentation is the product.
| If you want to… | Start here |
|---|---|
| Fix something small and well-specified | Good first issues — each names the files, the tests, and an honest time estimate |
| Improve a page that confused you | Edit it directly; the pencil icon at the top of every page opens a PR |
| Report a bug or request a feature | Open an issue |
| Propose a task or an environment | Contributing guide |
| Ask something | Discussions — questions are welcome and expected |
You do not need HPC access or a cluster. The whole benchmark runs against frozen snapshots on a laptop, and the direct_qa adapter needs no API key.
What recognition means here¶
- Code, docs, tests, corpus, and review all count. Reviewing someone else's PR carefully is a contribution, and it gets listed.
- Release notes name contributors for the version their change shipped in, and the changelog links the person next to the fix.
- Substantial corpus or methodological contributions may warrant co-authorship on a paper that depends on them. If you think that applies to your work, say so — the awkwardness of asking should not be what decides who gets credit.
- Contributions stay listed even if you later step away.
- You can decline. If you would rather not appear here, say so in the PR and you won't.
What you can expect from us¶
A first response within three working days, even when that response is only "seen, I'll look properly on Friday". If a PR of yours goes quiet for longer, ping it — that is our failure, not rudeness on your part.
We hold ourselves to that publicly because we have already missed it: PR #25 sat for a day while an overlapping implementation was written and merged in parallel, which is the one thing a maintainer most owes a contributor not to do. The rework of that PR preserves the contributor's commits and credit precisely because the mistake was ours.
Acknowledgements¶
- CINECA, for publishing the Marconi100 ExaData release, which is what lets six of AOBench's environments be grounded in real Tier-0 operational data rather than invented. A benchmark of this kind is only as credible as the real data underneath it.
- The authors of BFCL, τ-bench, SWE-bench, and TRAIL, whose evaluation designs AOBench borrows from directly and cites in related work.
The canonical, machine-readable record of everyone listed here is AUTHORS.md in the repository root; this page is its presentation. Author metadata for citation purposes lives in CITATION.cff — see Cite AOBench.