Roadmap¶
Shipped¶
Core benchmark (v0.3)¶
- 88 tasks × 29 environment bundles, 5 roles, 10 QCATs, 12 scorers / 7 weighted dimensions.
- CLEAR scorecard (Cost / Latency / Efficacy / Assurance / Reliability).
- M100 ExaData-grounded environments and tasks.
Multi-surface engine access (v0.4, in progress on feat/aobench-futures-30-features)¶
Every surface calls the shared BenchmarkService façade — identical scores across transports.
| Surface | Status | Notes |
|---|---|---|
REST API (FastAPI) — aobench serve rest | ✅ | run/score/report/trace/compare/robustness/datasets + async jobs + SSE; API-key→role auth + rate limiting |
FastMCP server — aobench serve mcp | ✅ | tools (run_task/score_trace/validate_benchmark/robustness) + aobench:// resources; JWT auth hook |
| A2A evaluation | ✅ | Agent-Card conformance + delegation / comms-cost / attribution / lifecycle / card-poisoning scorers |
| CLI / terminal evaluation | ✅ | mock Slurm shims, destructive-command guard, end-state verifier |
Cross-cutting (v0.4)¶
- OTel-GenAI trace exporter · deterministic replay cassettes · in-toto attestation.
- Carbon/cost/CO₂e accounting + contamination guard · pass^k + bootstrap CIs (measurement rigor).
- Futuristic HPC scorers: incident RCA (+ log evidence), escalation/abstention, carbon-aware scheduling, predictive maintenance.
- MCP-usage scorers: tool-selection, injection-resistance, elicitation.
In progress / partial (🟡)¶
| Item | Shipped | Deferred half |
|---|---|---|
| Async run queue (F2) | job registry + async submission (single process) | durable arq/Redis worker — #10 |
| Event delivery (F3) | SSE + polling | outbound webhooks — #11 |
| Typed SDK (F4) | REST auth + OpenAPI | generated client SDK — #12 |
| Datasets/Experiments API (F5) | datasets read side | experiment-result persistence DB — #13 |
| MCP auth (F7) | JWT scaffold | full OIDC → RBAC mapping — #14 |
| A2A access (F12) | adapter core (injected transport) | live A2A HTTP transport — #15 |
| CLI access (F19) | adapter core (injected executor) | Docker/gVisor executor — #19 |
| Futuristic HPC (F30) | carbon-aware + predictive-maintenance scorers | thermal digital-twin surrogate — #17 |
Deferred — external dependency (⛔)¶
- F8 — async MCP Tasks primitive (experimental in MCP 2025-11-25) — #18.
- F18 — containerized HPC terminal runner + tiered sandbox (needs a sandboxing dependency) — #19.
Next milestones¶
- Land
feat/aobench-futures-30-featuresinto the mainline and cut v0.4. - Infra features as dependencies come online: Redis worker (F2), Docker sandbox (F18/F19), live A2A transport (F12), experiment DB (F5), OIDC (F7), SDK codegen (F4), thermal twin (F30).
- Reporting suite — report renderers on the data→stats→model→render pipeline — #16.
- Clean up the pre-existing test failures in
governance_report/run_cmd/ ablation scripts — #8. - Quality debt: lint
scripts/— #6 · strict typing — #7.
How the roadmap is decided¶
Priority follows what actually blocks people, not what is most interesting to build. If a limitation stops you using AOBench, say so in Discussions — knowing which limitation genuinely bites is what decides the order things get fixed in.
Every roadmap item is tracked by a GitHub issue, and several are marked help wanted. Larger items (effort: large) are worth discussing in the issue before starting.
Related: limitations — the standing constraints, with the ones we intend to fix marked as such · open research questions