Skip to content

AOBench Framework Documentation

AOBench is a benchmark framework for evaluating AI agent systems in High-Performance Computing (HPC) environments. It uses role-aware tasks, deterministic HPC state snapshots, and trace-based scoring across six dimensions.

This index is the entry point to the framework documentation.

Document map

Document Purpose
Overview Five principles, v0.1 implemented scope, long-term goal
Background Motivation, the gap, related work
Architecture Conceptual benchmark architecture: layers, entities, workflow
Implementation Developer guide: how to extend adapters, tools, scorers, tasks
Environments Snapshot bundle format and core design
Evaluation Evaluation protocol, twelve scorers, hard-fail rules, trace and result schemas
Taxonomy Roles, QCATs, knowledge sources, RBAC tiers, task metadata schema
System Architecture Authoritative current-state reference — component map, end-to-end data flow, scoring pipeline, CLEAR scorecard, architecture diagrams
Scoring Dimensions Per-scorer formulas and weight profiles

For a complete component map, end-to-end data flow, and scoring pipeline details, see System Architecture.

Outside this directory

Document Purpose
docs/reference/commands.md Full CLI reference (every flag of every sub-command)
docs/reference/environments-overview.md Inventory of all 23 environment bundles
docs/guides/adapters-and-tools.md Plain-English adapter and tool guide
docs/reference/system-architecture.md System architecture + Mermaid diagrams
docs/guides/langfuse-integration.md Observability backend
CONTRIBUTING.md How to contribute tasks, adapters, scorers
GitHub README Repository entry point and quick start
CHANGELOG.md Release notes
CONTRIBUTING.md Contribution guide
SECURITY.md Vulnerability reporting and threat model