Figure 1 — AOBench System Architecture

End-to-end pipeline from specification to scored output. A Task Specification (role, query, evaluation criteria, gold trajectory) and a deterministic Environment Bundle (SLURM scheduler state, RBAC policy, telemetry snapshots) are loaded by the BenchmarkRunner, which drives a pluggable Agent Adapter through a bounded tool-call loop (at most 10 rounds) against a role-filtered Tool Registry backed by five mock HPC tool families. Each step is recorded by the TraceWriter. The resulting Trace is evaluated across five dimensions by the AggregateScorer; a Governance hard-fail zeroes the entire task score. Per-task BenchmarkResult records feed the run-level CLEAR scorecard (Efficacy, Assurance, Reliability, Cost, Latency) and downstream report artifacts.

SPECIFICATION EXECUTION EVALUATION OUTPUT Task Specification 80 tasks · 10 QCATs · 5 roles task_id, role, qcat, difficulty query_text, eval_criteria allowed_tools, hard_fail_cond. gold_trajectory, checkpoints hybrid_scoring_config schemas/task.py · TaskSpec Environment Bundle 26 deterministic snapshots slurm_state.json rbac_policy.yaml (v1.1) telemetry/*.parquet, *.csv incident_metadata.json docs/*.md · inventory.json schemas/snapshot.py · EnvBundle Agent Adapter pluggable LLM backends direct_qa (zero-tool baseline) openai:gpt-4o / gpt-4o-mini anthropic:claude-3-5-sonnet mcp (stdio / SSE transports) vLLM / OpenRouter (OAI compat.) adapters/base.py · BaseAdapter BenchmarkRunner loads · routes · orchestrates · persists Execution Loop (≤ 10 tool-call rounds) LLM Agent receives task prompt + tool schemas (JSON) emits tool_call(s) receives observations emits stop → final_answer adapter.run(context) adapters/{openai,anthropic,mcp} tool call observation Tool Registry role-filtered dispatch RBAC policy enforcement ✕ denied → permission_denied ✕ dangerous_arg → hard_fail rbac_policy.yaml per bundle hpc_tool_catalog.yaml tools/registry.py · ToolRegistry Mock HPC Tools slurm telemetry docs rbac facility tools/*.py HPC Tool Families ▸ slurm — query_jobs, job_details, list_nodes, list_partitions, cancel_job ▸ telemetry — query_timeseries, query_node_metrics, query_mem_events ▸ docs — retrieve (policy / guides) ▸ rbac — check, get_allowed_tools, check_permission ▸ facility — get_power_usage, query_node_power, set_power_cap 5 families · 16 methods TraceWriter records each step Trace ordered steps · tool_calls · observations · final_answer · token counts AggregateScorer (profile: default_hpc_v01) Outcome weight = 0.30 exact / numeric / semantic / hybrid +HybridScorer path Tool Use weight = 0.20 selection · args sequence · penalty BFCL-decomposed Grounding weight = 0.15 answer ∩ observation token overlap HPC entity matching Governance weight = 0.20 RBAC violations hard_fail → score = 0 violation_vector (6 flags) Efficiency weight = 0.05 step count ≤5→1.0, ≥20→0.0 linear interpolation BenchmarkResult dimension_scores (5 dims) aggregate_score ∈ [0, 1] hard_fail · cup_score violation_vector (6 flags) cost_usd · latency_s · tokens data/runs/<run_id>/ — per task CLEAR Scorecard E — Efficacy (mean outcome) A — Assurance (RBAC compliance rate) R — Reliability (pass^k, k = 8) C — Cost (min-max, inverted) L — Latency (min-max, inverted) reports/clear_report.py — per run Report Artifacts JSON run summary HTML self-contained report Role × QCAT slice breakdown Langfuse trace export Leaderboard submission reports/ · exporters/ · leaderboard/