AOBench System Architecture — Paper Section¶

This document is a paper-ready System Description section for AOBench. The accompanying rendered flowchart is at docs/reference/architecture-diagram.html (v01 preserved at architecture-diagram-v01.html).

3. System Architecture¶

AOBench is a benchmarking framework for evaluating tool-using AI agents in High-Performance Computing (HPC) operations. Its architecture separates three concerns that are often conflated in agent benchmarks: task specification (what the agent must accomplish and how success is judged), execution environment (the deterministic world the agent acts in), and evaluation logic (how the interaction record maps to scores). This separation enables fully reproducible, extensible evaluation with explicit governance semantics.

Figure 1 gives a schematic view of the end-to-end pipeline. We describe each architectural layer in turn.

3.1 Specification Layer¶

Every evaluation instance in AOBench is defined by exactly two artefacts loaded at run time: a Task Specification and an Environment Bundle.

3.1.1 Task Specification (TaskSpec)¶

A TaskSpec is a structured JSON record that encodes what the agent must do and the conditions under which it will be judged. The schema (schemas/task.py) captures the following fields:

AOBench v0.3 ships 80 task specifications covering all 10 QCATs across all 5 HPC roles, split into 59 development and 21 held-out test tasks (Section 3.5).

3.1.2 Environment Bundle (EnvironmentBundle)¶

An EnvironmentBundle (schemas/environment.py) is a versioned, self-contained snapshot of an HPC facility's operational state. Each bundle is a directory containing:

• slurm_state.json — complete SLURM scheduler state: jobs, nodes, partitions.
• rbac_policy.yaml (v1.1) — per-role access control lists: allowed_tools, partition access tiers, and per-method restrictions.
• telemetry/*.parquet, *.csv — time-series node metrics and memory event records.
• incident_metadata.json — active incidents and severity metadata.
• docs/*.md — HPC documentation files (policies, tutorials, FAQs).
• inventory.json — physical rack, node, and GPU inventory.

AOBench ships 23 such bundles (env_01–env_23). Crucially, all data are static snapshots — no live cluster is required. This makes every benchmark run fully deterministic and reproducible: given the same task and environment identifiers, any agent receives identical tool responses regardless of when or where the benchmark runs.

3.2 Execution Layer¶

The execution layer drives an agent through a bounded interaction with the simulated HPC environment and captures the resulting trace.

3.2.1 BenchmarkRunner¶

BenchmarkRunner (runners/runner.py) is the central orchestrator. Its main entry point, run(task_id, env_id) → BenchmarkResult, performs the following steps in sequence:

1. Load the TaskSpec via TaskLoader.
2. Load and validate the EnvironmentBundle via SnapshotLoader (ten fidelity checks T1–T10 must pass).
3. Construct a role-filtered ToolRegistry from the bundle.
4. Instantiate the requested adapter.
5. Build an ExecutionContext (runners/context.py) bundling the task, environment, tool registry, and run identifier.
6. Invoke adapter.run(context) → Trace.
7. Score the trace via AggregateScorer.
8. Persist the BenchmarkResult and Trace to data/runs/<run_id>/.
9. Optionally export to an observability backend.

3.2.2 Agent Adapters¶

AOBench defines a BaseAdapter interface (adapters/base.py) with a single method run(context: ExecutionContext) → Trace. Four concrete backends are provided:

All adapters implement an internal loop of at most 10 rounds. In each round the LLM receives the task prompt together with the JSON schemas of the tools available for its role. The model may respond with one or more tool calls; the Tool Registry processes each call and returns a structured observation. The loop terminates when the model emits a stop signal or the round limit is reached.

3.2.3 Tool Registry and Mock HPC Tools¶

The ToolRegistry (tools/registry.py) is the execution-time gate between the agent and the simulated HPC environment. It enforces two categories of restriction.

Role-based filtering. Only methods listed in the task's allowed_tools and permitted by the environment's rbac_policy.yaml for the agent's role are exposed. A call to a non-permitted method returns a structured permission_denied observation — the agent receives this as feedback and may adjust its strategy.

Dangerous-argument detection. If a tool call's arguments match conditions defined in benchmark/configs/hpc_tool_catalog.yaml, the registry sets a hard_fail flag on the trace. This flag propagates to the Governance scorer and zeroes the entire task score regardless of any other dimension.

Five mock tool families read their data directly from the EnvironmentBundle snapshot, ensuring full determinism:

The full method catalog with role-visibility rules and dangerous-argument conditions is in benchmark/configs/hpc_tool_catalog.yaml (16 entries).

3.2.4 TraceWriter and Trace Schema¶

Every interaction step — agent messages, tool calls, and observations — is appended to a Trace object (schemas/trace.py) by the TraceWriter (runners/trace_writer.py). The finalised Trace includes the ordered sequence of TraceStep objects (each carrying a ToolCall and its Observation), the final_answer string, the hard_fail flag and reason, the model name, and prompt and completion token counts.

The Trace schema is normative across all adapters: any conforming adapter produces the same trace format, making scorer behaviour adapter-independent.

3.3 Evaluation Layer¶

3.3.1 Scoring Dimensions¶

The AggregateScorer (scorers/aggregate.py) accepts a (TaskSpec, Trace) pair and orchestrates six independent scorers, combining their outputs according to a named weight profile. The default profile default_hpc_v01 assigns the following weights:

We describe each dimension below.

3.3.2 Outcome (weight 0.30)¶

OutcomeScorer (scorers/outcome_scorer.py) measures whether the agent's final_answer is correct. Three base modes are available, selected by the task's eval_criteria.match_type:

• exact_match — case-insensitive string equality after normalisation.
• numeric — passes if $|\hat{y} - y^| / y^ \leq 0.05$ (±5% relative tolerance).
• semantic_match — a blend of 60% rapidfuzz token-sort ratio and 40% numeric proximity, designed for answers that mix numeric values with HPC entity names.

For tasks with a hybrid_scoring_config, OutcomeScorer is replaced by HybridScorer (scorers/hybrid_scorer.py), which routes to one of two paths:

• Deterministic path (scorers/deterministic.py) — DAComp-style component scoring (CS) with cascading failure propagation (CFS) through declared upstream_deps, and strict all-or-nothing scoring (SR).
• Rubric path (scorers/rubric_scorer.py) — an LLM judge evaluates the answer against a structured rubric from prompts/judge/, returning a normalised score. Optionally, a Good-Sufficient-Bad (GSB) comparative scorer (scorers/gsb_scorer.py) is blended in with a configurable weight $\alpha$.

3.3.3 Governance (weight 0.20)¶

GovernanceScorer (scorers/governance_scorer.py) is the primary safety dimension. It detects role-based access control (RBAC) violations and policy breaches, tracking them in a six-flag ViolationVector:

When hard_fail_trigger (V6) fires, the AggregateScorer forces the entire task aggregate_score to 0.0, regardless of scores on all other dimensions. This models the real-world requirement that an HPC AI assistant must never operate outside its authorised access scope.

3.3.4 Tool Use (weight 0.15)¶

ToolUseScorer (scorers/tool_use_scorer.py) decomposes tool-use quality into four sub-scores, following a BFCL-inspired decomposition:

$$\text{selection} = \frac{|\mathcal{E} \cap \mathcal{A}|}{|\mathcal{E}|}, \quad \text{argument} = \frac{1}{|\mathcal{A}|}\sum_{c \in \mathcal{A}} \mathrm{arg_match}(c), \quad \text{sequence} = \frac{\mathrm{LCS}(\mathcal{E}, \mathcal{A})}{|\mathcal{E}|},$$

$$\text{penalty} = \max!\left(0,\ 1 - 0.3 \cdot |\mathcal{F}|\right),$$

where $\mathcal{E}$ is the expected tool-call set, $\mathcal{A}$ the actual set, and $\mathcal{F}$ the set of forbidden calls made by the agent. Per-argument matching uses ±5% relative tolerance for numeric arguments and exact string equality otherwise.

When a gold_trajectory is defined, the scorer upgrades to a weighted blend:

$$\text{tool_use} = 0.5 \cdot \text{base} + 0.3 \cdot \text{NED} + 0.2 \cdot \text{F1}_{\text{steps}},$$

where NED is the normalised edit distance between the actual and gold trajectory sequences.

3.3.5 Grounding (weight 0.10)¶

GroundingScorer (scorers/grounding_scorer.py) estimates how well the agent's answer is grounded in the evidence returned by tool calls. It computes key-token overlap between the answer tokens and the tokens present in all tool observations:

$$\text{grounding} = \frac{|\mathcal{K}(\text{answer}) \cap \mathcal{K}(\text{observations})|}{|\mathcal{K}(\text{answer})|},$$

where $\mathcal{K}(\cdot)$ extracts key tokens: multi-digit numeric strings, HPC entity identifiers (matching patterns node*, gpu*, rack*, job*, partition*), and domain status words (e.g. running, failed, pending). This penalises models that produce plausible-sounding answers not supported by the retrieved evidence.

3.3.6 Robustness (weight 0.10)¶

RobustnessScorer (scorers/robustness_scorer.py) measures score consistency across $n$ repeated runs of the same task under identical conditions. The primary metric is pass$^k$:

$$\mathrm{pass}^k = \prod_{i=0}^{k-1} \frac{c - i}{n - i},$$

where $c$ is the number of passing runs, $n$ the total runs, and a run passes if aggregate_score $\geq$ 0.5 (default threshold). The aobench robustness command computes pass$^k$ for $k \in {1, 2, 4, 8}$. The CLEAR scorecard uses pass$^k$ with $k=1$ and threshold 0.5 by default.

3.3.7 Efficiency (weight 0.05)¶

EfficiencyScorer (scorers/efficiency_scorer.py) applies a linear penalty on interaction length:

$$\text{efficiency} = \begin{cases} 1.0 & \text{steps} \leq 5 \ 0.0 & \text{steps} \geq 20 \ \frac{20 - \text{steps}}{15} & \text{otherwise} \end{cases}$$

This rewards concise task completion and penalises excessive back-and-forth.

3.3.8 Workflow (weight 0.10)¶

WorfEvalScorer (scorers/workflow_scorer.py) is activated when a task provides a ground_truth_workflow DAG. The scorer uses WorkflowGraphBuilder to reconstruct the agent's executed workflow as a directed acyclic graph from the trace, then computes structural similarity against the gold DAG using the WorfEval methodology — capturing node coverage, edge agreement, and ordering correctness. Tasks without a ground_truth_workflow receive a workflow score of 0 with zero weight contribution.

3.3.9 Aggregate Score and Hard-Fail Gate¶

The AggregateScorer combines dimension scores using the active weight profile:

$$S_{\text{agg}} = \sum_{d} w_d \cdot s_d, \quad \sum_d w_d = 1,$$

subject to the hard-fail gate:

$$S_{\text{final}} = \begin{cases} 0.0 & \text{if hard_fail} \ S_{\text{agg}} & \text{otherwise} \end{cases}$$

When a task defines checkpoints, the CheckpointScorer (scorers/checkpoint_scorer.py) evaluates intermediate correctness milestones and computes a partial-credit score $s_{\text{partial}}$ that replaces $s_{\text{outcome}}$ in the aggregate computation.

3.4 Output Layer¶

3.4.1 BenchmarkResult¶

Each scored run produces one BenchmarkResult record (schemas/result.py) containing: the six dimension scores, the aggregate score $\in [0, 1]$, the hard_fail flag and reason, the CuP-gated efficacy score (scoring/cup_scorer.py), the ViolationVector (six flags), estimated cost in USD, wall-clock latency in seconds, and token counts. Results are persisted as JSON under data/runs/<run_id>/.

3.4.2 CLEAR Scorecard¶

The CLEAR scorecard (reports/clear_report.py) aggregates per-task BenchmarkResult records into five run-level dimensions that span efficacy, safety, reliability, and operational cost:

The composite CLEAR score is the unweighted mean: $\text{CLEAR} = \frac{1}{5}(E + A + R + C + L)$.

Additional diagnostic metrics reported alongside CLEAR include: Cost-Normalised Accuracy (CNA $= \text{outcome}/\text{cost_usd} \times 100$), Cost Per Success (CPS $= \text{total_cost}/n_{\text{successful}}$), CuP-gated efficacy, and per-flag violation ratios from the ViolationVector.

3.4.3 Report Artifacts¶

The aobench report sub-commands generate three output surfaces: a structured JSON run summary (reports/json_report.py), a self-contained HTML report with per-task detail (reports/html_report.py), and a Role × QCAT stratification table showing performance broken down by requester role and query category (reports/slice_report.py). An optional LangfuseExporter (exporters/langfuse_exporter.py) ships the full trace and score data to an observability backend for interactive analysis across runs.

3.5 Dataset and Splits¶

The benchmark comprises 80 task specifications across 10 Query Categories (QCATs) and 5 HPC roles, evaluated against 23 deterministic environment bundles. Table 1 summarises the QCAT taxonomy; Table 2 the role definitions.

The dataset is split into 59 development tasks (~74%) and 21 held-out test tasks (~26%), stratified by QCAT × role × difficulty. The split was constructed deterministically: for each stratum with ≥2 tasks, the hardest task (by difficulty field, ties broken by ascending task_id) is assigned to the test set; for QCATs where every stratum has exactly one task (DATA, DOCS, FAC), the single hardest task in the QCAT enters the test set. This ensures all 10 QCATs and all 5 roles appear in both splits. The split was frozen on 2026-05-03 (benchmark/tasks/dataset_splits.py). Test-set tasks are locked behind the AOBENCH_UNLOCK_TEST=1 environment variable; all published experiments use the development split.

3.6 Validation Gates¶

Before any evaluation run, the aobench validate benchmark command applies ten fidelity checks (T1–T10) to every task and environment bundle:

Any task failing a gate is excluded from evaluation. These checks prevent the benchmark from accepting tasks that are unsolvable, ambiguous, trivially answered without tool use, or contaminated by prompt leakage — failure modes that have been documented in prior agent evaluation work.

Component Reference¶