Skip to content

Environment catalog

AOBench ships 29 deterministic environment snapshot bundles: 23 synthetic and 6 built from real operational data from CINECA's 980-node Marconi100 Tier-0 supercomputer, via the public M100 ExaData release.

A bundle is a directory of frozen files — SLURM state, telemetry time series, documentation, RBAC policy, incident metadata — that the mock tools read from. Nothing in a run touches a live system, which is what makes a result reproducible on someone else's laptop years later.

Reproduce this page from your own checkout:

aobench list envs --json

All bundles

Environment Grounding Cluster Scenario Tasks Sources Snapshot
env_01 synthetic aobench-cluster-a job_failure 10 slurm, telemetry, docs, rbac, incidents User OOM Failure
env_02 synthetic aobench-cluster-a queue_congestion 9 slurm, telemetry, docs, rbac, incidents Queue Congestion / Long Pending Jobs
env_03 synthetic aobench-cluster-a thermal_power 10 slurm, telemetry, power, inventory, docs, rbac, incidents Thermal and Power Monitoring
env_04 synthetic aobench-cluster-a rack_energy 4 power, inventory, docs, rbac, slurm Rack Energy Comparison
env_05 synthetic aobench-cluster-a cooling_failure 7 power, rack, inventory, cooling, incidents, docs, rbac Cooling Unit Failure
env_06 synthetic aobench-cluster-a energy_anomaly 2 slurm, telemetry, docs, rbac, incidents GPU Power Spike
env_07 synthetic aobench-cluster-a energy_anomaly 1 slurm, telemetry, docs, rbac, incidents PUE Degradation Cooling Issue
env_08 synthetic aobench-cluster-a node_degradation 1 slurm, telemetry, docs, rbac, incidents Thermal Throttling on node03
env_09 synthetic aobench-cluster-a node_degradation 1 slurm, telemetry, docs, rbac, incidents Memory ECC Errors Flapping Node
env_10 synthetic aobench-cluster-a policy_violation 1 slurm, telemetry, docs, rbac, incidents Policy Violation Restricted Partition
env_11 synthetic aobench-cluster-a policy_violation 1 slurm, telemetry, docs, rbac, incidents Account Over Allocation Limit
env_12 synthetic aobench-cluster-a queue_congestion 1 slurm, telemetry, docs, rbac, incidents Fairshare Starvation Priority Inversion
env_13 synthetic aobench-cluster-a capacity_planning 1 slurm, telemetry, docs, rbac, incidents Six Month CPU Utilisation Trend
env_14 synthetic aobench-cluster-a capacity_planning 2 slurm, telemetry, docs, rbac, incidents GPU Demand Forecast Expansion
env_15 synthetic aobench-cluster-a multi_job_interference 1 slurm, telemetry, docs, rbac, incidents Multi-Job Memory Interference
env_16 synthetic aobench-cluster-a scheduler_misconfiguration 2 slurm, telemetry, docs, rbac, incidents Wrong Default Partition Misconfiguration
env_17 synthetic aobench-cluster-a job_failure 1 slurm, telemetry, docs, rbac, incidents MPI Communication Timeout Network Fault
env_18 synthetic aobench-cluster-a job_failure 1 slurm, telemetry, docs, rbac, incidents Checkpoint File Missing Restart Fails
env_19 synthetic aobench-cluster-a energy_anomaly 1 slurm, telemetry, docs, rbac, incidents GPU Idle Energy Waste Not Released
env_20 synthetic aobench-cluster-a multi_job_interference 1 slurm, telemetry, docs, rbac, incidents Lustre IO Contention Multi-Job Interference
env_21 synthetic aobench-cluster-a storage_management 7 filesystem, docs, rbac Storage Quota Pressure
env_22 synthetic aobench-cluster-b facility_incident 8 bms, telemetry, docs, rbac Cooling Alarm Response
env_23 synthetic aobench-cluster-c architecture_review 7 topology, inventory, docs, rbac Capacity Expansion Planning
env_m100_01 real M100 marconi100 node_degradation 2 slurm, telemetry, docs, rbac, incidents M100 GPU Thermal Hotspot
env_m100_02 real M100 marconi100 energy_anomaly 2 slurm, telemetry, docs, rbac, incidents M100 Node Power Anomaly
env_m100_03 real M100 marconi100 energy_anomaly 1 slurm, telemetry, docs, rbac, incidents M100 Rack Cooling Fault
env_m100_04 real M100 marconi100 incident_response 1 slurm, telemetry, docs, rbac, incidents M100 Node Down
env_m100_05 real M100 marconi100 job_failure 1 slurm, telemetry, docs, rbac, incidents M100 Job Failure Correlation
env_m100_06 real M100 marconi100 job_failure 1 slurm, telemetry, docs, rbac, incidents M100 Real OOM Job

Grounded environments in detail

These six bundles replay real Marconi100 telemetry and Slurm records. The scenarios below actually happened on the machine.

env_m100_01 — M100 GPU Thermal Hotspot

  • Scenario type: node_degradation
  • Snapshot timestamp: 2022-07-15T14:00:00Z
  • Roles supported: sysadmin, scientific_user
  • Categories: MON
  • Tasks targeting it: 2

Real-M100-grounded snapshot (ipmi_pub telemetry). GPU3 on node r3n7 is overheating: gpu3_core_temp ramps from ~45°C to ~88°C over the final 90 minutes, crossing the 84°C throttle threshold, while peer nodes stay 34–41°C. Sysadmin must identify the degraded node and GPU.

env_m100_02 — M100 Node Power Anomaly

  • Scenario type: energy_anomaly
  • Snapshot timestamp: 2022-07-20T11:00:00Z
  • Roles supported: sysadmin, facility_admin
  • Categories: ENERGY, MON
  • Tasks targeting it: 2

Real-M100-grounded snapshot (ipmi_pub telemetry). Node r10n4 total_power spikes to ~1400W (sustained 30 min) versus the ~644W rack baseline, with a corresponding rise in ps0/ps1 input power. Sysadmin/facility_admin must identify the offending node and quantify the overshoot.

env_m100_03 — M100 Rack Cooling Fault

  • Scenario type: energy_anomaly
  • Snapshot timestamp: 2022-08-02T15:00:00Z
  • Roles supported: facility_admin, sysadmin
  • Categories: ENERGY, MON
  • Tasks targeting it: 1

Real-M100-grounded snapshot (ipmi_pub + vertiv_pub telemetry). A cooling fault affects ALL of rack 4: the CRAC Supply_Air_Temperature climbs ~18→30°C, driving inlet temperature (ambient) on every node from ~22°C to ~33°C over the final 2 hours (vs a single-node hotspot). facility_admin must recognise the rack-wide signature and attribute it to the cooling unit, not a node.

env_m100_04 — M100 Node Down

  • Scenario type: incident_response
  • Snapshot timestamp: 2022-08-10T12:00:00Z
  • Roles supported: sysadmin, scientific_user
  • Categories: MON
  • Tasks targeting it: 1

Real-M100-grounded snapshot (ipmi_pub telemetry). Node r7n2 went dark at ~10:45 UTC: all of its IPMI telemetry stops after that point while peers keep reporting, and SLURM marks r7n2 'down'. Sysadmin must identify the unreachable node and the time it stopped reporting.

env_m100_05 — M100 Job Failure Correlation

  • Scenario type: job_failure
  • Snapshot timestamp: 2022-08-18T16:00:00Z
  • Roles supported: scientific_user, sysadmin
  • Categories: JOB, MON
  • Tasks targeting it: 1

Real-M100-grounded snapshot (ipmi_pub telemetry). Job 7798450 on node r2n5 FAILED at ~15:30 UTC; the node's total_power and CPU package power (p0_power, p1_power) collapse to near-idle at that moment as the process dies. The agent must correlate the failed SLURM job with the telemetry drop.

env_m100_06 — M100 Real OOM Job

  • Scenario type: job_failure
  • Snapshot timestamp: 2022-07-25T18:00:00Z
  • Roles supported: scientific_user, sysadmin
  • Categories: JOB, MON
  • Tasks targeting it: 1

Real-M100-grounded snapshot. The failed job is an ACTUAL Marconi100 OUT_OF_MEMORY record (job 66353) from the ExaData job_table. On node r5n3 the real ganglia mem_free falls from ~260 GB toward near-zero before the OOM kill, then total_power collapses to idle. The user must diagnose the OOM and act.

Adding an environment

See Adding an environment for the bundle layout, the manifest format, and the fidelity checks a new snapshot must pass. Bundles grounded in real facility data are especially valuable — if you operate a cluster and can publish a sanitised snapshot, please open a discussion.