Recovering which CPU cores are disabled on each individual processor die, from the out-of-band sensor telemetry supercomputers already collect. Reproducible results for 1,962 POWER9 sockets of CINECA's Marconi100.
For each month, enumerate the metric=pX_coreY_temp partitions and count samples per
(node, socket, core). A core index is present for a socket if it emits any sample in the
window. The disable map of socket s is then M[s,k] ∈ {0,1} over the twelve slices
k = ⌊core/2⌋.
That is the whole method. Everything else is making it reliable.
The key argument is that the BMC does not renumber active cores compactly. Were that so, every socket would report indices 0–15 and there would be exactly one observed pattern. Instead we observe arbitrary 16-element subsets of {0,…,23} and 443 distinct patterns, so the index is fixed to the core rather than to its rank among the survivors; IBM’s core numbering ties it to a position on the die (see Findings).
That argument is internal. It is also confirmed externally, by two collectors that never touch the IPMI sensor family:
The single OS-level anomaly in the month used for thermal work is itself a confirmation: exactly one node (145) reports one core short, and over the same two days the BMC loses exactly one core sensor on that node before the socket returns with a different map. Two collectors with nothing in common flag the same node, and the same number of cores, in the same 48 hours. Across the whole record, all nine departures of the OS count from 128 are matched by a BMC loss of that many cores.
Both collectors confirm the count; which positions are disabled rests on the non-compact indexing above.
Each has a detector, and one of them caught a real defect in our own pipeline.
| Failure mode | Detector |
|---|---|
| Partial ingestion — extraction silently truncates, so absent metrics look like harvested cores | A completeness gate refusing to emit a month unless all 48 metrics are present |
| Compact renumbering — a BMC that renumbers survivors would yield nothing but a constant | Arbitrary subsets and 443 patterns; a renumbering BMC produces exactly one |
| Per-metric collector faults — a dropped stream mimics deconfiguration | Odd active-core counts, which no baseline map produces (a single-core field deconfiguration can; the OS CPU count tells them apart); and control sensors on the same socket keeping full cadence |
| Socket-tag swaps — exchanged p0/p1 tags silently mix two dies in a longitudinal study | A mirror test, which identifies 19% of steady-state map changes as relabelling, which moves no die |
The first of these was found the hard way: three months had been written from incomplete extractions, one missing all 24 sensors of socket p1, which would have read as a socket with no active cores at all. Any pipeline that infers “hardware absent” from “data absent” must gate on extraction completeness, or partial ingestion masquerades as hardware failure.
p{0,1}_core{0..23}_temp
partitions are extracted, behind the completeness gate above.Days are UTC calendar days throughout; DATA.md
documents every derived table.
The clean-day threshold is the load-bearing discretionary choice, so its effect is stated plainly: moving it from 0.50 to 0.99 discards 18% of socket-days and changes nothing that matters — the same 1,962 sockets, 442–443 patterns, an identical 61.2% slice-0 rate and an identical lot contrast.
Cores on a socket share workload and inlet air, producing a dominant common mode. We remove it by subtracting the per-timestamp cross-core mean, double-centre, and compute residual correlations at the sensors’ native 20 s cadence — no resampling.
That last point matters more than it sounds. An earlier version aligned onto a one-minute grid, a step introduced to work around an unrelated bug and never revisited. It is not neutral: coarser averaging suppresses high-frequency noise and inflates every correlation, within-slice more than cross-slice, growing the contrast from +0.063 at 20 s to +0.140 at 15 min. We quote the native-resolution figure, which is the most conservative.
Because common-mode removal forces each row of the residual matrix to sum to approximately zero,
negative values at large index gaps are partly mechanical. We therefore rely on contrasts at
identical index distance and on ordering tests, which the constraint cannot bias: it treats every
core alike, so it can rescale an ordering but not create one. Every thermal interval resamples the
49 racks, from per-socket matrices (analysis/thermal_socket_build.py). Equal index distance is not
equal die distance: across a quad boundary, index neighbours sit in opposite bands of the die.
For the fleet census the series is decimated in time rather than in nodes: 20 s slots are counted from 00:00 UTC on the month’s first day and every fifth slot is kept, which lowers the sample count without smoothing and so, unlike a coarser alignment grid, does not inflate correlations.
Every socket has exactly four harvested slices, so the constraint induces negative dependence and a naive independence null is invalid. We use the curveball algorithm, which preserves both row sums (4 per socket) and column sums (per-slice marginals) while randomising association structure, and assert both on every draw.
Null draws are made exchangeable with the observed matrix by Besag and Clifford’s parallel method:
40 global curveball trades lead from the observed matrix to a hidden one, and each of 100,000 draws is
40 further trades from it. Every Monte-Carlo p-value, (b+1)/(B+1), is then valid however slowly the
chain mixes; mixing diagnostics and a cross-check with the sequential sampler are in
analysis/curveball_null.py. (An earlier implementation executed only trades between sockets that
share no harvested slice; its null moments agree with the corrected sampler’s.)
Next: Findings · Reproduce · Limitations