X4 synthetic calibration: do pilot forecasts and their intervals hit a known truth?

Refs: issue #976 (X4, hypothesis H2), parent #969. Evidence class: synthetic. Nothing on this page is a measurement of real coding agents. The empirical X4 result is E046 (point forecasts, one feasible held-out split, fragile). This page adds what E046 did not have: a check of the forecasters and of the 95% intervals they report against a ground truth that is known exactly, as a function of pilot size k.

Why a simulation, and what it can and cannot show

Real interval coverage needs the population’s true coverage C(M) on held-out data. Every SWE-bench split reachable today is already opened (E046 used them), and the preregistered fresh data (E048, PREREG_AGENT_VALUE_FORECAST_V2) does not exist yet. Intervals computed on opened splits would be post-hoc. So this study uses seeded simulators in which C(M) and the blind-spot floor 1 - C(M) are exact.

It can show whether the machinery (forecasters plus bootstrap intervals, imported unchanged from the frozen E046 code and forecast_intervals.py) is accurate and calibrated when the data are generated by the kind of model the forecasters assume or approximate. It cannot show that real agents behave like these simulators. A forecaster that fails here would be unfit for real data; one that passes has only cleared a necessary bar.

Design

Four scenarios, parameters fixed before any forecaster output was inspected (SCENARIOS in forecast_interval_calibration.py): rasch_moderate, rasch_hard_tail (wider difficulty spread, bigger blind spot), rasch_unsolvable (10% of tasks no agent can solve) and family_clustered (6 model families that fail together on a per-task shock).
Each scenario draws 6 independent populations of 60 agents over 300 tasks. Truth is the exact union coverage of all 60 agents; the floor is its complement, so the floor error equals the coverage error.
Per population and per k in {3, 5, 10, 20}: 30 random pilots. Each forecaster is scored by bias, mean absolute error (MAE), the share of pilots within 0.03, and the empirical coverage of its 95% percentile-bootstrap interval (100 replicates), resampling either tasks (task) or the pilot’s agents (agent).
Forecasters: independence, beta_binomial, incidence (Chao et al. 2014, Chao2-style) and pilot_only (observed coverage). The IRT/Rasch forecaster is excluded because it needs the full outcome matrix (see forecast_intervals.py).
Declared before the first full run: tolerance 0.03; an interval is calibrated if pooled coverage is at least 0.90 against a nominal 0.95; the verdict at k = 10 is falsified if no forecaster/scheme has MAE <= 0.03 in any scenario, supported if some forecaster/scheme has MAE <= 0.03 and a calibrated interval in every scenario, mixed otherwise. No scenario or threshold was adjusted after the run.

Result at k = 10: `mixed`

The recorded decision is mixed. The beta-binomial point forecast meets the 0.03 tolerance in three of four scenarios, but no forecaster/scheme is both within the tolerance and calibrated in any scenario at k = 10. Exactly one (scenario, forecaster/scheme) cell reaches the 0.90 coverage bar, and its point forecast misses the tolerance.

scenario	forecaster/scheme	bias	MAE	within 0.03	95% interval coverage	mean width
rasch_moderate	`beta_binomial/task`	-0.0143	0.0163	0.84	0.478	0.033
rasch_moderate	`beta_binomial/agent`	-0.0143	0.0163	0.84	0.367	0.101
rasch_moderate	`incidence/task`	-0.0488	0.0514	0.34	0.628	0.102
rasch_moderate	`incidence/agent`	-0.0488	0.0514	0.34	0.506	0.229
rasch_hard_tail	`beta_binomial/task`	-0.0144	0.0283	0.59	0.761	0.082
rasch_hard_tail	`beta_binomial/agent`	-0.0144	0.0283	0.59	0.483	0.128
rasch_hard_tail	`incidence/task`	-0.0622	0.0700	0.21	0.678	0.163
rasch_hard_tail	`incidence/agent`	-0.0622	0.0700	0.21	0.256	0.229
rasch_unsolvable	`beta_binomial/task`	+0.0474	0.0482	0.18	0.267	0.058
rasch_unsolvable	`beta_binomial/agent`	+0.0474	0.0482	0.18	0.939	0.117
rasch_unsolvable	`incidence/task`	-0.0461	0.0504	0.30	0.722	0.135
rasch_unsolvable	`incidence/agent`	-0.0461	0.0504	0.30	0.478	0.220
family_clustered	`beta_binomial/task`	-0.0004	0.0086	0.97	0.672	0.020
family_clustered	`beta_binomial/agent`	-0.0004	0.0086	0.97	0.550	0.068
family_clustered	`incidence/task`	-0.0333	0.0367	0.48	0.817	0.087
family_clustered	`incidence/agent`	-0.0333	0.0367	0.48	0.517	0.192

Reading the table (k = 10, pooled over 6 populations x 30 pilots):

Point accuracy. The beta-binomial MAE is below 0.03 in rasch_moderate, rasch_hard_tail (0.0283, close to the line) and family_clustered. It fails in rasch_unsolvable (0.0482, bias +0.0474). A likely, untested explanation: tasks no agent can solve look to the method-of-moments fit like very hard tasks that more agents would eventually reach, so it over-forecasts. The Chao2-style incidence forecaster under-forecasts in every scenario (bias -0.033 to -0.062) and exceeds 0.03 in all four at k = 10.
Interval calibration fails. Among the forecasters in the table, the task bootstrap, which models only task-sampling noise, covers 0.267 to 0.817 for these forecasters, far below 0.95. The agent bootstrap is wider but does not fix it (0.256 to 0.939). The one high figure, 0.939 for beta_binomial/agent in rasch_unsolvable, belongs to a forecast whose bias is +0.0474, so the interval is wide enough to reach a truth the point estimate misses; it is not evidence of calibration.
Two failure patterns, not separated by this study. Where the bias is large (rasch_unsolvable, incidence everywhere) a percentile interval around a biased estimator cannot cover the truth. Where the bias is negligible (family_clustered beta-binomial, bias -0.0004) coverage is still only 0.550 to 0.672, so those bootstrap intervals are also too narrow for the pilot-to-pilot variation. Both need a different interval construction, not a wider nominal level.
independence looks accurate only where truth is near saturation (it forecasts 1.0 and C(M) is 0.98 in two scenarios). It fails where the floor is about 12% (MAE 0.12), and its interval has zero width, so coverage is 0. Treat its “accurate in two scenarios” entry in the report as an artefact.
pilot_only (the pilot’s own coverage) has MAE 0.089 to 0.147 at k = 10, a similar magnitude to the 0.0965 gap E045 measured on real data.

Exploratory, outside the declared rule: k = 20

H2 is stated for k <= 10, so k = 20 does not enter the verdict. For orientation, the Chao2-style task bootstrap at k = 20:

scenario	forecaster/scheme	bias	MAE	within 0.03	95% interval coverage	mean width
rasch_moderate	`incidence/task`	-0.0116	0.0179	0.83	0.945	0.065
rasch_hard_tail	`incidence/task`	-0.0199	0.0296	0.59	0.917	0.126
rasch_unsolvable	`incidence/task`	-0.0116	0.0201	0.74	0.939	0.106
family_clustered	`incidence/task`	-0.0091	0.0155	0.89	0.917	0.054

At k = 20 this forecaster is below 0.03 in all four scenarios and its interval coverage is 0.917 to 0.945, so in simulation it is both accurate and calibrated once the pilot is twice H2’s size. That observation was made after the declared rule was fixed and is a hypothesis for real data, not a finding.

Controls

tests/test_x4_forecast_interval_calibration.py checks that the scoring cannot be fooled:

an oracle that returns the truth scores MAE 0 and coverage 1;
a constant-guess estimator and a sound estimator shifted by -0.2 are both caught on MAE, bias and tolerance rate, and the constant one on coverage;
the verdict rule is exercised on hand-built inputs for falsified, supported and both mixed routes, and a broken estimator cannot win it end to end;
the committed report’s pooled rows and decision are recomputed from its recorded per-population cells, its constants and scenario parameters match the module, its analysis digests match the current source files, and the first two pilots of an audit cell are regenerated from the seed;
every row of the two tables above is checked against the committed report.

Limits

Synthetic. Rasch-like generators with exact truth favour forecasters built on a per-item difficulty distribution. Real submissions are not exchangeable agents and share scaffolds and base models (E047).
Finite populations. Truth is the 60-agent pool, as in E046. The latent large-population floor is not scored.
Pilots in a population share a truth, so pooled coverage is not an average of independent trials. The report keeps per-population coverage and the worst population’s coverage (min_population_coverage).
One coverage threshold (0.90) and one level (0.95) were declared. Other interval constructions (for example bias-corrected or analytic iNEXT intervals) were not tried and may calibrate better; trying them now and keeping the best would be tuning on this run.
The real-data question stays open until E048 provides fresh submissions. Rasch intervals remain unsupported.

Reproduce

PYTHONPATH=. python experiments/forecast_interval_calibration.py \
  --out experiments/results/X4-synthetic-interval-calibration.json

About ten minutes of single-core CPU; standard library only. The output is experiments/results/X4-synthetic-interval-calibration.json.

This site is open source. Improve this page.