Skip to content

Synthetic cross-session EEG evaluation harness (spec)

All numbers here are synthetic, not human EEG. Every signal comes from an authored generator in scripts/hci_eeg_cross_session_synthetic.py. Nothing on this page is evidence that YazSes can decode brain activity. ADR-030 rules out a production EEG interface, and nothing here changes that.

Status: research draft. The script and its tests were written on 2026-11-06 and first run the same day. The "expected outputs" below were hypotheses written before that run; the first-run results record which held. H2 was refuted.

Why this exists

The closed predecessor, #696 (closed 2026-11-03), produced a synthetic motor-imagery stress replay. Its results page records that this replay did not test cross-session or participant-disjoint evaluation:

  • design/hci-huggingface/EEG_MI_SYNTHETIC_STRESS_RESULTS.md:141 says seeds 0-9 against 10-19 is "not a cross-session or participant-disjoint split";
  • :157 lists "cross-session independent data" as not run;
  • :174 gives the next step: freeze a participant- or session-disjoint split, never split overlapping windows of one recording across train and test, and fit preprocessing, calibration and model on training data only.

This harness turns :174 into code that can be tested before any real dataset is touched. Its job is to show that the evaluation pipeline cannot leak. Classification accuracy is not the point. No open issue tracks this follow-up (:180). Reopening #696 or opening a new issue is a maintainer decision.

What the script does

Part Function Contract
Generator generate(seed, n_subjects, n_sessions) Pure and seeded (numpy.random.default_rng). Each subject has an authored mu amplitude, mu frequency and imagery suppression ratio. The last subject is an authored non-responder with suppression 0.97. Each session gets its own gain (log-normal, σ 0.35), mu-amplitude drift (σ 0.20) and noise level. Each block gets an amplitude jitter (σ 0.25), which correlates the windows inside it. Blocks are 4 s and alternate rest and imagery, 12 per session. Windows are 1 s with a 0.25 s hop and never cross a block boundary.
Features bandpower_features Log mean Hann-windowed power in 8-13 Hz and 16-26 Hz on C3 and C4, giving 4 features.
Normalisation fit_standardizer z-score whose mean and scale come only from the rows passed in. evaluate passes only that fold's training rows.
Classifiers fit_lda, predict_1nn Two-class LDA with 0.10 trace shrinkage. Its posterior comes from the equal-covariance logit. 1-nearest-neighbour serves as the high-capacity classifier that window-overlap leakage misleads. numpy only, with no scikit-learn.
Splits split_by_subject, split_by_session Leave-one-subject-out, and leave-one-session-index-out pooled over subjects.
Negative control split_leaky_windows Random window-level 3-fold split. Marked negative_control: true in the report.
Leak audit overlapping_window_pairs Counts train/test pairs from the same (subject, session) recording that share any sample.
Per-subject failure evaluate Balanced accuracy per subject. Any subject below 0.60 goes in failed_subjects.
Activation replay replay_activations, activation_metrics Causal rule: posterior ≥ 0.70 for 2 consecutive windows, with a 2 s refractory period. A fire on a rest window counts as a false activation. The replay reports false activations per hour of synthetic rest exposure, the exposure in seconds and an exact one-sided 95% Poisson upper bound (poisson_upper_bound).
Report run, main JSON with synthetic: true, a notice, limitations, every authored parameter, rounded metrics and a report_sha256 over the canonical JSON. --out writes UTF-8 with LF line endings.

Expected outputs (hypotheses, not results)

These are predictions from the generator's construction. None has been observed.

  1. H1. Disjoint splits do not overlap. overlapping_train_test_window_pairs is 0 for subject_disjoint and session_disjoint, and greater than 0 for leaky_window_split. This holds by construction and is pinned by tests.
  2. H2. The leaky split inflates accuracy. For 1-NN, leaky balanced accuracy minus session-disjoint balanced accuracy is at least 0.05 at seed 0. The expected cause is that a test window's nearest neighbour is a 75%-overlapping window from the same block. For LDA, which has 4 features and few parameters, the inflation is expected to be small. That would itself be worth reporting: leakage hurts high-capacity models most. This is pinned on seed 0 only. If it fails, check the generator (block jitter, overlap) before assuming the classifier is wrong.
  3. H3. The authored non-responder fails. Under the subject-disjoint LDA, the non-responder's balanced accuracy is near 0.5, and it appears in failed_subjects.
  4. H4. Session gain drift lowers session-disjoint accuracy. Session-disjoint LDA accuracy is expected to sit below the leaky split. Training normalisation does not remove another session's gain shift. The size of the drop is not predicted.
  5. H5. The false-activation rate is uncertain. At the defaults every one of the 18 recordings (6 subjects × 3 sessions) is held out once, so the synthetic rest exposure is 18 × 6 rest blocks × 4 s = 432 s, or 7.2 minutes. Even with zero false fires, the exact 95% upper bound is therefore about 25 per hour (3.0 × 3600 / 432). The point rate is not a safety estimate.
  6. H6. The report is deterministic. run(0) == run(0) within one process. report_sha256 is not pinned across platforms, because FFT and BLAS rounding can differ in the last bits before the 6-decimal rounding.

How to validate

uv run python -m pytest tests/test_night_r04_eeg_cross_session.py -v
uv run ruff check scripts/hci_eeg_cross_session_synthetic.py tests/test_night_r04_eeg_cross_session.py
uv run python scripts/hci_eeg_cross_session_synthetic.py --seed 0 --out /tmp/eeg-xs-0.json

Record H1-H6 as confirmed or refuted on this page, with the observed numbers and the date. Do not tune the generator to make H2 pass. A tuned generator would make H2 hold by design and show nothing.

First-run results (2026-11-06)

Synthetic numbers only. Linux x86_64, CPython 3.12.3, numpy 2.4.6, --seed 0 at the defaults (6 subjects × 3 sessions, 2808 windows), unless a row says otherwise.

Hypothesis Outcome Observed
H1 Confirmed Overlapping train/test window pairs: 0 for subject_disjoint, 0 for session_disjoint, 9526 for leaky_window_split.
H2 Refuted 1-NN inflation (leaky − session-disjoint balanced accuracy) is 0.852564 − 0.828348 = 0.024217, below the predicted 0.05. LDA inflation is 0.006054. Across seeds 0-5 the 1-NN inflation was positive every time (0.024 to 0.092) but reached 0.05 in only 3 of 6 seeds (1, 2, 3). The leak is real but smaller than predicted at these settings. The generator was not tuned; the test now pins only the direction (inflation > 0 at seed 0).
H3 Confirmed Subject-disjoint LDA balanced accuracy for the non-responder (subject 5) is 0.500, and it is the only entry in failed_subjects. The other subjects range 0.878 to 0.994. Subject 5 was the only failed subject in seeds 0-5.
H4 Confirmed, effect tiny Session-disjoint LDA 0.872863 vs leaky 0.878917. At seed 3 the order reversed (LDA inflation −0.001781), so this is not a robust drop.
H5 Confirmed Rest exposure 432 s. Session-disjoint LDA replay: 2 false activations, 16.7 per hour, 95% upper bound 52.5 per hour. Subject-disjoint: 1, 8.3 per hour, upper bound 39.5 per hour. Hits: 88 of 108 imagery blocks in both.
H6 Confirmed within one process run(0) == run(0) holds (test passes). report_sha256 remains unpinned across platforms.

Non-goals

  • No real EEG, devices, BrainFlow boards or ThinkGear runtime. #827 (ThinkGear G0) is open, and nothing here depends on it.
  • No dataset download, network access or new dependency. numpy is already a base dependency.
  • No model, threshold or calibration intended to transfer to people. No accuracy claim.
  • No YazSes runtime integration, config key, CLI command or IPC method, under the ADR-030 freeze.
  • No calibrated confidence. The LDA posterior is only as calibrated as the equal-covariance assumption, and the generator violates that assumption on purpose across sessions.
  • No artifacts, unknown mental states or head motion. Those stay with scripts/hci_eeg_mi_synthetic.py and scripts/hci_eegeog_replay.py.

License and ethics review checklist for any later public dataset

Nothing has been downloaded. Before anyone fetches a public dataset (for example a PhysioNet motor-imagery recording or a MOABB-indexed dataset), a reviewer completes this list in writing:

  • Dataset landing page cited (not a file URL), with version, DOI and date read. Candidates to check: PhysioNet EEG Motor Movement/Imagery, https://physionet.org/content/eegmmidb/1.0.0/, and the MOABB dataset catalogue, https://moabb.neurotechx.com/docs/datasets.html. Neither page was re-read on 2026-11-06. The reviewer must verify the licence terms from the landing page and must not rely on this page.
  • Licence permits research use, derived features and publication of aggregate results. Record attribution and share-alike terms, and any credentialed or data-use-agreement requirement.
  • Every MOABB dataset is covered individually. MOABB is an index and loader, and each underlying dataset carries its own licence and consent terms.
  • Participant consent covers secondary analysis of this kind. No re-identification attempts. No linking to other data.
  • Session structure is documented: which recordings share a participant and which share a day. Without this, a session-disjoint split cannot be built.
  • Raw data stays outside the repository. AGENTS.md rule 11 keeps private tiers private, and no participant data goes to third-party tools.
  • The split is frozen and committed before any model is fitted. The H1 overlap audit is re-run on the real metadata.
  • Human-subject work stays separately gated (#425/#427/#814, per EEG_MI_SYNTHETIC_STRESS_RESULTS.md:178).

Provenance

  • Predecessor: #696, state CLOSED, closedAt 2026-11-03T23:11:48Z (read-only gh issue view, 2026-11-06).
  • Related prior scripts: scripts/hci_eeg_mi_synthetic.py (single-recording MI stress replay) and scripts/hci_eegeog_replay.py.
  • Base commit: main at 8a72bbc6.