Spec: Biosignal Validation Protocol — Measuring Phantom Activations and Hit Rate on Real Hardware¶

Goal¶

Every number in yazses.biosignal (detector thresholds, quality-gate limits, the budget, the governor schedule) is a design choice validated on synthetic signal only (ADR-v2-152, "Evidence and what it does not show"). This protocol is how that changes. It specifies how a person with real hardware measures, on their own machine, how often the switch fires when they did not mean it and how often it fires when they did, how those measurements are combined across people and boards without moving any raw signal, and how much evidence is needed before anyone may say anything about them.

It produces upper bounds and lower bounds with their method stated. It does not produce an "accuracy".

Non-goals¶

• not a clinical study, a medical evaluation, or a claim about any person's health or condition; YazSes is not a medical device, and nothing here is reviewed by an ethics board;
• not a way to collect raw signal, recordings, calibration profiles or thresholds (those never leave the machine; see "Reporting back");
• not a tuning loop: the protocol measures a frozen configuration, it does not search for one;
• not a replacement for yazses biosignal evaluate on a labelled recording, which scores the detectors offline; this protocol measures the whole chain, live, in the activities where mistakes actually happen;
• not an identity, emotion, fatigue or health measurement of any kind.

Definitions¶

These are used identically in the field-test output, in the ADR gates and in the user pages. They follow yazses.biosignal.fieldtest, which is the authority for how each number is computed.

Who and what hardware¶

• Participants are adults using their own hardware on their own machine, who have read the consent text below. Intended users of the feature (people who cannot press or hold a key reliably, or whose hands are busy) are the population that matters most and are asked for specifically, but are never asked for health information; the protocol records none.
• Hardware recorded (written by hand beside the shared file, free text, no photographs): board and BrainFlow id, or the source kind for a non-BrainFlow device (lsl: with the publishing app, thinkgear:, bitalino: with the sensor per input, myo:), or "serial YESP sensor"; electrode type, placement in words (for example "left forearm, flexor side"), sampling rate, the mains_hz setting, operating system, the yazses version, and which of squeeze, double blink and jaw clench were enabled (the standard session cues all three gestures; a switch that was off cannot fire and its row is to be ignored). The report file does not carry these.
• A configuration is a unit of evidence. The same person on a different board, a different electrode site or a re-seated electrode (with a fresh calibration) is a different configuration, and results are never pooled across configurations without also being reported per configuration.
• Minimum for the ADR gates: see ADR-v2-152, "Falsifiable acceptance criteria" (participant and configuration counts are set there, not here, so the numbers live in one place).

Preconditions¶

1. yazses biosignal doctor reports no failing check for the source, and its closing Verdict: line is read before starting. A failing doctor is a result in its own right: record it, do not run phases through it.
2. For any source other than synthetic or a BrainFlow board already in yazses biosignal boards, yazses biosignal check-source <source> --seconds 30 passes on the participant's machine. A failing clause (gaps, rate drift, stalls) is recorded as a result and the session does not start: the integrity gate would close the switch during the phases and the rates would describe the transport, not the detectors.
3. A calibration profile made for this configuration exists (yazses biosignal calibrate). The wall-clock time from starting calibrate to a written profile, the number of attempts, and whether it succeeded are noted by the participant (gate A3). The command does not time the person, only the data.
4. Defaults are the defaults. A participant may run with their own [biosignal] values, but then the result describes those values.
5. For the evaluation rounds the participant does not edit thresholds, the budget or the mapping. A retune is allowed only between rounds (see "Tuning rounds").

Phases¶

yazses biosignal field-test walks the participant through these in order, prompting on screen. The tool advances on signal seconds, not on wall time, so a stream with gaps simply takes longer. The no-intent phases come first and the gestures last, so a tired participant still gives the phantom evidence that matters most. Durations are the tool's design defaults, sized for a short session; they are not derived from any measured variance, and --minutes stretches or shrinks the whole sequence proportionally.

A default run is about 7.5 minutes of no-intent phases plus 45 seconds per gesture (about ten minutes for all three). The weights the tool applies to build its workday estimate (rest 0.20, speech 0.25, chew 0.05, hands busy 0.35, movement 0.15) are an illustrative activity mix for a desk-bound day, not data; the per-phase numbers are the evidence and the weighted figure is a convenience.

Phases that do not apply to a configuration are still run and are simply uninformative (a forearm electrode with no jaw channel makes speech and chew informative mostly for the squeeze; a headband makes them central). The jaw-clench switch is off by default and is measured only when the participant turns it on; running it once with the speech guard on and once off measures the unguarded rate the guard is meant to remove. rest is deliberately first, before the participant has watched the live monitor, so that watching the signal does not change how they sit.

The tool's own verdict is not the ADR gate¶

The command ends with a verdict: usable, promising, needs_tuning, do_not_enable or inconclusive. It is a per-person advisory rung, built from the weighted pooled phantom bound, with hit rate judged on the point estimate (at least 80% of cues) because with the default eight cues a Wilson lower bound above 80% is unattainable (eight of eight gives about 68%). It exists to help one person decide whether to leave a switch on. The ADR gates are stricter and are read on the per-phase numbers and the conservative interval ends; a usable verdict from one session is neither sufficient nor necessary for a gate. The standard session cues every gesture whether or not that switch is on, and the verdict looks at every conclusive gesture phase, so a switch that was off (and so could not fire) can drag the headline toward do_not_enable; in that case read the per-phase rows of the switches that were on, not the headline.

Tuning rounds¶

Threshold changes are the easiest way to produce a good-looking result that means nothing. The rules:

• A round is one or more complete field-test sessions on a frozen configuration.
• Between rounds a maintainer or participant may change thresholds, the budget or the mapping, and must write down what changed and why.
• Never evaluate on the data used to choose the change. A configuration is judged only on a round that ran after it was frozen. A result from a round whose configuration was chosen by looking at that round's own counts is discarded from the aggregate.
• The ADR's kill criteria count rounds: a limited number of retuning rounds per configuration is allowed before it is recorded as failing.

Aggregation: FieldTestAggregate¶

A session produces a report (report_to_dict, schema 1): per phase the usable and excluded seconds, the phantom count and per-kind counts, the cues scored, voided and hit, the extra activations and the cue-to-event latencies (to 0.1 s); plus the verdict, its reasons, the budget, the workday figures and the caveat. FieldTestAggregate (with report_from_dict, which recomputes every derived number from the counts so a stored file cannot disagree with itself) merges reports. What it does, and what anyone aggregating by hand must also do:

1. Counts and seconds add; rates and bounds are recomputed. Never average per-session rates. A pooled rate is total phantoms over total usable exposure, with the bound computed on the totals. An inconclusive phase in every session can become conclusive once its usable seconds add up.
2. Like with like. Phases are matched by name. Reports that counted different trigger kinds are refused, and so is a phase that is a gesture phase in one report and not in another.
3. The aggregate has no notion of participant, configuration or session identity. It does not deduplicate (adding the same file twice counts it twice) and it cannot show a per-participant row. Whoever merges must therefore: build one aggregate per participant per configuration and report those rows beside the pooled one; add each file once; and report the worst participant next to the pool. A pooled bound can hide one participant for whom the switch is unusable, and a pooled pass with a failing individual is read as "not yet" for any claim about people.
4. Configurations stay separate in the report. A cross-configuration pool is an additional row, never a replacement for the per-configuration ones.
5. Exposure is the denominator. A session in which the switch was mostly gated or suspended contributes little usable time and so a wide bound; it is not discarded and not up-weighted.
6. The bound's method is stated, every time: one-sided 95% Poisson upper bound on frozen thresholds, not an accuracy. The workday figure additionally applies a Bonferroni split across the phases used.

Minimum evidence for a claim¶

Claims are about the configuration and the code revision measured. A later change to a detector, the quality gate or the governor invalidates the evidence for the changed behaviour until it is re-run.

Reporting back without raw signal¶

yazses biosignal field-test --json prints, and --out <file> also writes (owner-only, never overwriting an existing file), the session report. It contains counts, whole-second durations, interval bounds, the verdict and its reasons, the phase labels and the cue-to-event latencies. It contains no sample values, no recordings, no amplitudes, no thresholds, no profile contents and no event timestamps. It also contains no board, placement or version information: those are written by the participant beside it.

• The participant reads the file before sharing. It is a small JSON document meant to be read.
• Sharing is a deliberate act by the participant: attaching the file to an issue or a discussion by hand. The command never sends anything, and no yazses component reads the file back from a network location.
• Do not attach a recording, a profile or a screenshot of the live signal. If a maintainer needs to see signal to understand a failure, that is a separate, explicit, opt-in contribution governed by the research-consent rules in ADR-v2-150, not part of this protocol.
• Maintainers publish only aggregates: pooled rows, and per-participant rows under pseudonyms (P1, P2, ...), and only when there are enough participants (the ADR gate's count) that a row does not identify a person. Free-text hardware notes are edited to remove names and anything about health.
• A participant can withdraw at any time before publication by asking for their file to be removed from the aggregate. After publication only the pooled numbers remain, which cannot be separated.

Consent text (shown by the command before it starts)¶

The command narrates, before the first prompt, that nothing is typed or acted on, that no signal is saved, and that only counts of false activations are reported. The fuller statement for anyone running this as part of a collection effort: what is counted (activations by activity, exposure, hits); what is not stored (signal); that nothing leaves the machine unless the participant shares the JSON by hand; that this is not a medical test and says nothing about their health; that they can stop at any time (Ctrl-C ends the session and reports what was gathered, marked as stopped early) and that stopping early produces a smaller sample, not an error.

Stress tests¶

Field time cannot cover failure modes that happen once a month. Each of these is injected on purpose, once with the synthetic source or a replay (cheap, repeatable, required) and, for the ones that need it, once on real hardware (required before the ADR's stuck-mic gate can be called met). The pass condition is always the safe behaviour, and for every case a hold, if one is open, is closed exactly once.

A stress run fails on any stuck-mic event, any activation emitted from an untrusted window, or any exception that reaches the daemon. A fix restarts the whole list on the new code.

Relationship to the HCI × Hugging Face EEG/EOG feasibility protocol¶

A separate research track (epic #692, draft PR #711, design/hci-huggingface/EEG_EOG_FEASIBILITY_PROTOCOL.md on branch feat/hci-research-metrics; not yet on main, so it is cited by name and not linked) asks the same question from the research side: can EEG or EOG give a coarse, deliberate activation whose false-activation behaviour is comparable to the switches YazSes already studies? The two documents are meant to be read together. Nothing was merged between the branches; this section is the agreed mapping, written on this side only.

Open question (for the maintainer, shared with that protocol): the minimum exposure before an hourly rate is printed at all. Their provisional value is 600 s of scored exposure per rate. Here the report already carries the upper bound that makes a short session honest, but it still prints a rate figure for a phase with 30 s of usable signal. Adopting their rule would mean showing the count and the exposure below the minimum and the bounded rate only above it. That is a small change to format_report and report_to_dict in fieldtest.py, not made in this round; it should wait until the number is frozen on their side so the two stay one definition.

What the existing suite pins, and what it cannot¶

The synthetic test suite (tests/test_biosignal_*.py) is the regression net for the logic; it does not substitute for any row above on real hardware. Which rows it covers at the logic level is tracked in design/specs/biosignal-pipeline.md ("Test checklist: what the suite pins, and what remains").

Open questions¶

1. Is a default session (about two minutes of rest, one of chew) a useful unit at all for a rare-event process? Reaching an upper bound of 6 per hour in one phase takes about half an hour of usable signal in it, so the evidence comes from many sessions or stretched ones (--minutes). The first reports decide whether a longer default or a per-phase length option is needed; the CLI has no protocol-file option today, only protocol_from_dict in Python.
2. Should the aggregate carry a participant and a configuration key so that per-participant rows and de-duplication stop being the merger's job? Today they are, and a mistake there is silent.
3. Is a prompted speech phase representative of speech during real dictation, when the speaker is also holding the switch? Likely not entirely; the live ledger over many days is the cross-check.
4. The field test acts on nothing, so it cannot observe a stuck microphone. Is the stress list plus the live status enough, or does the protocol need an instrumented dictation phase?
5. How many participants with the intended-user profile can be reached, given that the protocol asks for no health information and so cannot confirm who they are? Self-description in the issue is the only route, and it is voluntary.