Hands-free task inventory and evidence gaps¶

Date: 2026-11-05 · Baseline: main at a241a77b (v2.48.1 line) · Issue: #813 (NEXT-A11Y-01), epic #790 Status: Research note (not a decision). ADR-030 freeze unchanged. Scope: read-only audit of this repository. No src/ change, no new capability, no participant, no recording, no external source (none was needed). Siblings: #814 co-design gates · #815 arbitration · #816 stop/unintended-activation threat model · #817 measurement · #818 Wayland/AT limits · #819 go/no-go proposal. Delivery stays in #102 / #692.

1. Question and answer in one paragraph¶

Open question of #813: can a person complete editing, app navigation, dialog response and stop/recovery workflows with existing voice/gaze/head/switch capabilities? Answer from the repository alone: not demonstrated, and for four task families demonstrably not complete. [Verified-in-code] Most single capabilities exist and have synthetic tests. [Verified-in-docs] No hands-free capability has a hardware record or a user study in the validation registry. [Verified-in-code] Several complete tasks have a first step that needs a keyboard, a pointer or a terminal, and the way back from a pause or a restart has no hands-free route at all.

2. Method, labels and evidence scale¶

Read: AGENTS.md, docs/roadmap.md, docs/validation-matrix.md, docs/features.md (generated registry), docs/how-to/hands-free-setup.md, hands-free-voice-stop.md, gaze-look-to-pane.md, design/eye-control/* (README, VALIDATION_MATRIX, EVALUATION, RISK_REGISTER, validation-slots.json, generated/validation-coverage.md), ADR-v2-148, and src/yazses/{handsfree,gaze,facegesture,headpointer,gesture,mousegrid,pilot,windowctl,commands,eyeeval}. Nothing was run on a camera or a desktop.

Labels: [Verified-in-code] read in src/ or tests/ (or absence found by grep; a grep absence is not proof of absence). [Verified-in-docs] stated in a repository document, not independently re-checked. [Assumption] my inference. [Unknown] not determinable from the repository.

Evidence level per task (highest present): none < synthetic (named test file; runs in CI on the OS matrix, [Verified-in-docs] E20 in docs/validation-matrix.md) < hardware record < real-user study. A synthetic test proves a rule on fabricated input; it cannot show that anything was seen, heard or reachable on a real desktop ([Verified-in-docs] design/eye-control/EVALUATION.md, E0 versus E3-E7). No row below reaches hardware record.

Tier words are those of docs/features.md: core, recommended (on by default), recommended, optional, experimental, planned (not wired). [Verified-in-docs]

3. Inventory of complete hands-free tasks¶

"Hands-free entry" asks whether the first step of the task can be taken by a person with no keyboard and no mouse. NO means that step needs a key, a pointer or a terminal. CAM means it depends on a working camera. BURST means it needs a dictation burst to have been started first (task 3).

4. Reachability by platform¶

[Verified-in-docs] unless marked. Y shipped, - not available, ? not established.

[Assumption] On Windows and macOS the only documented commit is dwell, and its overlay precondition on those OSes is not stated in the hands-free setup page; I did not trace it.

5. Gaps ranked by "user stranded" severity¶

Severity = how long a person without keyboard and mouse is left without any way to proceed, and whether anything tells them.

1. No hands-free resume (tasks 10, 12, 14, 15). [Verified-in-docs] Resume is only yazses handsfree resume or the Settings page; there is deliberately no resume phrase and no resume gesture action (facegesture/switches.py ACTIONS has none). Every pause, every daemon restart, every update and every reboot ends in PAUSED. [Verified-in-code] Only a hold-only face switch re-arms itself, and only after a stale-signal disarm, not after a user pause or restart. Stranded until a second person or a physical key helps.
2. Opening the microphone has one hands-free path, and it is the camera (task 3). [Verified-in-code] If the camera is lost or the face switch is paused or FAULTED, the user cannot start a burst; wakeword is unwired, autostop needs a hotkey tap. [Verified-in-docs] The voice stop phrase and every voice command (tasks 4-9) inherit that dependency: "once hands-free is paused the face switch no longer starts a burst".
3. Click commit is missing where the overlay is not drawn (task 7). [Verified-in-docs] On native Wayland (GNOME default) the pointer may move but nothing clicks; the preset binds no select/confirm/cancel. The documented precision fallback (mousegrid) is experimental and X11/macOS only.
4. Setup, calibration and updates have no hands-free entry (tasks 1, 13, 14). [Verified-in-docs] The documents say so for calibration but do not mark the whole path as assisted setup. [Assumption] Acceptable for a caregiver-assisted first install; it is a stranding risk for self-service updates and setting changes.
5. No window/app switching off X11 (task 8). [Verified-in-docs] No voice route on Wayland, Windows, macOS; the user depends on the pointer reaching a taskbar.
6. Unexpected dialogs are unmodelled (task 9). [Verified-in-code, grep] Only generic routes exist; no scenario, no wrong-target measure.
7. Stop-everything and camera-off are not hands-free (task 11). [Verified-in-docs] Pause keeps the camera pipeline available; the user may believe it is off. Privacy expectation, not a stranding risk.
8. Lock/suspend behaviour unknown (task 15). [Unknown] Not found in the daemon; could be benign (stale gate) or could re-arm on a stale pose.
9. Zero hardware or participant evidence for any of the above. [Verified-in-docs] See section 6.

5a. Workflow map: where each task's chain depends on another¶

install / enable / calibrate / restart / resume (tasks 1, 2)   <- keyboard, pointer or terminal
        |
        v
camera healthy + face switch ARMED (task 3)  <-- single hands-free root for every spoken task
        |-- burst --> dictate (3) --> correct (4) --> undo (5) --> navigate (6)
        |-- burst --> window (8) / dialog keys (9) / pilot, grid click (7) / stop phrase (10)
        '-- camera --> head pointer (7, 8) / chord stop (10)
camera lost / pause / restart / update / reboot  --> PAUSED or FAULTED (10, 12, 14, 15)
        '-- resume: terminal or Settings page only      <- the exit edge needs hands

[Verified-in-code/docs] for each edge individually; the single-root picture is my synthesis. [Assumption] A user with a physical switch or a biosignal board (biosignal, experimental) has a second root; that route has its own open evidence plan (ThinkGear note) and is not analysed here.

6. Coverage by existing validation slots¶

[Verified-in-code] design/eye-control/validation-slots.json (schema 2) holds 16 cells; generated/validation-coverage.md reports 16 PLANNED, 0 READY, 0 with any report, 0 independent hosts, 3 of 8 packs registered.

Not covered by any pack, slot or fake scenario: tasks 1, 4, 5, 6, 9, 13, 14 and the lock/suspend half of 15. [Verified-in-docs] docs/validation-matrix.md is a platform/install matrix (dictation, injection, start/stop) and contains no hands-free row.

7. Proposed next evidence¶

7a. Synthetic, participant-independent (no hardware; repository-only; each needs maintainer approval under the ADR-030 freeze, because test code is not src/ but a new scenario harness is scope)¶

7b. Human gates (cannot be closed by Claude or CI; no recruitment, recording or upload is authorized by this note)¶

• H1 Hardware T0/T2/T3/T4/T7 runs on each OS/session (community QA, #428 family; T2-T4 and T7 need cells registered first).
• H2 Independent check that the stop phrase and the face switch behave on real voices and faces, with false activations per hour and exposure reported (R-03, R-13, R-23). [Verified-in-docs] Risk register rows.
• H3 Co-design and consent review before any participant study (#814, #419); separate community QA from publishable human-participant research (#813 boundary text).
• H4 Fatigue, precision and task-completion measurement with people who actually cannot use a keyboard or mouse (#817).
• H5 Real screen-reader / assistive-technology pass on the Hands-free page (current tests use Qt's offscreen platform).
• H6 Maintainer decision on whether an inventory finding becomes a proposed ADR amendment (#819); nothing here authorizes implementation.

7c. Proposed stop/recovery acceptance model (for #816/#817 to adopt or reject)¶

A task row would count as "complete hands-free" only if all of the following held in a recorded run. Each is a proposal, not an existing gate.

1. Entry: the first and every later step is taken by a modality the user can produce; the run log names it.
2. Independent stop: at least one stop route works with the camera lost and the pointer misbehaving (ADR-v2-148 already requires one independent of the pointer; the synthetic proof is test_handsfree_stop_from_faulted.py). [Verified-in-code]
3. Explicit, hands-free re-arm: the way back from PAUSED/FAULTED needs no key, pointer or terminal, and its first frame never acts ([Assumption] today only the second half is tested; I did not read every safety test).
4. No replay: after any interruption no earlier pose, switch event or gaze target fires.
5. Unintended actions are counted per hour of exposure with the exposure stated; zero in a short run is reported as an upper bound, not as zero.
6. Intervention count: any keyboard, mouse or terminal use during the run is recorded and makes the task "assisted".

Failure modes to look for: a stop route that needs the failed sensor; a resume that fires a held gesture; a guard that fires so often it is switched off (AGENTS.md rule 9); a fallback that raises consequence (handsfree/recovery.py forbids it, [Verified-in-code]).

Alternative approaches the evidence does not yet separate: voice-only always-listening start (privacy and false-wake cost, wakeword unwired); a physical or biosignal switch as the root; a caregiver-assisted setup path declared as such; leaving resume manual on purpose because an automatic one is itself a risk (ADR-v2-148 recovery hierarchy item 5, voice-stop page). [Verified-in-docs] The last is a deliberate design position, so gap 1 is a tension to weigh in #819, not an oversight to patch.

7d. Privacy and review notes for any later human study¶

[Verified-in-docs] The hands-free pipeline keeps frames in memory and reports only states, names and ages (handsfree/observability.py). A study that records faces, voices or biosignals is a separate data transfer governed by design/eye-control/DATA_SHARING.md, GOVERNANCE.md and AGENTS.md (no real user audio or voiceprints to a third-party coding agent). Community QA reports and research participation must stay separate. Unintended-action risk to the participant (a stuck click or open microphone) needs a stop condition written before the first session. None of this is started here.

7e. Falsifying evidence for the answer in section 1¶

The "not demonstrated" conclusion is falsified by a registry record showing a person completing tasks 3, 10 and 15 on a stated OS/session and version without a key, pointer or terminal. Nothing of that kind exists today. [Verified-in-docs]

8. Explicit non-claims¶

• This note reports no user study, no participant data and no measurement of any person.
• It makes no clinical claim and no accessibility-compliance claim (no WCAG, Section 508, EN 301 549 or similar statement). The one contrast-ratio test it cites checks a formula on rendered pixels, not conformance.
• It does not say any capability works or fails for a real user; it says what the repository shows and does not show.
• Synthetic tests and CI are not hardware evidence and are not counted as such.
• Statements labelled [Assumption] or [Unknown] must not be quoted as findings.
• The freeze is unchanged: no feature, ADR acceptance, release or roadmap commitment follows from this note.