Eye / camera control test and validation plan¶

This plan makes hardware-heavy accessibility work reviewable without requiring every contributor or CI runner to own the same camera, face, desktop, or eye tracker.

Test pyramid¶

Layer A — pure unit tests (required for every PR)¶

Inputs are numeric values and fake clocks; no camera, display server, or optional dependency.

Cover:

• calibration transforms;
• gaze confidence;
• route decisions;
• implicit calibrator update/reject/apply;
• head-pose → cursor mapping;
• dwell state machine;
• face-gesture hysteresis/debounce;
• gesture-chord resolution;
• consequence/confirmation policy.

A task that can be expressed as pure state but only has a webcam test is not done.

Layer B — adapter contract tests (required)¶

Inject fake cv2, MediaPipe, platform APIs or portal D-Bus responses, following the pattern already used by tests/test_gaze_mediapipe.py and tests/test_gaze_l2cs.py.

Required invariants:

• optional modules are imported lazily;
• disabled camera features do not probe/open a camera;
• one camera owner serves multiple consumers;
• camera close is idempotent;
• MediaPipe failure degrades without crashing dictation;
• no frame bytes are written;
• signal timestamps/confidence propagate correctly;
• a consumer never needs to import MediaPipe types.

Layer C — deterministic numeric trace replay (new)¶

Store small JSON fixtures containing derived numbers only, for example:

{
  "name": "brow-raise-single-event",
  "fps": 30,
  "samples": [
    {"t": 0.000, "brow_inner_up": 0.12, "confidence": 0.98},
    {"t": 0.033, "brow_inner_up": 0.14, "confidence": 0.98}
  ],
  "expected_events": []
}

Fixtures must not contain photos, face meshes, biometric templates, screenshots, gaze-labelled text, or identity metadata.

Use traces for:

• expression detector enter/exit thresholds;
• one-event-per-gesture;
• refractory period;
• head jitter / deadzone;
• dwell reset on movement;
• signal loss;
• pointer freeze during a face commit gesture;
• false-activation regression.

Layer D — platform contract tests¶

Pointer injection backends get a common suite:

• move relative;
• click press + release ordering;
• scroll;
• unsupported operation reports capability honestly;
• close/restart;
• failure does not repeat the previous action.

For XDG RemoteDesktop, fake the D-Bus exchange and assert:

• POINTER is requested only when a pointer consumer is enabled;
• existing KEYBOARD request remains intact when dictation injection needs it;
• restore token behavior is unchanged;
• pointer methods are never sent before permission/session start;
• refusal is surfaced and no action occurs.

Layer E — live hardware smoke test (required before changing feature tier)¶

Unit tests can prove control logic, not real vision accuracy. Hardware validation is separate and must report environment.

Minimum report:

Do not ask contributors to publish their face, video, disability, diagnosis, or typed content.

Capability-specific validation¶

Gaze routing¶

Already covered in CI:

• map raw gaze through calibration;
• route only above confidence threshold;
• fallback to focused window;
• preserve burst gaze snapshot for deixis;
• destructive confirmation;
• the same model options, camera device, numbers and decisions on the legacy and shared-source paths (#396; the four layers are listed in design/specs/eye-shared-perception.md).

Hardware metrics to collect:

• calibration error;
• wrong-window rate;
• no-route/fallback rate;
• drift over time;
• error by lighting/glasses/device when volunteered.

Do not use a single "accuracy %" without defining the target and denominator.

Implicit calibration¶

Unit gates:

• low-confidence sample rejected;
• high residual/outlier rejected;
• update cost does not grow with sample history;
• candidate cannot replace baseline when held-out error is worse;
• persisted baseline survives crash/failure;
• rollback returns exact prior map.

Hardware comparison:

1. run explicit calibration;
2. record baseline validation error;
3. collect natural clicks;
4. evaluate candidate on a held-out set;
5. report before/after error and elapsed session time.

Head pointer¶

Pure gates:

• inside deadzone -> zero delta;
• sign/direction mapping is correct;
• signal loss -> zero motion;
• movement after re-centre uses new neutral;
• dwell resets after moving beyond radius;
• exactly one click per dwell until re-armed;
• pausing suppresses both motion and dwell click;
• every state transition (each pair of held states stacked and unwound, close, recenter and dwell toggling from every state) acts on nothing while held and settles one frame on release, and any sample/control sequence keeps the no-motion, arming and shown-before-firing rules (tests/test_headpointer_state_matrix.py).

Daemon reachability (tests/test_headpointer_reachability.py, #404). This uses the real _build_pipeline(), with only the camera, the face model, the pointer sink and the clock faked:

• enabling [headpointer] alone is what builds a runner, and the shipped config builds none;
• a head turn reaches the platform PointerSink;
• face loss and camera failure each give zero motion;
• with dwell live, startup and face re-acquisition never click, and an aim-and-rest clicks once;
• headpointer/ imports only perception/signals.py, never camera-owning code.

The registry's import-closure honesty test cannot stand in for these tests. The daemon imports the package from places that build no pointer.

Hardware observations:

• target acquisition time for large targets;
• overshoot/correction count;
• accidental clicks/hour;
• whether speaking changes head control;
• whether face-gesture commit disturbs pointer position.

No universal throughput threshold should be invented before baseline data exists.

Face switch¶

Pure gates:

• neutral calibration establishes baseline without emitting an event;
• crossing enter threshold for less than minimum hold emits nothing;
• sustained deliberate gesture emits exactly one event;
• detector must cross exit threshold before re-arming;
• refractory interval suppresses bounce;
• low confidence emits nothing;
• ordinary blink trace does not trigger a default non-blink gesture.

Primary hardware metric:

false activations per hour, by configured gesture.

Also record missed deliberate activations. Both matter; optimizing only recall can make a switch unusable.

Wayland pointer¶

Live smoke:

• portal consent shown only when needed;
• restored session works after restart when compositor supports restore;
• motion/click reaches a normal Wayland app;
• denial leaves YazSes usable for ordinary dictation;
• no /dev/uinput or privileged helper is required for the portal path.

Hands-free bundle¶

Scenario test:

1. start from keyboard/mouse untouched;
2. target a text field;
3. dictate text;
4. move pointer to a large control;
5. select/commit;
6. recover from deliberately obscuring the camera;
7. pause the hands-free controls;
8. resume;
9. invoke a destructive action and verify confirmation;
10. quit/disable and verify camera release.

A bundle is not ready if any step requires an undocumented terminal command.

Performance / resource measurements¶

Record rather than guess:

• FaceLandmarker FPS;
• camera resolution;
• CPU utilization;
• memory delta;
• frame-to-signal latency;
• signal-to-pointer latency;
• dropped-frame rate;
• number of physical camera opens.

The architectural hard gate is one physical camera owner, not a specific FPS chosen without data.

Privacy and security assertions¶

Tests/review must confirm:

• feature disabled -> no camera open;
• camera frames remain in memory;
• no network egress is added by runtime perception;
• model download follows the existing explicit/lazy model path;
• logs contain state/error summaries, never raw landmarks or frame dumps by default;
• measurement export contains derived metrics only;
• face switch performs no identity/emotion inference;
• destructive actions still pass through existing confirmation policy.

CI commands¶

At minimum, task PRs run the narrow tests named in their issue plus:

uv run python -m pytest tests/ -q
uv run ruff check .
uv run mypy src/yazses

If the full repository currently has an unrelated known failure, the PR must show the narrow suite green and link the pre-existing failure rather than weakening/skipping a new test.

Tier promotion rule¶

A feature may move from planned -> experimental when:

• runtime path exists;
• hermetic CI exists;
• disable/failure paths are honest;
• privacy rules are satisfied;
• setup/troubleshooting docs exist.

Experimental -> recommended additionally requires:

• real-user/hardware evidence across more than one environment;
• false-activation/error measurements appropriate to the modality;
• no unresolved severe accessibility safety issue;
• defaults justified by data rather than one developer's setup.

Evaluation level cross-reference¶

This test plan describes how to test individual contracts. The programme-wide evidence ladder and cross-platform/person replication rules live in EVALUATION.md.

Use METRICS.md for exact field definitions and provenance. Use DATA_SHARING.md before asking a community tester to post a result.

Automated implementation work is tracked by #421–#424. Public no-code hardware slots are #428–#437. Research protocol/analysis work is #425–#427.

A public hardware report should default to study_mode=community_qa. It becomes study_mode=research only when collected under a named research protocol; changing the JSON label after the fact is not consent.