Eye / camera control test and validation plan¶

This plan makes hardware-heavy accessibility work reviewable without requiring every contributor or CI runner to own the same camera, face, desktop, or eye tracker.

Test pyramid¶

Layer A — pure unit tests (required for every PR)¶

Inputs are numeric values and fake clocks; no camera, display server, or optional dependency.

Cover:

• calibration transforms;
• gaze confidence;
• route decisions;
• implicit calibrator update/reject/apply;
• head-pose → cursor mapping;
• dwell state machine;
• face-gesture hysteresis/debounce;
• gesture-chord resolution;
• consequence/confirmation policy.

A task that can be expressed as pure state but only has a webcam test is not done.

Layer B — adapter contract tests (required)¶

Inject fake cv2, MediaPipe, platform APIs or portal D-Bus responses, following the pattern already used by tests/test_gaze_mediapipe.py and tests/test_gaze_l2cs.py.

Required invariants:

• optional modules are imported lazily;
• disabled camera features do not probe/open a camera;
• one camera owner serves multiple consumers;
• camera close is idempotent;
• MediaPipe failure degrades without crashing dictation;
• no frame bytes are written;
• signal timestamps/confidence propagate correctly;
• a consumer never needs to import MediaPipe types.

Layer C — deterministic numeric trace replay (new)¶

Store small JSON fixtures containing derived numbers only, for example:

{
  "name": "brow-raise-single-event",
  "fps": 30,
  "samples": [
    {"t": 0.000, "brow_inner_up": 0.12, "confidence": 0.98},
    {"t": 0.033, "brow_inner_up": 0.14, "confidence": 0.98}
  ],
  "expected_events": []
}

Fixtures must not contain photos, face meshes, biometric templates, screenshots, gaze-labelled text, or identity metadata.

Use traces for:

• expression detector enter/exit thresholds;
• one-event-per-gesture;
• refractory period;
• head jitter / deadzone;
• dwell reset on movement;
• signal loss;
• pointer freeze during a face commit gesture;
• false-activation regression.

Layer D — platform contract tests¶

Pointer injection backends get a common suite:

• move relative;
• click press + release ordering;
• scroll;
• unsupported operation reports capability honestly;
• close/restart;
• failure does not repeat the previous action.

For XDG RemoteDesktop, fake the D-Bus exchange and assert:

• POINTER is requested only when a pointer consumer is enabled;
• existing KEYBOARD request remains intact when dictation injection needs it;
• restore token behavior is unchanged;
• pointer methods are never sent before permission/session start;
• refusal is surfaced and no action occurs.

Layer E — live hardware smoke test (required before changing feature tier)¶

Unit tests can prove control logic, not real vision accuracy. Hardware validation is separate and must report environment.

Minimum report:

Do not ask contributors to publish their face, video, disability, diagnosis, or typed content.

Capability-specific validation¶

Gaze routing¶

Already covered in CI:

• map raw gaze through calibration;
• route only above confidence threshold;
• fallback to focused window;
• preserve burst gaze snapshot for deixis;
• destructive confirmation.

Hardware metrics to collect:

• calibration error;
• wrong-window rate;
• no-route/fallback rate;
• drift over time;
• error by lighting/glasses/device when volunteered.

Do not use a single "accuracy %" without defining the target and denominator.

Implicit calibration¶

Unit gates:

• low-confidence sample rejected;
• high residual/outlier rejected;
• update cost does not grow with sample history;
• candidate cannot replace baseline when held-out error is worse;
• persisted baseline survives crash/failure;
• rollback returns exact prior map.

Hardware comparison:

1. run explicit calibration;
2. record baseline validation error;
3. collect natural clicks;
4. evaluate candidate on a held-out set;
5. report before/after error and elapsed session time.

Head pointer¶

Pure gates:

• inside deadzone -> zero delta;
• sign/direction mapping is correct;
• signal loss -> zero motion;
• movement after re-centre uses new neutral;
• dwell resets after moving beyond radius;
• exactly one click per dwell until re-armed;
• pausing suppresses both motion and dwell click.

Hardware observations:

• target acquisition time for large targets;
• overshoot/correction count;
• accidental clicks/hour;
• whether speaking changes head control;
• whether face-gesture commit disturbs pointer position.

No universal throughput threshold should be invented before baseline data exists.

Face switch¶

Pure gates:

• neutral calibration establishes baseline without emitting an event;
• crossing enter threshold for less than minimum hold emits nothing;
• sustained deliberate gesture emits exactly one event;
• detector must cross exit threshold before re-arming;
• refractory interval suppresses bounce;
• low confidence emits nothing;
• ordinary blink trace does not trigger a default non-blink gesture.

Primary hardware metric:

false activations per hour, by configured gesture.

Also record missed deliberate activations. Both matter; optimizing only recall can make a switch unusable.

Wayland pointer¶

Live smoke:

• portal consent shown only when needed;
• restored session works after restart when compositor supports restore;
• motion/click reaches a normal Wayland app;
• denial leaves YazSes usable for ordinary dictation;
• no /dev/uinput or privileged helper is required for the portal path.

Hands-free bundle¶

Scenario test:

1. start from keyboard/mouse untouched;
2. target a text field;
3. dictate text;
4. move pointer to a large control;
5. select/commit;
6. recover from deliberately obscuring the camera;
7. pause the hands-free controls;
8. resume;
9. invoke a destructive action and verify confirmation;
10. quit/disable and verify camera release.

A bundle is not ready if any step requires an undocumented terminal command.

Performance / resource measurements¶

Record rather than guess:

• FaceLandmarker FPS;
• camera resolution;
• CPU utilization;
• memory delta;
• frame-to-signal latency;
• signal-to-pointer latency;
• dropped-frame rate;
• number of physical camera opens.

The architectural hard gate is one physical camera owner, not a specific FPS chosen without data.

Privacy and security assertions¶

Tests/review must confirm:

• feature disabled -> no camera open;
• camera frames remain in memory;
• no network egress is added by runtime perception;
• model download follows the existing explicit/lazy model path;
• logs contain state/error summaries, never raw landmarks or frame dumps by default;
• measurement export contains derived metrics only;
• face switch performs no identity/emotion inference;
• destructive actions still pass through existing confirmation policy.

CI commands¶

At minimum, task PRs run the narrow tests named in their issue plus:

uv run python -m pytest tests/ -q
uv run ruff check .
uv run mypy src/yazses

If the full repository currently has an unrelated known failure, the PR must show the narrow suite green and link the pre-existing failure rather than weakening/skipping a new test.

Tier promotion rule¶

A feature may move from planned -> experimental when:

• runtime path exists;
• hermetic CI exists;
• disable/failure paths are honest;
• privacy rules are satisfied;
• setup/troubleshooting docs exist.

Experimental -> recommended additionally requires:

• real-user/hardware evidence across more than one environment;
• false-activation/error measurements appropriate to the modality;
• no unresolved severe accessibility safety issue;
• defaults justified by data rather than one developer's setup.

Evaluation level cross-reference¶

This test plan describes how to test individual contracts. The programme-wide evidence ladder and cross-platform/person replication rules live in EVALUATION.md.

Use METRICS.md for exact field definitions and provenance. Use DATA_SHARING.md before asking a community tester to post a result.

Automated implementation work is tracked by #421–#424. Public no-code hardware slots are #428–#437. Research protocol/analysis work is #425–#427.

A public hardware report should default to study_mode=community_qa. It becomes study_mode=research only when collected under a named research protocol; changing the JSON label after the fact is not consent.