Eye / camera control evaluation protocol¶

Status: programme-wide evaluation contract
Date: 2026-09-22
Related: TEST_PLAN.md, DATA_SHARING.md, PAPER_EVIDENCE.md, METRICS.md

This file answers four different questions that must not be collapsed into one benchmark:

1. Does the code behave deterministically? — automated CI.
2. Does it work on this operating system / computer / camera? — hardware QA.
3. Does it work for different people, repeatedly? — human evaluation.
4. Can the resulting evidence support a paper claim? — research protocol + ethics/consent gate.

A green CI run is not a usability study. One successful webcam demo is not cross-platform support. A public GitHub test report is not automatically research consent.

Evaluation ladder¶

E0 — pure CI: every relevant PR¶

Who: nobody; fully automated.
Hardware: virtual/runner only.
Paper eligible: yes for software-correctness claims, with provenance.

Run on ordinary CI:

• pure gaze calibration math;
• confidence/fallback policy;
• implicit calibration candidate promotion/rejection;
• display coordinate transforms;
• Head-Pointer mapping/deadzone;
• dwell state machine;
• face-switch hysteresis/hold/refractory;
• Gesture Chords;
• global ACTIVE/PAUSED/FAULTED safety state;
• stale-signal watchdog;
• privacy/schema validators;
• fake camera/MediaPipe adapters;
• fake pointer backends;
• portal D-Bus protocol fakes.

Goal: logic and policy regressions are caught without a camera.

E1 — OS matrix CI: every integration PR¶

Who: automated GitHub-hosted runners.
Hardware: no usable representative webcam; OS/runtime only.
Paper eligible: yes for build/install/platform-contract claims, not human performance.

Minimum matrix where supported:

• Ubuntu/Linux x86_64;
• Windows current hosted runner;
• macOS arm64/current runner;
• existing extra architecture legs where the repository workflow supports them.

Measure/record: - install success; - optional dependency resolution; - lazy import behavior; - tests; - platform backend import/probe; - permission/manifest static assertions; - result schema validation.

Do not claim real camera/pointer usability from E1.

E2 — deterministic synthetic trace replay¶

Who: automated.
Input: checked-in numeric traces with no face images.
Paper eligible: yes as regression/simulation evidence; label it synthetic.

Trace families:

• gaze drift / confidence drop;
• head jitter / motion / loss / recovery;
• dwell enter/reset/fire/re-arm;
• brow/mouth/blink blendshape-like normalized traces;
• talking-like mouth movement;
• face commit + simultaneous head-motion coupling;
• stale timestamps;
• topology change mid-session.

Primary purpose: make failures reproducible.

E3 — one-machine hardware smoke¶

Who: contributor/tester.
Hardware: real camera / OS / compositor.
Paper eligible: engineering evidence; human performance only if a research protocol separately covers the tester.

Scripted duration: approximately 10–20 minutes.

Minimum checks: 1. feature off -> camera unopened; 2. enable -> permission/status correct; 3. calibrate/recenter; 4. perform a small target/switch task; 5. deliberately lose camera/tracking; 6. verify no stale/repeated action; 7. pause/kill; 8. resume/re-arm; 9. disable -> camera released; 10. save privacy-safe result JSON/report.

E4 — cross-computer replication¶

Question: does the same OS/software work on different physical machines/cameras?

For each supported platform/session, collect results on at least: - two different computers; - preferably two camera models/integrated cameras; - at least one non-1.0 display scaling configuration where relevant.

Keep each result separate. Do not average machine latency across unlike hosts.

E5 — cross-person replication¶

Question: does the calibration/control work for different people on comparable hardware?

This is where participant-specific behavior matters: - gaze calibration; - head neutral pose/range; - face-switch threshold; - speech/face coupling; - comfort/fatigue.

If the results are intended for publication as human-participant evidence, E5 must run under the research/ethics and consent process in DATA_SHARING.md and PAPER_EVIDENCE.md.

Community QA reports remain useful even when they are not paper-eligible.

E6 — repeated-session / test-retest¶

Question: is the feature stable over time for the same person/computer?

Recommended: - 3 sessions on different times/days when feasible; - same software/config; - record whether calibration was reused or repeated; - report drift and false actions.

This distinguishes "worked once after calibration" from usable stability.

E7 — controlled human study¶

Only needed for claims about: - task completion; - speed/throughput; - error/false activation across people; - usability; - fatigue/comfort; - comparison with another input method.

This is a research study, not an issue-tracker QA task.

Before recruitment: 1. freeze protocol and hypotheses/analysis; 2. obtain the required institutional/local ethics determination; 3. prepare informed consent; 4. define data retention/withdrawal; 5. version the software/test task; 6. decide what will and will not be shared publicly.

Current ACM venues require authors to explain the relevant ethics/review context for human-participant work and comply with their research environment's requirements. Do not collect publication-intended human data first and ask the ethics question afterward.

Platform validation matrix¶

The programme should eventually cover these distinct environments, because "Linux" or "Windows" alone hides the relevant differences.

A contributor does not need to test every bucket. Each small issue should ask for one bucket.

Capability evaluation scripts¶

Gaze routing task¶

One trial: 1. show 4 large target panes/regions; 2. prompt "look at target N"; 3. user triggers the normal gaze-routing moment; 4. record intended target and resolved outcome.

Record: - correct; - wrong target; - fallback/no-route; - confidence bucket; - calibration age; - trial time.

Recommended QA block: 40 trials (10/target) after practice.

Primary metric: correct / all intended trials.

Also report wrong-target and fallback separately; fallback is safer than wrong-target and must not be collapsed into one "error".

Head-Pointer task¶

Use large generated targets so the task does not depend on private desktop content.

For each target: - target center; - target width; - pointer start; - movement start/end; - click outcome.

Record: - completion; - movement time; - miss/overshoot; - accidental click; - pauses/recenters; - tracking losses.

For research, the same data can support standard pointing analyses such as Fitts-style throughput, but the analysis plan must be frozen before using that as a paper outcome.

Face-switch task¶

Run blocks: - deliberate gesture block; - neutral/no-action block; - normal speaking block when mouth gestures are tested.

Record: - requested activation count; - detected count; - misses; - extra/false activations; - duration; - activation latency; - gesture name; - configured thresholds/hold/refractory.

Primary safety metric: false activations/hour, not only classifier accuracy.

Hands-free workflow task¶

Use a fixed, non-sensitive demo application/fixture.

Example: 1. focus a text field; 2. dictate a short provided phrase; 3. move to a large button; 4. commit; 5. trigger a safe navigation command; 6. simulate camera loss; 7. pause; 8. resume; 9. finish task.

Record: - task completion; - total time; - number of recovery actions; - number/type of fallbacks; - unintended actions; - whether keyboard/mouse intervention was required.

Replication structure¶

A result must identify a replication cell:

software version
× OS/session
× computer
× camera
× display topology
× person/session (when human)
× feature/config

Do not report "n=10 tests" when all 10 are the same person on the same laptop unless the claim is explicitly about repeated sessions.

For human studies, distinguish: - participants — independent people; - sessions — repeat observations; - trials — repeated actions inside a session.

Never treat 400 gaze trials from one person as n=400 participants.

What can be automated vs what cannot¶

Failure reporting¶

A failed validation is a valuable result.

Testers select one: - PASS; - FAIL — reproducible; - PARTIAL — some steps work; - BLOCKED — dependency/permission/hardware prevents test.

Do not silently discard failures from paper-quality datasets. The exclusion rule must be written before analysis, and technical invalidation (e.g. wrong version) must be distinguishable from a poor feature result.

Promotion gates¶

Experimental¶

Requires: - E0/E1 green; - E2 traces; - at least one E3 hardware smoke on each claimed platform; - privacy/permission/status complete; - global stop/recovery.

Recommended¶

Additionally requires: - E4 cross-computer evidence; - E5 cross-person evidence; - E6 stability where calibration matters; - false-action metrics; - accessibility/co-design evidence; - all research claims backed by paper-eligible data collected under the appropriate protocol.

Current no-code replication slots¶

The initial community QA matrix is now explicitly replicated:

• Windows: #428 + #429
• macOS: #430 + #431
• GNOME Wayland: #432 + #433
• KDE Wayland: #434 + #438
• X11: #435 + #439
• HiDPI/multi-monitor: #436 + #440
• test/retest same person/computer: #437

These issue pairs are engineering replication, not a substitute for the controlled human-study sample defined by #425.