Skip to content

Eye / camera control metrics dictionary

This file is the canonical name/meaning list for machine-readable evaluation results. A metric name must mean the same thing on Windows, macOS and Linux.

The evaluation harness should reuse the provenance principles already used by paper/benchmark/_common.py: a number without the machine/software/config that produced it is not a reproducible measurement.

Result envelope

Every result artifact should contain:

schema_version
study_mode
timestamp
software
machine
os_session
display
camera
feature
config
protocol
metrics
privacy

No field contains a face image, raw video, transcript, window title, username, home path or email.

Study mode

Required enum:

  • ci — automated CI;
  • synthetic — deterministic generated/replayed trace;
  • community_qa — public engineering test, not research-use by default;
  • research — collected under a named research protocol/ethics/consent process.

This field prevents a future analysis script from accidentally mixing QA with human-study data.

Software provenance

Required: - YazSes version; - git commit SHA when available; - Python version; - perception backend/model/version; - relevant package versions; - schema version; - result-generator version/command.

Prefer a config hash plus a whitelist of relevant non-sensitive feature settings.

Do not dump the entire user config.

Machine provenance

Collect automatically where possible:

  • OS name/version;
  • kernel/build;
  • architecture;
  • CPU model;
  • logical CPU count;
  • RAM GB;
  • session/compositor: X11 / GNOME Wayland / KDE Wayland / Windows / macOS;
  • system load at start/end for performance runs.

Do not collect: - hostname; - login username; - serial number; - MAC address; - full device UUID.

Display provenance

Required when gaze/pointer coordinates matter:

  • display count;
  • each display logical rectangle;
  • resolution;
  • scale factor;
  • primary display indicator;
  • topology fingerprint (pseudonymous/hash of non-secret geometry fields).

Optional for paper/user-entered: - physical diagonal/width; - approximate viewing distance.

Physical dimensions + viewing distance are needed only when converting pixels to angular error. Do not require them for ordinary QA.

Camera provenance

Collect: - backend/device label sanitized to model/product class where possible; - integrated vs external; - capture width/height; - nominal/measured FPS; - selected camera index only as local configuration, not a stable identity; - model/FaceLandmarker asset version.

Do not collect: - camera serial number; - USB serial; - a photo/frame; - raw face landmarks by default.

Runtime/performance

Recommended: - source FPS median / P5; - dropped/invalid sample rate; - frame-to-derived-signal latency P50/P95 if measurable; - derived-signal age at action P50/P95; - CPU process utilization; - RSS delta; - number of physical camera opens; - permission/startup time where relevant.

Interpret performance per machine. Do not average latency across unlike hosts into one "YazSes latency".

Gaze metrics

Counts: - trials; - correct_target; - wrong_target; - fallback_no_route; - invalid_tracking.

Rates: - correct-target rate; - wrong-target rate; - fallback rate.

Calibration: - explicit calibration points; - calibration/validation RMSE px; - held-out RMSE px; - optional angular error degrees only when display geometry + viewing distance are available; - calibration age/time since fit; - implicit sample count; - candidate-vs-baseline delta; - topology valid/stale.

Session: - error by time block to quantify drift; - reacquisition time after tracking loss.

Never report one generic "gaze accuracy %" without defining whether it means coordinate error, correct-window rate, or tracker availability.

Head-Pointer metrics

Per target/trial: - success; - movement time ms; - target distance/amplitude; - target width; - miss count; - overshoot/correction count if instrumented; - accidental click; - tracking-loss count/duration; - pause count; - recenter count.

Aggregate: - task completion rate; - median/P95 movement time; - miss rate; - accidental clicks/hour; - optional effective pointing throughput under a preregistered analysis.

Do not infer motor ability or medical condition from these numbers.

Face-switch metrics

Per block: - gesture; - intended activations; - detections; - misses; - false/extra activations; - duration seconds; - activation latency distribution; - configured enter/exit thresholds; - hold duration; - refractory interval; - confidence threshold.

Primary aggregate: - false activations/hour; - recall/intent detection rate; - median activation latency.

Keep neutral and normal-speaking blocks separately identifiable.

Semantic grounding metrics

Use for #441–#445 and any future "look at this field/button/item" refinement.

Per trial: - target source and coarse target ID (synthetic/non-sensitive); - eligible semantic candidate count; - candidates with usable bounds; - actionable candidate count; - candidates spatially plausible after coarse-target filtering; - optional candidate count after an intent/role hint; - result state: grounded, ambiguous, unresolved; - correctness when ground truth is available: correct / wrong; - stale/coordinate-space-mismatch reason where applicable; - resolver latency when performance is being studied.

Aggregate: - grounded-correct rate; - grounded-wrong rate; - ambiguity rate; - unresolved/abstention rate; - semantic coverage (targets with at least one usable candidate); - actionable coverage; - candidate-count distribution before/after spatial filtering; - candidate-count distribution after optional intent hint.

Safety rule: wrong target and abstention are not the same error. A resolver that declines an ambiguous target may be safer than one that always returns an element.

Do not store private UI text, screenshots, raw accessibility objects, email/document content or window titles containing personal information. Controlled task fixtures should use synthetic labels.

Hands-free workflow metrics

  • task completion yes/no;
  • total completion time;
  • unintended action count;
  • recovery action count;
  • fallback modality count/type;
  • keyboard/mouse intervention required yes/no;
  • pause/kill successfully invoked yes/no;
  • recover-from-camera-loss success yes/no.

Research-only subjective measures may include participant-reported: - ease/confidence; - comfort; - fatigue/change over session; - preference.

Use a frozen questionnaire/scale in the study protocol; community QA should keep subjective questions short and optional.

Human-study identifiers

Research datasets need linkage without identity.

Use: - random participant_id assigned by the study; - session_id; - trial number.

Do not put GitHub username, email or real name in the analysis dataset.

The consent/contact record, if required, is stored separately from measurements with access limited to the research team.

Missing data

Use explicit missing/null + reason: - not supported; - not measured; - permission denied; - tracking unavailable; - participant stopped; - technical invalidation.

Do not encode missing as zero.

Exclusions

Any result excluded from analysis must retain: - exclusion reason; - whether exclusion was technical/protocol/user withdrawal; - whether exclusion rule was defined before analysis.

Never delete poor performance merely because it is inconvenient.