Eye control: what a webcam can honestly know¶

Updated 2026-08-07. Part of the research series.

Eye tracking sounds solved — the Apple Vision Pro selects UI elements with your gaze, and Tobii has sold precise infrared trackers for years. But those systems use dedicated IR hardware: a measurement study of the Vision Pro found 1.11° and 0.93° accuracy in two test setups (Huang et al., 2024). The interesting scientific question for everyone else is: how far can you get with the webcam already in your laptop bezel?

The physics sets the budget¶

At a normal 50–70 cm viewing distance, 1° of gaze error is roughly 1 cm on screen. That single conversion factor decides what interactions are possible:

---
config:
  themeVariables:
    xyChart:
      backgroundColor: "transparent"
      titleColor: "var(--md-default-fg-color)"
      xAxisLabelColor: "var(--md-default-fg-color)"
      yAxisLabelColor: "var(--md-default-fg-color)"
      xAxisTitleColor: "var(--md-default-fg-color)"
      yAxisTitleColor: "var(--md-default-fg-color)"
      xAxisTickColor: "var(--md-default-fg-color--lighter)"
      yAxisTickColor: "var(--md-default-fg-color--lighter)"
      xAxisLineColor: "var(--md-default-fg-color--lighter)"
      yAxisLineColor: "var(--md-default-fg-color--lighter)"
      plotColorPalette: "#8f6fd6"
---
xychart-beta
    title "Gaze error by hardware (degrees; lower is better)"
    x-axis ["Dedicated IR", "Webcam + calibration", "Research model, in-dataset", "Webcam, no calibration"]
    y-axis "Angular error (degrees)" 0 --> 5
    bar [1.11, 2.9, 3.92, 4.17]

Two conclusions fall straight out of the table:

1. Caret-level gaze typing on a webcam is not honest in 2026. A text caret is millimetres tall; the signal is centimetres wide.
2. Zone- and window-level targeting is comfortably inside the budget — and Google's Look to Speak shipped a 3-way webcam gaze selector on-device, proving the coarse regime is robust enough for production (Google).

That is why Glance-Type is deliberately look-to-pane, never look-to-caret.

A licensing trap most projects miss¶

Nearly every pretrained appearance-based gaze model is non-commercial by data contamination: the popular training sets (Gaze360, ETH-XGaze, MPII) carry CC BY-NC-style terms, so even MIT-licensed code produces encumbered weights. In our survey, MediaPipe's face/iris landmarks were the only clean-licensed route to a gaze signal — which is why it is YazSes's default backend, with the heavier L2CS-Net strictly opt-in. If you are building on this research, check the licence of the dataset, not just the repository.

How the pipeline works¶

flowchart LR
    A[Webcam frame\nin-RAM only] --> B[MediaPipe FaceLandmarker\n478 landmarks incl. iris]
    B --> C[Normalised iris offset\nper eye]
    C --> D{Left/right eyes\nagree?}
    D -- "divergent → low confidence" --> H[Fall back to\nfocused window]
    D -- "agree → confident" --> E[Affine calibration map\ngaze → screen point]
    E --> F[Smallest window\nunder the point]
    F --> G[Route dictation /\nresolve 'this']

The confidence gate is a 2026-08 addition with a story worth telling: the two eyes estimate the same gaze independently, so their disagreement is a free, per-frame landmark-quality signal (blur, extreme head pose, occlusion). Before we measured it, the backend reported every frame as fully confident — the config knob existed, but gated nothing. Fake precision is worse than honest coarseness, because a user can work around a system that admits when it doesn't know.

Frames are processed in RAM during a hold and are never written to disk or sent anywhere — see the privacy statement.

Deixis: the oldest idea in multimodal HCI, finally on a desktop¶

In 1980, Richard Bolt's Put-That-There demoed pointing plus speech resolving pronouns (Bolt, SIGGRAPH '80). The modern revival is in AR and VR: GazePointAR (CHI 2024) substitutes the gazed object into the query before the language model sees it (Lee et al.), and augmenting speech transcripts with gaze and pointing improved a language model's coreference-resolution accuracy by 26.5% over a speech-only baseline in a 12-participant VR study (Bovo et al.).

Curiously, we found no open-source desktop implementation of gaze+speech deixis — the field moved to headsets and left the desktop niche empty. So we built it: in command mode, "close this", "focus that window", "minimize that" act on the window your gaze snapshot picked, and destructive actions confirm first because coarse gaze can misroute:

flowchart TD
    S[Spoken command with a\ndemonstrative: 'close this'] --> P{Gaze snapshot\nfrom hold-start}
    P -- "gaze-routed target" --> Q{Destructive?}
    P -- "no target" --> N[Ignore — never\ntype it literally]
    Q -- "yes (close)" --> C[Actionable toast:\nClose it / Keep it]
    Q -- "no (focus, minimize)" --> X[Act immediately]
    C -- confirmed --> X

The interaction grammar follows what a CHI 2026 scoping review of gaze and speech converged on: gaze grounds and disambiguates; a second cheap modality commits (Khan et al.) — Vision Pro's pinch, Talon's pop sound, MAGIC-style cursor warping (Tian et al.). In YazSes, the hold-to-talk key is the pinch.

Open questions¶

We would genuinely like community measurements on these. Each is a self-contained study; bring results to the discussions:

1. Implicit calibration from mouse clicks. A click is ground truth for "where you looked ~100 ms ago"; the literature reaches 2.9° with zero explicit calibration (Sugano et al.). How stable is that across laptop lids, docking stations, and glasses-wearers? Nobody has published longitudinal desktop data.
2. Eye-agreement confidence vs ground truth. Our per-frame confidence is a proxy. How well does left/right iris-offset divergence actually correlate with gaze error across faces, lighting, and webcams?
3. What is the right zone granularity? 3×3 grid, window bounding boxes, or editor split-panes? Is there a measurable sweet spot where routing errors stop annoying users?
4. Wayland. Compositors forbid external window focus — the whole desktop half of gaze routing is X11-only today. Can the emerging libei/portal stack express "focus the window at (x, y)" safely?

References¶

Evidence grade: measured (peer-reviewed measurement), vendor (claimed by the maker), secondary (review or write-up).

1. Huang, Z., Zhu, G., Duan, X., Wang, R., Li, Y., Zhang, S., Wang, Z. "Measuring eye-tracking accuracy and its impact on usability in Apple Vision Pro." arXiv:2406.00255, 2024. arXiv — measured (1.11° and 0.93° in two setups)
2. Sugano, Y., Matsushita, Y., Sato, Y., Koike, H. "Appearance-based gaze estimation with online calibration from mouse operations." IEEE Transactions on Human-Machine Systems 45(6), 2015. doi:10.1109/THMS.2015.2400434 — measured
3. "Methodological recommendations for webcam-based eye tracking: a scoping review." Research Methods in Applied Linguistics, 2025. ScienceDirect — secondary
4. Kong, A., Ahuja, K., Goel, M., Harrison, C. "EyeMU interactions: gaze + IMU gestures on mobile devices." ICMI '21, 2021. doi:10.1145/3462244.3479938 — measured
5. Abdelrahman, A. A., Hempel, T., Khalifa, A., Al-Hamadi, A. "L2CS-Net: fine-grained gaze estimation in unconstrained environments." arXiv:2203.03339, 2022. arXiv — measured (in-dataset)
6. Google. "Look to Speak" (Android accessibility app using webcam gaze selection). blog.google — vendor
7. Bolt, R. A. "Put-that-there: voice and gesture at the graphics interface." SIGGRAPH '80, 1980. doi:10.1145/800250.807503 — measured
8. Lee, J., Wang, J., Brown, E., Chu, L., Rodriguez, S. S., Froehlich, J. E. "GazePointAR: a context-aware multimodal voice assistant for pronoun disambiguation in wearable augmented reality." CHI '24, 2024. doi:10.1145/3613904.3642230 — measured
9. Bovo, R. et al. "Augmenting speech transcripts of VR recordings with gaze, pointing, and visual context for multimodal coreference resolution." arXiv:2509.08689, 2025. arXiv — measured (26.5% improvement, 12 participants, VR)
10. Khan, A. A., Weidner, F., Rhee, J., Abdrabou, Y., Bianchi, A., Velloso, E., Gellersen, H., Newn, J. "Gaze and speech in multimodal human-computer interaction: a scoping review." CHI '26, 2026. doi:10.1145/3772318.3791662 — secondary
11. Tian, X. et al. "3D-MAGIC: expanding MAGIC pointing to stereoscopic displays." CHI EA '26, 2026. doi:10.1145/3772363.3798896 — measured

See also: how to set up Glance-Type, the research index for how this compares to voice and muscle input, and muscle & brain control, where gaze dwell becomes a switch input for people who cannot press a key.