Skip to content

State of the Art: Gaze, EMG, BCI, and Multimodal Input (2025–2026)

Date: 2026-08-07 · Tier: design/ — public engineering research Companion: local/offline STT engine SoA · voice-dictation market landscape · ADR-v2-129: Killer Features 10x · the v2 cognitive layer

Feasibility ratings below are scoped to integration into a CPU-only, offline, Linux-first Python daemon.

1. Webcam-based gaze tracking

  • License structure: nearly every pretrained appearance-based gaze model carries non-commercial taint through its training data (Gaze360, ETH-XGaze CC BY-NC-SA, MPII), even where the code itself is MIT-licensed — L2CS-Net's published weights included. MediaPipe is the cleanly-licensed option among the ones surveyed. (The PyPI package named l2cs is unrelated to the L2CS-Net model — a naming trap worth flagging for anyone else evaluating this space.)
  • Realistic accuracy: 1° of gaze angle corresponds to roughly 0.9–1.2 cm at 50–70 cm viewing distance. Uncalibrated webcam gaze runs ~4° error; with person-specific calibration, ~2–3°; implicit calibration from mouse clicks reached 2.9° (Sugano et al., IEEE 7050250); EyeMU measured 1.7 cm on-phone. Dedicated IR hardware (Tobii/Vision Pro) reaches ~1°. Plain webcams support coarse zone/window targeting, not caret-level precision — that is a property of the sensor, not an implementation gap. Progress in 2025–26 has been in CPU efficiency (MobileGaze ONNX/CPU), not in accuracy.
  • Calibration: explicit 9-point calibration drifts within a session. The field's answer is implicit calibration from mouse clicks — a click is a strong proxy for where the eyes were looking ~100 ms earlier — reaching 2.9° with zero explicit calibration steps (SalGaze/vGaze/EyeO extend the idea).
  • Interaction research: a CHI 2026 scoping review (103 studies) converges on a pattern: gaze grounds or disambiguates speech, and a second, cheap modality commits the action (Vision Pro's gaze+pinch, Talon's gaze+pop, MAGIC warping reporting +20.7% throughput). Google's Look to Speak demonstrates that coarse, three-way webcam gaze selection ships on-device today.
  • Commercial watch: Beam Eye Tracker (Eyeware) shipped a Linux+macOS public beta on 2026-07-23, $29.99 one-time, with a PyPI beam-eye-tracker package — the first turnkey commercial webcam-gaze SDK with a Linux Python path.

Feasibility: high — implicit click-based calibration and gaze-grounded command disambiguation (deixis) are CPU-trivial additions on top of an existing gaze pipeline.

2. Consumer/prosumer EMG

  • Meta Neural Band (Ctrl-Labs): shipped 2025-09-30 bundled with the $799 Ray-Ban Display, US-only, with expansion paused as of 2026-01. Its developer toolkit exposes six fixed gestures with no raw EMG access and no Linux/Python path, which rules it out as a general-purpose trigger. The underlying science is notable (Nature 2025's generic neuromotor interface work reports >90% cross-user accuracy and 20.9 WPM handwriting-style decoding); released datasets and checkpoints are CC-BY-NC and tied to Meta's specific 16-channel/2 kHz hardware geometry. The paper does not report latency or false-activation rate.
  • Buyable Linux/Python hardware, ranked by fit: (1) MindRove Armband — 8ch at 500 Hz, raw data, an official Linux+Python SDK, also supported by LibEMG; (2) Mudra Link ($199) — continuous pressure 0–100% with sub-10 ms latency claims, but its host application is macOS/Windows only; (3) OpenBCI Cyton/Ganglion with BrainFlow (MIT-licensed), fully open and offline; (4) second-hand Myo devices via pyomyo; (5) DIY MyoWare 2.0 (~$50) or BioAmp EXG Pill (CERN-OHL) — sensors that map naturally onto a serial-protocol activation backend.
  • What's decodable today: discrete gesture classification is a solved problem (92–96% accuracy; zero-shot cross-user generalization is the 2024–26 advance, e.g. ReactEMG at 92%). Continuous squeeze-force regression is comparatively underused despite strong results (grip-force regression r=0.97). Because EMG typing tops out around 20.9 WPM against ~150 WPM for speech, EMG's most defensible role in a dictation product is as a trigger, not as a text-entry channel. Squeeze-onset latency of 35–125 ms (threshold/TKEO/GMM methods) is well under the perceptible-delay threshold. The metric that is not published anywhere surveyed here, and is the one that would actually gate a design decision, is false-activation rate.
  • Software: LibEMG (pip install libemg; supports Myo/MindRove/SiFi/OYMotion) and BrainFlow (MIT) are the standard offline Linux stacks for this hardware class.

Feasibility: medium-high — device breadth and graded-pressure semantics are achievable in pure Python, fully offline.

3. Consumer BCI/EEG

  • The most credible offline option is Muse 2/S/Athena — raw EEG over plain BLE, no account required, via muselsl or BrainFlow ≥5.22.0 (2026-05). Neurosity Crown claims cloud dependence (a BrainFlow-based local workaround exists); Emotiv's raw EEG access is behind a paid license; Galea is a ~$25k enterprise headset; Neurable has no public SDK; PiEEG (~$350) is the open DIY path.
  • Switch-grade signals from consumer EEG are muscle/eye artifacts, not decoded intent: blink detection reaches ~99.5% accuracy at ~1.3 s latency with 0.10 false positives/minute (roughly 50 phantom activations across a workday in the worst case); jaw-clench detection reaches ~90% but is confounded by the act of talking — which rules it out for a dictation product specifically. SSVEP needs occipital electrode placement and a continuously flashing visual stimulus; motor-imagery approaches need 3–10 training sessions and fail outright for an estimated 15–30% of users.
  • A dedicated EMG electrode strictly dominates EEG-artifact detection on signal quality, latency, and false-positive rate for this use case. If EEG were ever added, the honest framing is a double-blink/jaw-clench accessibility switch via Muse+BrainFlow — not a headline capability.

Feasibility: low for anything beyond an accessibility-switch role.

4. Multimodal fusion and silent speech

  • Voice+gaze deixis — resolving "this"/"that" against where the user is looking — traces from Bolt's "Put-That-There" (1980) through GazePointAR (CHI 2024), G-VOILA, GazeGPT, and SemanticScanpath (2025). All use late fusion: the gaze target is serialized alongside the transcript rather than fused at the model level. Quantified payoff: +26.5% coreference accuracy on demonstratives (arXiv 2509.08689). EyeSayCorrect (IUI 2022) built desktop gaze+speech editing before the field's attention moved to AR glasses; as of this survey, no open-source desktop gaze-zone-plus-ASR deixis system was found.
  • Silent speech: AlterEgo was spun out in 2025 with no announced ship date; the open sEMG-to-speech research line (Gaddy & Klein) sits around 68% WER on open-vocabulary decoding — research-grade, not product-grade. Lip reading/audio-visual ASR (Auto-AVSR, Whisper-Flamingo) performs strongly but every surveyed pretrained checkpoint carries a non-commercial license and none run CPU-real-time.
  • Whispered speech is a notable sleeper result: off-the-shelf Whisper runs ~18.8% WER on whispered speech versus 2–6% for voiced speech, but a per-user fine-tune drops that to 0.38% — speaker-dependent whispered ASR is effectively a solved problem. DualVoice (UIST 2022, Rekimoto) demonstrated the resulting design pattern: whispered speech routes to commands, normal voice routes to literal text, on a plain microphone. Distinguishing whispered from voiced speech is CPU-trivial — whisper has no fundamental frequency, so an autocorrelation-based voicing check (80–450 Hz) combined with zero-crossing rate and energy is sufficient.

Feasibility: high for gaze-grounded deixis and whisper/voice detection (both zero new dependencies on an existing gaze/audio pipeline); low for lip reading; not viable at present for silent-speech sEMG.

5. Accessibility

  • Talon offers a free tier that runs on-device, but its eye-tracking mode needs Tobii hardware, and the project states plainly that Wayland support is not planned. Commercial AAC eye-gaze systems (TD Pilot ~$10k, TD I-Series ~$20k) are Windows/iPad-only, and insurance approval delays access further.
  • The Linux/Wayland accessibility-input landscape has several abandoned or broken tools: eViacam has been unmaintained since ~2019, Google's Project Gameface was archived in 2025-09, mousetweaks' dwell-click is broken on Wayland, and GNOME's Newton still cannot synthesize mouse events there. Dragon-dependent users migrating off Windows have essentially one free-software peer, Numen.
  • The resulting gap an offline, open-source Linux tool could address: Wayland-capable voice control, a maintained webcam head/face mouse (no actively maintained competitor found), dwell-click via libei, and a free alternative to the $10–20k Windows-locked AAC price point. Advocacy groups such as Team Gleason have publicly called for cheaper open communication technology.

Summary of buildable directions identified

  1. Deixis resolution — resolving "close this"/"put it there" against the gaze zone or window already tracked at hold-start. No new dependencies once a gaze pipeline exists; no open-source desktop equivalent was found.
  2. Implicit gaze calibration from mouse clicks — incremental refinement of an affine calibration map from ordinary clicks (2.9° reported in the literature).
  3. A whispered-speech command channel — a numpy-only fundamental-frequency gate; a whispered burst routes to commands, voiced speech dictates (the DualVoice pattern). No dictation product surveyed has this.
  4. EMG device breadth and squeeze-pressure semantics — BrainFlow/LibEMG adapters behind a pluggable backend interface; a light squeeze for talk, a hard squeeze for command; publishing a measured false-activation rate rather than assuming one.
  5. A hands-free accessibility profile — dwell-to-talk on gaze zones plus head-nod/jaw-open/brow-raise switches from an already-running face-landmark model, with uinput/libei injection for Wayland compatibility. No actively maintained competitor was found on Linux for this combination.

Explicitly not recommended, given the evidence above: consumer EEG as anything beyond an accessibility switch, lip-reading/AVSR, silent-speech sEMG, and the Meta Neural Band as an integration target. Worth continuing to watch: a Linux SDK from Beam Eye Tracker, and a Linux host application for Mudra Link.

What this study fed into

This study, together with the STT engine SoA, was direct research input to ADR-v2-129, which shipped gaze deixis, the sotto-voce whispered-command channel, and the pluggable activation-source seam that the EMG backend now uses. See the v2 cognitive layer for the design notes on what was built from this, and design/adr/ for the individual ADRs (search "gaze", "EMG", or "sotto-voce").