Skip to content

ADR-v2-135 — Face-gesture switch access: a held facial movement as the hotkey

Status: Accepted (2026-09-02) · implemented in the same change Context links: [[adr-v2-129-killer-features-10x]] (the activation-source seam this is the third source through), [[adr-v04-003-emg-serial]] (the second source, and the pattern copied here), [[adr-v2-010-gaze-routed-dictation]] (the camera and the model, already running for a different purpose), [[adr-011]] (frames in RAM, nothing leaves the machine), issue #102 (epic: hands-free bundle)

Context

YazSes has two activation sources: a keyboard hotkey, and an EMG armband over serial or BLE. Both assume something the users this project was built for may not have — a key you can press, or hardware you can buy and wear. The gap is the classic switch access case: someone who can speak, and can move their face, and cannot reliably do either of the other two.

The free Linux stack for that is a graveyard. eViacam has been unmaintained since ~2019 and Google's Project Gameface was archived in 2025; commercial eye-gaze AAC devices are $10,000–20,000 and Windows/iPad-locked (evidence and citations: docs/research/muscle-brain-control.md). Issue #102 collects this as the "hands-free bundle" epic and says each sub-feature is independently shippable.

The parts were already here and already paid for. Glance-Type downloads MediaPipe's FaceLandmarker (~3.7 MB) and runs it on the webcam; that model reports 52 ARKit-style blendshape activations per frame — jawOpen, browInnerUp, mouthPucker and the rest — alongside the landmarks gaze uses. ADR-v2-129 built the seam (_build_activation_sources) precisely so a non-keyboard trigger could be added without touching the pipeline. Nothing read the blendshapes.

Decision

Add an opt-in face-gesture switch: [facegesture] enabled = false, a third source through the ADR-v2-129 seam. A held facial movement is the key — the mic opens while you hold it and closes when you relax — so the transcript, the guards, the injector and every downstream feature are reached unchanged.

Split the way the rest of the v2 layer is split: facegesture/detector.py is the entire activation policy and is pure (a score per frame in, "start" / "end" / nothing out), so it is tested frame by frame with no camera, no model and no MediaPipe; facegesture/backend.py is the capture loop and duck-types HotkeyBackend exactly as the two EMG transports do, taking the same two callbacks and the same mode decision.

Three choices inside the policy are load-bearing:

  1. Hysteresis, not a threshold. A blendshape score is continuous and noisy. A gesture held right at a single threshold crosses it several times a second, and each crossing would be a separate recording — a burst of one-word transcripts instead of one sentence. The mic opens at hold_threshold and closes only below release_threshold.
  2. A frame-count debounce (min_hold_frames). Talking, laughing and yawning all spike jawOpen transiently; a switch that fires on one frame fires on all of them. This is the Midas-touch defence, and its latency is affordable here specifically because the audio path already prepends [accessibility] pre_speech_padding_ms of ring-buffered audio — the words spoken during the debounce are still in the recording. That is a property of this pipeline, not of switch access in general.
  3. No reading counts toward release. A frame with no face — the user turned away, the camera dropped a frame — is not "still holding". The alternative is a microphone that stays open until they come back. It goes through the same min_release_frames debounce as a low score, so one blurred frame does not cut a sentence in half.

mode defaults to full_text here, where [emg] mode defaults to command: a squeeze is usually a second input beside a keyboard, and a face gesture is the replacement for one.

Consequences

  • The first maintained, free, offline face-switch activation on Linux, using only the webcam already in the lid. No new dependency: the same gaze extra (opencv + mediapipe) and the same model asset — enabling it after Glance-Type installs nothing, which system/depsize.py now prices correctly for both slugs.
  • Privacy is unchanged (ADR-011): frames are processed in-RAM, never stored, never sent.
  • Off by default and experimental — features enable facegesture requires --force. The thresholds shipped are reasoned, not measured on real faces; that measurement is the ask in #102, and the honest position is to say so rather than to imply a tuned default.
  • Cost, and it is real: a camera and a face-landmark inference run for as long as the daemon does, where the keyboard hook costs nothing. That is why it is opt-in rather than a fallback the daemon reaches for on its own.
  • Not addressed here, and still open in #102: dwell-to-talk (gaze, not gesture), and Wayland injection via libei — the desktop half remains X11-only, though this feature does not need it, because it drives the same injector the hotkey does.