Killer Features 10x — State-of-the-Art Study & Program (2026-08-07)¶
Date: 2026-08-07 · Tier: design/ — public engineering research Companion: voice-dictation market landscape · local/offline STT engine SoA · gaze / EMG / BCI / multimodal input SoA · ADR-v2-129: Killer Features 10x
Synthesis of the three companion state-of-the-art studies above plus a full codebase reachability audit (2026-08-07, 132 subpackages). Written as the direct planning input to what became ADR-v2-129.
Goal at the time: make the project's differentiating input features an order of magnitude more real and more capable — gaze, voice, and eventually muscle-based control — with every heavy addition staying opt-in and lazy-installed, so the base dictation install stays small.
1. Where the project stood at the time¶
Already differentiated (nothing surveyed matched these): fully offline across Linux and 3 operating systems; offline diarized meeting minutes; a local, encrypted, self-improving learning loop; a reliability layer (microphone auto-heal, a no-text-target guard, an end-to-end pipeline verifier); gaze-routed dictation; an EMG hotkey (designed but not yet wired); SSH remote forwarding.
Behind the surveyed market on: voice-editing of already-typed text (a cloud-only feature elsewhere), per-app tone matching, custom AI prompt modes, and sub-second perceived latency as a default.
What the codebase audit found: 72 of 132 subpackages (~4.3k LOC) were transitively unreachable from the daemon and CLI at the time; 69 of 139 feature- registry entries toggled configuration that nothing read (features enable was technically lying about what it turned on); the EMG backend was complete but never constructed by any code path; a wake-word predicate existed with no spotter behind it; the STT engine was hard-wired to faster-whisper at three separate levels (no Protocol, no factory, and the streaming engine reached into a private attribute); and the gaze pipeline hard-coded confidence=1.0 for its default backend, silently making a documented confidence threshold a no-op.
What "10x" meant here: not new packages, but (a) making the advertised exotic input methods real and honestly reported rather than half-wired, (b) upgrading the perception layer with the zero-dependency moves the two companion SoA studies surfaced, and © breaking the single-engine STT assumption for a real step change in accuracy and latency.
2. The resulting program¶
Wave 1 — perceptual input (zero new required dependencies; all opt-in)¶
| # | Feature | SoA basis | Why it mattered |
|---|---|---|---|
| 1.1 | Real gaze confidence for the default backend (landmark-stability/presence-derived), replacing the hard-coded 1.0 | MediaPipe/MobileGaze practice | The confidence threshold starts actually gating routing instead of doing nothing |
| 1.2 | Gaze deixis — resolving "close this"/"focus that" against the gaze zone snapshotted at hold-start | GazePointAR/SemanticScanpath late fusion; +26.5% coreference payoff; an open gap in the surveyed field | Turns gaze routing into gaze reference |
| 1.3 | A whispered-speech command channel via a numpy fundamental-frequency gate | The DualVoice pattern (UIST 2022) | A socially quiet mode-switch channel not seen elsewhere in the survey |
| 1.4 | A pluggable activation-source seam, with the daemon constructing an EMG backend when configured | Nature 2025 sEMG validation; EMG's realistic role as a trigger, not a text channel | Makes the advertised muscle-control input real, and opens the same seam for a future wake word or other switch |
| 1.5 | Registry honesty — features enable refuses, with a clear message, any entry whose package nothing imports yet | Project value: never claim a capability is on when it is not | Restores trust in the feature registry |
Wave 2 — engine (opt-in, lazy-installed)¶
| # | Feature | SoA basis | Why it mattered |
|---|---|---|---|
| 2.1 | An SttEngine protocol and factory, removing the streaming engine's dependency on a private attribute | codebase audit | Unlocks a second engine everywhere in the pipeline |
| 2.2 | A Parakeet TDT 0.6B backend via onnx-asr int8, auto-installed on features enable stt-parakeet, model auto-downloaded on first use | 6.32% WER, beating large-v3's 7.44%, at ~30x realtime CPU; no silence hallucination; CC-BY-4.0; pure Python | Better-than-large-v3 accuracy at roughly the latency of a 5-second burst, fully offline |
Wave 3 — offline command mode¶
Voice-editing of selected text with a local LLM: select text anywhere, hold the command key, say "make this shorter" or "fix the grammar" or "turn into bullets." Reuses the existing command-key mode, clipboard machinery, output-side rewrite guards, and grammar constraints. Every product offering this in the market survey was cloud-only.
Deferred at the time (recorded, not built in this wave)¶
- Implicit gaze calibration from mouse clicks (needs a click listener; a later iteration).
- A hands-free accessibility bundle (dwell-to-talk, face-gesture switches, Wayland-compatible injection) — recorded as a full wave of its own.
- Moonshine v2 streaming preview; decode-time phrase boosting; small-model cleanup presets; per-app tone profiles; custom prompt modes.
- Explicitly set aside: consumer EEG triggers (artifact-grade, dominated by EMG on every axis measured), lip-reading/AVSR (non-commercial weights, not CPU-real-time), silent-speech sEMG (~68% WER), and the Meta Neural Band (no raw EMG access, no Linux path).
3. Design invariants this program stayed inside¶
Offline by construction and off by default (see the project's privacy/threat model); every heavy addition ships as a lazy-installed optional extra so the base install stays lean; each new input integrates at a single daemon seam behind a Protocol; and probe_backend-style honest error messages for anything not yet shipped, rather than a silent no-op.
What shipped¶
ADR-v2-129 records this program as accepted and implemented in the same change. See design/v2-cognitive-layer/ for the resulting design notes, and design/adr/ for the individual decisions (gaze deixis, sotto-voce, the activation-source seam, and the pluggable STT engine each has its own ADR).