Voice control: the year local speech passed the cloud¶

Updated 2026-08-07. Part of the research series.

For a decade the deal was: accurate speech recognition lives in a datacenter. That deal quietly expired. The measurements below are why a fully-offline dictation daemon is no longer a compromise — and where the remaining hard problems actually are.

Accuracy: the 2026 leaderboard, CPU edition¶

Word-error rates (WER) on multi-domain evaluations, with the constraint that matters here — runs on a laptop CPU:

---
config:
  themeVariables:
    xyChart:
      backgroundColor: "transparent"
      titleColor: "var(--md-default-fg-color)"
      xAxisLabelColor: "var(--md-default-fg-color)"
      yAxisLabelColor: "var(--md-default-fg-color)"
      xAxisTitleColor: "var(--md-default-fg-color)"
      yAxisTitleColor: "var(--md-default-fg-color)"
      xAxisTickColor: "var(--md-default-fg-color--lighter)"
      yAxisTickColor: "var(--md-default-fg-color--lighter)"
      xAxisLineColor: "var(--md-default-fg-color--lighter)"
      yAxisLineColor: "var(--md-default-fg-color--lighter)"
      plotColorPalette: "#8f6fd6"
---
xychart-beta
    title "Word error rate, multi-domain (percent; lower is better)"
    x-axis ["whisper small.en", "whisper-large-v3", "Moonshine v2", "Parakeet TDT 0.6B", "Canary-Qwen 2.5B"]
    y-axis "WER (%)" 0 --> 10
    bar [8.5, 7.4, 6.7, 6.3, 5.6]

Bars use the midpoint where a source reports a range; the table below gives the range, the CPU speed, and the licence — which is usually the number that decides whether you can actually ship it.

The headline: a 0.6B model now beats the 1.5B flagship at a quarter of the compute — the transducer (TDT) architecture, not scale, did it. A pleasant side effect: transducers emit nothing on silence, so the whole class of [BLANK_AUDIO] / "thanks for watching" hallucinations Whisper produces on quiet audio (survey) disappears at the source. This is why yazses features enable stt-parakeet exists — and why it lazy-installs its runtime so pure-Whisper users carry zero extra weight.

Latency: the physics of "feels instant"¶

The commercial bar is explicit: Aqua Voice starts capturing in <50 ms and lands text in 450 ms–1 s; cloud tools like Wispr Flow take 1–3 s for the round-trip (comparison). Users call the first "instant" and the second "fine". Offline tools that wait 2–5 s after speech get abandoned — it is a top complaint in reviews of the most popular open-source tool (review).

For hold-to-talk, the whole latency story is one number — decode time after key release:

sequenceDiagram
    participant U as You
    participant D as Daemon
    participant E as STT engine
    U->>D: hold key, speak 5 s
    Note over D: audio buffered live<br/>(zero decode cost at ~30x RT)
    U->>D: release key
    D->>E: decode 5 s burst
    Note over E: whisper small: ~600 ms<br/>Parakeet TDT: ~170 ms
    E->>D: text
    D->>U: typed into the focused window

At ~30× realtime, batch decode of a 5-second burst is ~170 ms — inside the "instant" budget without any streaming machinery. Streaming still matters for long-form dictation preview; the state of the art there is Moonshine v2's cached streaming encoder (148–258 ms measured, with its streaming mode reported as more accurate than its own batch mode — Kudlur et al.).

The whisper channel: one microphone, two modes¶

A dictation tool has a mode problem: the same audio channel must carry text ("write this down") and commands ("delete that"). Keyboards solve it with a second key. But there is a purely acoustic solution hiding in phonetics:

Whispered speech has no fundamental frequency. Voiced speech is driven by vocal-fold vibration (an F0 around 80–300 Hz plus harmonics); whispering replaces that periodic source with turbulent airflow — aperiodic, flatter spectrum. Detecting the difference needs no ML at all: an autocorrelation voicing check plus spectral tilt, in numpy, per frame.

flowchart LR
    A[Held burst] --> B{Median voicing\nacross frames}
    B -- "periodic (F0 present)" --> C[Dictation:\ntype the words]
    B -- "aperiodic + flat tilt" --> D[Command:\nparse, never type]

The interaction design is DualVoice (Rekimoto, UIST 2022): whisper = command channel, normal voice = literal text, on a plain microphone (Rekimoto). It was a research prototype that never shipped in a product; YazSes now ships it as the sotto-voce channel ([whispermode] command_channel). Two honest caveats from the literature: STT accuracy on whispered speech is markedly worse than on voiced speech (~18.8% WER off-the-shelf vs 2–6%; per-user fine-tuning closes it to <1% — Farhadipour et al.), which is tolerable because command phrases are short and grammar-matched; and a median vote across the burst is needed so one breathy word can't flip a sentence.

Personalization: what actually moves WER¶

Ranked by measured return-on-effort for a personal dictation tool:

1. Decode-time phrase boosting — rescoring the decoder toward your vocabulary, up to ~20k phrases with no retraining and no speed penalty on transducer models (Andrusenko et al., TurboBias). Strictly stronger than Whisper's initial_prompt, which is capped at 224 tokens, only reliably affects the first 30-second window, and raises hallucination risk with dense term lists (OpenAI's own guide, Jogi et al.).
2. Corpus-mined vocabulary — what yazses tune already proposes from the encrypted local learning corpus, with no data leaving the machine.
3. LoRA fine-tuning — real gains (−2.2 WER multilingual; dramatic for atypical speech with ~1.4 h of data — Diabolocom) but needs a consumer GPU; CPU-only training remains impractical in 2026.

The ordering matters for accessibility: option 3 is the one that helps dysfluent and atypical speakers most, and it is the one gated behind hardware most of them don't have. Making option 1 work on a CPU transducer is therefore an accessibility problem disguised as an engineering problem.

Open questions¶

Discuss → · Benchmark harness issue →

1. Whisper-detection thresholds across voices. Our voicing/tilt gate ships with literature-derived defaults. How do they hold up across quiet talkers, breathy voices, tonal languages, and cheap microphones? A false "whispered" verdict silently eats a sentence — what's the measured false positive rate in the wild?
2. Phrase boosting on ONNX transducers. TurboBias lives in NeMo; the lightweight onnx-asr path has no boosting hook yet. What is the cheapest faithful reimplementation — and does edit-distance post-correction get 80% of the win for 5% of the work (Lall & Tan)?
3. Latency benchmarking as a feature. No offline tool publishes measured release-to-text times per model and CPU. What would a fair, reproducible yazses bench protocol look like?
4. Code-switching. Stock Whisper cannot mix two languages in one utterance ("one language per 30 s window"); adapter-based approaches reach ~14% mixed error rate but need per-pair training. Which language pairs matter most to real dictation users?

References¶

Evidence grade: measured (peer-reviewed measurement), vendor (claimed by the maker), secondary (review or benchmark write-up).

1. Ruan, S., Wobbrock, J. O., Liou, K., Ng, A., Landay, J. "Comparing speech and keyboard text entry for short messages in two languages on touchscreen phones." arXiv:1608.07323, 2016. arXiv — measured (153 vs 52 WPM, English)
2. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I. "Robust speech recognition via large-scale weak supervision." arXiv:2212.04356, 2022. arXiv — measured
3. NVIDIA. "Parakeet TDT 0.6B" model card. Hugging Face — vendor
4. Independent CPU benchmark: Whisper vs Parakeet TDT. snailtext.app — secondary
5. Kudlur, M. et al. "Moonshine v2: ergodic streaming encoder ASR for latency-critical speech applications." arXiv:2602.12241, 2026. arXiv — measured
6. "Best open-source speech-to-text models in 2026: benchmarks." Northflank. northflank.com — secondary
7. "Aqua Voice vs Wispr Flow" latency comparison. getvoibe.com — secondary
8. Handy (open-source dictation) user review. getvoibe.com — secondary
9. Rekimoto, J. "DualVoice: speech interaction that discriminates between normal and whispered voice input." UIST '22, 2022. doi:10.1145/3526113.3545685 — measured
10. Farhadipour, A. et al. "Leveraging self-supervised models for automatic whispered speech recognition." arXiv:2407.21211, 2024. arXiv — measured
11. Andrusenko, A. et al. "TurboBias: universal ASR context-biasing powered by GPU-accelerated phrase-boosting tree." arXiv:2508.07014, 2025. arXiv — measured
12. OpenAI. "Whisper prompting guide." cookbook.openai.com — vendor
13. Jogi, Y. et al. "Improving rare-word recognition of Whisper in zero-shot settings." arXiv:2502.11572, 2025. arXiv — measured
14. Lall, V., Liu, Y. "Contextual biasing to improve domain-specific custom vocabulary audio transcription without explicit fine-tuning." arXiv:2410.18363, 2024. arXiv — measured
15. Diabolocom Research. "Fine-tuning ASR: focus on Whisper." diabolocom.com — secondary

See also: voice commands in practice, eye control for how gaze resolves "this" in a spoken command, and the research index for how speech compares to every other input channel.