whisper_streaming — Online Whisper STT with LocalAgreement Policy
Summary¶
whisper_streaming (github.com/ufal/whisper_streaming, MIT) is an open-source library implementing online, real-time speech transcription on top of faster-whisper. [EVIDENCE src-002] Its core innovation is the LocalAgreement policy: audio is chunked, each chunk is decoded twice, and only the prefix that is stable across both decode passes is emitted to the consumer. [EVIDENCE src-002] This prevents premature or incorrect partial outputs from being injected into the target application. End-to-end CPU latency is reported at 380–520 ms, depending on hardware and audio complexity. [EVIDENCE src-002]
Key Findings¶
- LocalAgreement policy: A hypothesis prefix is committed for output only when it appears identically in two consecutive decode iterations, preventing unstable partials from reaching the application. [EVIDENCE src-002]
- Self-adaptive latency: The system adjusts chunk size dynamically based on measured source complexity — short, simple utterances are committed faster, while syntactically complex speech triggers longer observation windows. [EVIDENCE src-002]
- Backend: Uses faster-whisper as its transcription engine, inheriting int8 CPU quantisation and VAD filtering. [EVIDENCE src-002]
- End-to-end latency: Measured at 380–520 ms on CPU for English speech; this covers audio capture, chunking, two decode passes, and LocalAgreement comparison. [EVIDENCE src-002]
- Long-form support: Designed specifically for long-form online speech (live captions, real-time dictation), not just short utterance classification. [EVIDENCE src-002]
- Algorithm summary: chunk audio → run Whisper decode twice → emit stable prefix → slide audio window to remaining unconfirmed audio → repeat. [EVIDENCE src-002]
- Companion paper: Formally described in arXiv 2307.14743 (Macháček et al. 2023), which provides evaluation on SimulEval and discusses latency-accuracy trade-offs. [EVIDENCE src-002]
Relevance to YazSes Innovation Domains¶
| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | HIGH | Direct implementation path for partial-hypothesis display in YazSes overlay |
| SSH/remote | MEDIUM | Streaming text tokens can be forwarded to remote host over SSH pipe; audio stays local |
| Code awareness | LOW | No code-vocabulary specialisation; LocalAgreement stabilises tokens regardless of vocabulary |
| Accessibility | HIGH | Stable prefix emission reduces disorienting text flicker for users relying on screen feedback |
| AR/VR/XR | HIGH | 380–520 ms latency is borderline for XR; adaptive chunking may reduce it below 300 ms for short commands |
| Gaming | MEDIUM | Latency acceptable for command dispatch; partial display of in-progress dictation is a UX win |
Gaps Surfaced¶
- gap-stream-1: 380–520 ms CPU latency exceeds the <200 ms threshold required for seamless XR/real-time feel [TODO: find source for XR latency threshold]; adaptive chunk sizing alone may be insufficient.
- gap-stream-2: LocalAgreement requires two full decode passes per chunk, doubling CPU load relative to single-pass inference — profiling needed to confirm viability on low-power laptops.
- gap-stream-3: No built-in mechanism to display partial (uncommitted) hypotheses; consuming applications see only committed text, which creates a perceptible "silence" period before first output.
- gap-stream-4: Self-adaptive latency heuristics are not documented with reproducible benchmarks beyond the companion paper — behaviour on non-English or technical vocabulary is unknown. [TODO: find source]