Turning Whisper into Real-Time Transcription System (arXiv 2307.14743)
Summary¶
This paper by Macháček et al. (2023) is the formal academic description of the LocalAgreement streaming policy implemented in whisper_streaming. [EVIDENCE src-003] It provides rigorous evaluation methodology via the SimulEval protocol and demonstrates that streaming Whisper achieves sub-1-second end-to-end latency on CPU hardware. [EVIDENCE src-003] The paper introduces the chunk_size parameter as the primary knob controlling the latency-accuracy trade-off: smaller chunks lower latency at the cost of transcription accuracy, while larger chunks increase accuracy but introduce perceptible delay. [EVIDENCE src-003]
Key Findings¶
- LocalAgreement policy (formal): A token prefix is emitted to the output stream only when it is identical across two consecutive decode passes on overlapping audio windows. [EVIDENCE src-003] This guarantees stability of emitted text without requiring a separate language model.
- Sub-1-second CPU latency: The paper reports end-to-end latency below 1 second on CPU-only hardware, covering audio buffering, dual decode passes, agreement comparison, and output delivery. [EVIDENCE src-003]
chunk_sizetrade-off: Smallerchunk_sizevalues (e.g. 0.5 s) minimise latency but increase word error rate on complex speech; larger values (e.g. 1.0–2.0 s) improve accuracy at the cost of delay. [EVIDENCE src-003] The optimal setting is content-dependent.- SimulEval evaluation protocol: The paper uses SimulEval (a standardised simultaneous translation/transcription evaluation framework) to measure Average Lagging (AL) and translation quality simultaneously, enabling objective comparisons. [EVIDENCE src-003]
- Acoustic boundary handling: The streaming approach must handle mid-word boundaries in audio chunks gracefully; LocalAgreement implicitly handles this by discarding unstable tokens at chunk boundaries. [EVIDENCE src-003]
- Long-form robustness: Unlike short-utterance systems, the algorithm is validated on multi-minute audio streams, making it relevant to continuous dictation use cases. [EVIDENCE src-003]
Relevance to YazSes Innovation Domains¶
| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | HIGH | Provides algorithmic foundation and evaluation methodology for YazSes streaming mode |
| SSH/remote | LOW | Paper does not address remote transport; latency figures assume local compute |
| Code awareness | LOW | Evaluation uses general English speech corpora; code/technical vocabulary not tested |
| Accessibility | MEDIUM | Sub-1s latency is meaningful for accessibility applications; SimulEval framework can benchmark AT use cases |
| AR/VR/XR | MEDIUM | Chunk-size tuning documented here is directly applicable to XR latency optimisation |
| Gaming | MEDIUM | Provides tunable latency knob (chunk_size) that game integration can exploit |
Gaps Surfaced¶
- gap-chunksize-1: Optimal
chunk_sizeis content-dependent and not automatically tuned; YazSes would benefit from a per-session adaptive tuner that monitors WER proxies to self-calibrate. - gap-simuleval-1: SimulEval is designed for translation tasks; a YazSes-specific evaluation harness measuring injection latency and correction accuracy does not yet exist.
- gap-technical-vocab-1: Paper evaluates on general English; WER behaviour on programming vocabulary, CLI commands, and identifier names is not characterised. [TODO: find source]
- gap-latency-floor-1: The paper does not identify the theoretical latency floor for LocalAgreement on CPU — it is unclear how much further optimisation is possible within this algorithm.