## Summary

This paper by Macháček et al. (2023) is the formal academic description of the LocalAgreement streaming policy implemented in whisper_streaming. [EVIDENCE src-003] It provides rigorous evaluation methodology via the SimulEval protocol and demonstrates that streaming Whisper achieves sub-1-second end-to-end latency on CPU hardware. [EVIDENCE src-003] The paper introduces the `chunk_size` parameter as the primary knob controlling the latency-accuracy trade-off: smaller chunks lower latency at the cost of transcription accuracy, while larger chunks increase accuracy but introduce perceptible delay. [EVIDENCE src-003]

## Key Findings

- **LocalAgreement policy (formal):** A token prefix is emitted to the output stream only when it is identical across two consecutive decode passes on overlapping audio windows. [EVIDENCE src-003] This guarantees stability of emitted text without requiring a separate language model.
- **Sub-1-second CPU latency:** The paper reports end-to-end latency below 1 second on CPU-only hardware, covering audio buffering, dual decode passes, agreement comparison, and output delivery. [EVIDENCE src-003]
- **`chunk_size` trade-off:** Smaller `chunk_size` values (e.g. 0.5 s) minimise latency but increase word error rate on complex speech; larger values (e.g. 1.0–2.0 s) improve accuracy at the cost of delay. [EVIDENCE src-003] The optimal setting is content-dependent.
- **SimulEval evaluation protocol:** The paper uses SimulEval (a standardised simultaneous translation/transcription evaluation framework) to measure Average Lagging (AL) and translation quality simultaneously, enabling objective comparisons. [EVIDENCE src-003]
- **Acoustic boundary handling:** The streaming approach must handle mid-word boundaries in audio chunks gracefully; LocalAgreement implicitly handles this by discarding unstable tokens at chunk boundaries. [EVIDENCE src-003]
- **Long-form robustness:** Unlike short-utterance systems, the algorithm is validated on multi-minute audio streams, making it relevant to continuous dictation use cases. [EVIDENCE src-003]

## Relevance to YazSes Innovation Domains

| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | HIGH | Provides algorithmic foundation and evaluation methodology for YazSes streaming mode |
| SSH/remote | LOW | Paper does not address remote transport; latency figures assume local compute |
| Code awareness | LOW | Evaluation uses general English speech corpora; code/technical vocabulary not tested |
| Accessibility | MEDIUM | Sub-1s latency is meaningful for accessibility applications; SimulEval framework can benchmark AT use cases |
| AR/VR/XR | MEDIUM | Chunk-size tuning documented here is directly applicable to XR latency optimisation |
| Gaming | MEDIUM | Provides tunable latency knob (chunk_size) that game integration can exploit |

## Gaps Surfaced

- **gap-chunksize-1:** Optimal `chunk_size` is content-dependent and not automatically tuned; YazSes would benefit from a per-session adaptive tuner that monitors WER proxies to self-calibrate.
- **gap-simuleval-1:** SimulEval is designed for translation tasks; a YazSes-specific evaluation harness measuring injection latency and correction accuracy does not yet exist.
- **gap-technical-vocab-1:** Paper evaluates on general English; WER behaviour on programming vocabulary, CLI commands, and identifier names is not characterised. [TODO: find source]
- **gap-latency-floor-1:** The paper does not identify the theoretical latency floor for LocalAgreement on CPU — it is unclear how much further optimisation is possible within this algorithm.