WhisperPipe — Low-Latency Streaming ASR with Consensus Engine (arXiv 2604.25611)
Summary¶
WhisperPipe (arXiv 2604.25611, preprint 2026) is a novel streaming ASR architecture designed for voice agents and live captioning. [EVIDENCE src-004] It combines Silero VAD with energy-based acoustic filtering to suppress false activations before audio reaches the Whisper model, and introduces a consensus engine for stable prefix commitment analogous to LocalAgreement but with candidate hypotheses generated at fixed intervals with word-level timestamps. [EVIDENCE src-004] The system achieves a median end-to-end latency of 89 ms, 48% lower peak GPU memory, and 80.9% lower average GPU utilisation compared to a baseline streaming Whisper implementation. [EVIDENCE src-004]
Key Findings¶
- Dual-stage filtering: Silero VAD combined with energy-based filtering reduces false activations by 34% relative to VAD-only filtering. [EVIDENCE src-004] This means fewer spurious decode invocations and lower overall compute load.
- Consensus engine: Instead of the dual-pass LocalAgreement approach, WhisperPipe generates candidate hypotheses at fixed intervals and commits the stable prefix across candidates, with word-level timestamps attached to each committed token. [EVIDENCE src-004]
- Median E2E latency 89 ms: Substantially lower than the 380–520 ms reported for whisper_streaming. [EVIDENCE src-004] However, this figure is from a GPU-accelerated deployment — CPU behaviour is not directly reported. [HYPOTHESIS]
- GPU efficiency: 48% reduction in peak GPU memory and 80.9% reduction in average GPU utilisation vs baseline. [EVIDENCE src-004] This makes it viable on consumer GPUs that may be present in gaming PCs and high-end developer machines.
- Word-level timestamps: Every committed token carries start/end timestamps. [EVIDENCE src-004] This enables cursor-positioning for post-injection correction (e.g. "delete last word" knowing where it sits in the text stream).
- Design targets: Explicitly designed for voice agents and live captioning. [EVIDENCE src-004] These overlap directly with YazSes's dictation and command dispatch use cases.
- Fixed-interval hypothesis generation: Unlike whisper_streaming's adaptive chunking, WhisperPipe generates candidates on a regular clock, simplifying integration with downstream consumers that poll on a timer. [EVIDENCE src-004]
Relevance to YazSes Innovation Domains¶
| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | HIGH | 89 ms median latency and word timestamps are directly applicable to partial-display and correction features |
| SSH/remote | MEDIUM | Fixed-interval token emission simplifies remote forwarding protocol design |
| Code awareness | LOW | No code-vocabulary layer; word timestamps could support identifier-level correction |
| Accessibility | HIGH | Low latency and word timestamps reduce cognitive load for users relying on voice as primary input |
| AR/VR/XR | HIGH | 89 ms on GPU is well within the <200 ms XR threshold; energy-based filtering handles headset mic noise |
| Gaming | HIGH | GPU efficiency means running WhisperPipe alongside a GPU-intensive game is feasible on a single consumer card |
Gaps Surfaced¶
- gap-whisper-pipe-1: The 89 ms latency figure is GPU-based; YazSes's CPU-only constraint means this metric may not transfer directly. A CPU benchmark of WhisperPipe is needed. [TODO: find source]
- gap-whisper-pipe-2: Energy-based filtering parameters (threshold, window size) are not publicly documented in the preprint — tuning for different microphone types (headset vs. condenser vs. laptop) requires further investigation. [TODO: find source]
- gap-whisper-pipe-3: Fixed-interval hypothesis generation may introduce periodic latency spikes if the interval aligns poorly with speaking pace — adaptive interval selection is not addressed.
- gap-whisper-pipe-4: Word-level timestamp accuracy under CPU int8 quantisation (YazSes's current mode) is not characterised in this paper.