Back to Basics: Revisiting ASR in the Age of Voice Agents (arXiv 2603.25727)
Summary¶
This 2026 paper (arXiv 2603.25727) reviews out-of-distribution (OOD) robustness of ASR systems for voice agent deployments, with particular focus on Whisper's failure modes in naturalistic speech. [EVIDENCE src-012] Key findings: Whisper omits disfluent segments in 11.86% of cases and adds them in 3.48% of cases; filler words ("um", "uh", "like") cause 8–15% WER increase; and code-switching (mixing languages mid-sentence) is a major failure mode. [EVIDENCE src-012] The paper recommends: (1) explicit disfluency detection before injection, (2) LLM-based post-correction for technical vocabulary, and (3) per-word confidence scoring to flag uncertain tokens. [EVIDENCE src-012]
Key Findings¶
- Whisper disfluency omission rate: Whisper omits disfluent speech segments (false starts, repairs, hesitations) in 11.86% of utterances. [EVIDENCE src-012] This means the injected text will silently lose portions of what the user said, which could corrupt dictated content in unexpected ways.
- Whisper disfluency insertion rate: In 3.48% of cases, Whisper adds disfluencies to the output that were not present in the input. [EVIDENCE src-012] This is a phantom word insertion problem.
- Filler word WER impact: Filler words such as "um", "uh", and "like" increase WER by 8–15%. [EVIDENCE src-012] Most users naturally produce filler words in continuous dictation. A pre-injection disfluency filter can mitigate this.
- Code-switching failure: Mixing languages mid-sentence (e.g. English sentence with a Spanish phrase) is a major failure mode for Whisper, causing significantly elevated WER. [EVIDENCE src-012] This is especially relevant for non-native English speakers who code-switch naturally.
- Recommendation 1 — disfluency detection: An explicit disfluency detection pass should be applied to transcripts before injection, removing filler words and false starts. [EVIDENCE src-012]
- Recommendation 2 — LLM post-correction: For technical vocabulary, LLM-based post-correction (e.g. "whisper said 'jason' but user likely meant 'JSON'") improves accuracy. [EVIDENCE src-012] This integrates naturally with YazSes's proposed LLM post-processing layer.
- Recommendation 3 — per-word confidence: Per-word confidence scores allow flagging uncertain tokens for review rather than silently injecting potentially wrong words. [EVIDENCE src-012]
Relevance to YazSes Innovation Domains¶
| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | HIGH | Disfluency handling must be integrated into the streaming pipeline, not just batch post-processing |
| SSH/remote | LOW | OOD robustness is a local ASR concern; remote transport does not affect this |
| Code awareness | HIGH | LLM post-correction for technical vocabulary directly addresses code identifier WER; confidence scoring flags misheard identifiers |
| Accessibility | HIGH | Disfluency omission is disproportionately harmful for users with atypical speech who rely on accurate transcription; confidence scoring enables user review |
| AR/VR/XR | MEDIUM | Disfluency filtering and confidence scoring improve command accuracy in XR where re-correction is disruptive |
| Gaming | MEDIUM | Filler word filtering prevents spurious game commands triggered by hesitation sounds |
Gaps Surfaced¶
- gap-disfluency-1: A lightweight, offline disfluency detection module that runs in the streaming pipeline between ASR output and text injection does not currently exist in YazSes. Building one requires a labelled disfluency corpus or a fine-tuned classifier.
- gap-confidence-1: faster-whisper exposes per-token log probabilities, but YazSes does not currently use them. Implementing per-word confidence thresholding requires surfacing these values to the injection layer.
- gap-codeswitching-1: Code-switching handling requires either a multilingual Whisper model or a language detection pass that adjusts the model mid-stream. The latency and accuracy implications of either approach are not yet characterised for YazSes.
- gap-llm-correction-1: LLM post-correction latency (~800 ms per src-010) may be unacceptable for interactive dictation. A lighter in-context correction approach (e.g. small LM with code vocabulary) may be needed as an intermediate step.