ASR for Non-Native English: Accuracy and Disfluency Handling (arXiv 2503.06924)
Summary¶
This 2025 paper (arXiv 2503.06924) characterises Whisper's performance on non-native English speech and disfluency handling. [EVIDENCE src-016] Key finding: Whisper-medium achieves 8.3% WER on native English but 18.7% WER on non-native English — a relative degradation of approximately 125%. [EVIDENCE src-016] The paper also revisits disfluency behaviour: Whisper omits disfluencies in 11.86% of utterances and inserts phantom disfluencies in 3.48% of utterances. [EVIDENCE src-016] Post-processing correction reduces non-native English WER by 12 percentage points. [EVIDENCE src-016] The authors conclude that non-native speakers require targeted fine-tuning or a dedicated correction pipeline to achieve acceptable accuracy.
Key Findings¶
- Native vs. non-native WER: Whisper-medium achieves 8.3% WER on native English and 18.7% WER on non-native English. [EVIDENCE src-016] The gap is 10.4 percentage points (relative: ~125% worse on non-native). This is a severe accuracy degradation for a substantial portion of the global developer population.
- Disfluency omission rate (11.86%): In 11.86% of utterances, Whisper omits disfluent segments that were present in the input audio. [EVIDENCE src-016] This figure is consistent with findings in src-012, providing corroborating evidence.
- Disfluency insertion rate (3.48%): In 3.48% of utterances, Whisper adds disfluencies not present in the input. [EVIDENCE src-016] These phantom insertions can corrupt otherwise correct dictated text.
- Post-processing correction impact: A post-processing correction pass reduces non-native English WER by 12 percentage points (from 18.7% to approximately 6.7%). [EVIDENCE src-016] This is a very large gain for a post-processing step and validates investing in a correction layer.
- Fine-tuning recommendation: The authors recommend targeted fine-tuning on non-native English corpora or a dedicated correction pipeline as the two primary mitigation paths. [EVIDENCE src-016]
- Scope of non-native population: The paper notes that non-native English speakers represent a large fraction of the global developer population. [EVIDENCE src-016] For YazSes specifically, developers worldwide use English-language coding tools and APIs regardless of their native language — non-native English accuracy is not a niche concern.
- Correction reduces but does not eliminate gap: Even after post-processing, non-native WER remains higher than native WER. [EVIDENCE src-016] Fine-tuning or a multilingual model (Whisper large-v3) may be required for full parity.
Relevance to YazSes Innovation Domains¶
| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | HIGH | 18.7% WER on non-native English in streaming mode would produce frequent injection errors; correction must be integrated into the streaming pipeline |
| SSH/remote | MEDIUM | Non-native developers using remote systems over SSH are a core user segment; WER quality directly affects their productivity |
| Code awareness | HIGH | Non-native speakers dictating code identifiers and commands face compounded WER (non-native + technical vocabulary); correction layer is doubly important |
| Accessibility | HIGH | Non-native speech patterns overlap with atypical speech patterns in some dimensions; the correction pipeline serves both user groups |
| AR/VR/XR | MEDIUM | Non-native speaker WER degradation applies in XR environments; headset mic noise further increases WER (src-011) |
| Gaming | MEDIUM | Gaming voice commands by non-native speakers face same WER degradation; correction layer benefits all use cases |
Gaps Surfaced¶
- gap-nonnative-1: YazSes currently has no post-processing correction layer. [EVIDENCE src-016] The 12-point WER reduction from post-processing is the largest single improvement opportunity identified across all surveyed sources.
- gap-nonnative-2: The correction pass described in the paper is not specified as lightweight or streaming-compatible. Its latency impact on end-to-end injection time is not reported. [TODO: find source]
- gap-nonnative-3: Fine-tuning Whisper on non-native English corpora requires a labelled dataset. Open non-native English speech corpora (e.g. L2-ARCTIC, ISLE) exist but their coverage of technical/code vocabulary is limited. [TODO: find source]
- gap-nonnative-4: The 12-point WER reduction from post-processing is measured on general English; its magnitude on technical/code vocabulary by non-native speakers is not characterised. Compounded gains or diminishing returns are both plausible. [HYPOTHESIS]
- gap-nonnative-5: Whisper-medium is tested in this paper; YazSes defaults to base.en (lower accuracy but faster CPU inference). The native vs. non-native WER gap for base.en is likely larger than the 10.4-point gap observed for medium. [HYPOTHESIS]