WhisperX — Forced Alignment and Speaker Diarization for Whisper
Summary¶
WhisperX (github.com/m-bain/whisperX, BSD-2-Clause) extends faster-whisper with post-hoc forced alignment using wav2vec2, producing precise word-level and character-level timestamps significantly more accurate than Whisper's native timestamp estimates. [EVIDENCE src-005] It also integrates pyannote-based speaker diarization for multi-speaker scenarios. [EVIDENCE src-005] While WhisperX is batch-mode focused rather than real-time streaming, its architectural insight about forced alignment enabling cursor-based text editing is directly applicable to YazSes's correction layer design.
Key Findings¶
- Forced alignment via wav2vec2: After Whisper transcription, WhisperX runs wav2vec2 forced alignment to produce character-level timestamps, substantially more precise than Whisper's native token-level estimates. [EVIDENCE src-005] This enables knowing exactly where in the injected text each word and character sits.
- Character-level cursor positioning: The precise timestamps allow a consuming application to calculate the cursor offset for any word in the injected text. [EVIDENCE src-005] This enables "delete last word", "correct previous sentence", and similar editing commands without relying on UI-level cursor queries.
- Speaker diarization: pyannote integration segments audio by speaker. [EVIDENCE src-005] In a multi-speaker environment (e.g. pair programming session, conference call), this identifies which speaker is dictating.
- Batch-mode focus: WhisperX processes complete recordings rather than streaming audio windows. [EVIDENCE src-005] Integrating its alignment stage into a streaming pipeline requires buffering and retroactive timestamp correction.
- BSD-2-Clause licence: Permissive — compatible with YazSes's distribution model. [EVIDENCE src-005]
- Text correction enablement: Forced alignment demonstrates that word timestamps can support cursor-based editing of injected text, bridging ASR output and editor state. [EVIDENCE src-005] This is a key architectural insight for YazSes's "voice editing" feature domain.
Relevance to YazSes Innovation Domains¶
| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | LOW | Batch-only; alignment stage cannot run before full utterance is available |
| SSH/remote | LOW | No remote-specific features; timestamp data could be forwarded but adds payload size |
| Code awareness | MEDIUM | Character-level timestamps enable precise cursor placement in code editors for correction commands |
| Accessibility | HIGH | Cursor-positioning unlocks "delete that", "correct last phrase" commands critical for motor-impaired users |
| AR/VR/XR | LOW | Batch mode incompatible with real-time XR interaction; diarization could identify active speaker in social VR |
| Gaming | LOW | Batch processing not suitable for in-game dictation; diarization potentially useful in team comms |
Gaps Surfaced¶
- gap-align-1: wav2vec2 forced alignment adds significant compute overhead (model load + inference pass); impact on total latency in a streaming context is not characterised. [TODO: find source]
- gap-align-2: Forced alignment requires a reference transcript first, creating a two-stage pipeline (Whisper → wav2vec2). Retro-active timestamp correction in streaming mode requires buffering injected text for potential post-correction — a stateful design challenge.
- gap-align-3: Character-level timestamps assume the injected text is not subsequently modified by the application (e.g. autocorrect, code formatter). A synchronisation mechanism between ASR timestamps and actual editor buffer state is needed.
- gap-diarize-1: Speaker diarization via pyannote requires significant additional memory and CPU; suitability for low-resource laptops is unclear. [TODO: find source]