Streaming Transcription with Real-Time Display and Rollback Correction

Description¶

Streaming transcription displays partial hypotheses in the active window while the user speaks, giving immediate visual feedback that recording is active and progressing. Every 300ms, the daemon emits a partial hypothesis to the injector, which inserts it at the cursor. When the final transcript is committed, the daemon uses cursor-relative selection (tracking the injection start position via word-level timestamps) to select and overwrite all partial text with the corrected final result. The user perceives real-time typing with invisible in-place correction at sentence end. Partial emission uses the LocalAgreement stability policy to avoid flickering on unstable prefixes. The disfluency filter (cap-004) runs on the final transcript before the overwrite step, producing clean committed text.

User Pain¶

Users who dictate long sentences into code editors, terminals, or document editors today see nothing until transcription completes — a silent 0.8–3 second pause after releasing the hold key. This silence is cognitively disruptive: users cannot tell if the system heard them, if it is still processing, or if it crashed. Users frequently re-speak or cancel, leading to double-injection or missed text. Compared to cloud dictation tools (Google Docs voice, Whisper API streaming), YazSes's batch-only mode feels unresponsive. Developers specifically report frustration when dictating multi-line docstrings or comments where early words are confident but the cursor shows no activity. [EVIDENCE src-002] [EVIDENCE src-003]

Proposed Solution¶

Core streaming pipeline:

Chunked audio feed: The recorder (audio/recorder.py) is modified to yield fixed-size chunks (300ms at 16kHz = 4800 samples) into an asyncio.Queue rather than accumulating a full buffer.
faster-whisper streaming mode: stt/faster_whisper.py gains a transcribe_streaming() async generator that calls model.transcribe() on a rolling window (last 2s of audio) every 300ms. Returns (partial_text, is_stable, word_timestamps).
LocalAgreement stability filter (stt/local_agreement.py): Implements the LocalAgreement algorithm [EVIDENCE src-002] [EVIDENCE src-003]. Maintains a committed prefix — the longest prefix that has remained identical across the last N=3 consecutive partial hypotheses. Only the committed prefix is injected; unstable suffixes are held back. This prevents flickering on uncertain word boundaries.
Streaming injector (inject/streaming.py):
On first partial: records injection_start (absolute cursor position, obtained via xdotool getactivewindow getwindowfocus + AT-SPI on Linux, AXValue on macOS).
On each subsequent partial: issues Backspace * len(prev_injected) followed by new partial text injection. For long deletions (>20 chars), uses clipboard replace to avoid visible flicker.
On final commit: issues a range-select from injection_start to current cursor using platform selection keys (Shift+Home / Shift+End / Ctrl+Shift+Home depending on context), then injects final text.
Word-level timestamp use [EVIDENCE src-001] [EVIDENCE src-005]: Word timestamps from faster-whisper are used to compute the exact character offset of injection_start, enabling precise range selection even if the cursor has drifted slightly due to partial injections.
Rollback on disfluency: The disfluency handler (cap-004) intercepts "delete that" / "scratch that" commands detected in the partial stream and triggers a full rollback to injection_start before clearing the buffer.

Config:

[streaming]
enabled = true
partial_interval_ms = 300
stability_window = 3          # LocalAgreement N
max_backspace_chars = 20      # above this, use clipboard replace

Graceful degradation: If the injector cannot determine cursor position (e.g., non-AT-SPI app), streaming falls back to batch mode silently with a [streaming_unavailable] debug log.

Feasibility Notes¶

Complexity: High. The streaming pipeline requires changes across four modules (recorder, STT, injector, daemon state machine). Backspace-based correction is fragile in some terminal emulators and rich-text editors where backspace semantics differ. The clipboard replace path for long deletions adds ~50ms latency.

Dependencies: No new packages beyond existing stack. AT-SPI on Linux requires python-atspi (already optional for accessibility work); graceful degradation ensures it is not a hard requirement. macOS AXValue access requires accessibility permissions already granted for keyboard injection.

Risks: - Cursor drift: if the user types while the daemon is injecting partials, injection_start offset becomes invalid. Mitigate with a "cursor dirty" flag: any keyboard event during partial injection aborts streaming and falls back to batch for that utterance. - Terminal emulators (kitty, alacritty) handle rapid backspace sequences differently; test matrix required. - faster-whisper rolling-window transcription at 300ms intervals is CPU-intensive; measure CPU% on a low-end machine (Raspberry Pi 4 as proxy); may need to increase interval to 500ms.

Success Metrics¶

First partial hypothesis appears within 400ms of speech onset (measured from hold key press).
Final committed text matches ground-truth transcript with ≤1 word error compared to batch mode baseline.
Zero cases of double-injection or missing text in 1000-utterance regression test.
CPU overhead of streaming mode vs. batch mode: <25% increase on a 4-core x86 machine.
User subjective rating of "responsiveness" improves by ≥1 point (5-point Likert) in A/B study vs. batch mode.

Evidence¶

[EVIDENCE src-001] Word-level timestamps from faster-whisper enable precise injection start tracking and range selection.
[EVIDENCE src-002] LocalAgreement algorithm for stable prefix selection in streaming ASR — prevents flicker on unstable hypotheses.
[EVIDENCE src-003] Streaming ASR partial hypothesis stability analysis — validates 300ms chunk interval and N=3 stability window.
[EVIDENCE src-005] faster-whisper CPU int8 benchmarks confirm rolling-window re-transcription is feasible within the latency budget.