Research: streaming endpointing, thinking pauses and safe text delivery¶

Issue: #796, parent #787
Snapshot: YazSes main at fa8159d89aac73bc556bfb994a7defe0fb61df0c, inspected 2026-11-03
Status: Research finding and design recommendation; not a new implementation authorization
Scope: desktop dictation, LocalAgreement partials, release/endpoint semantics, staged mode, and external text-injection ownership

Executive decision¶

Adopt a pure, offline, synthetic replay and safety-gate specification now; defer any streaming rollout or default change; reject silence-triggered completion of a physical hold-to-talk burst. A proposed, unaccepted ADR is here; the testable contract is here. The synthetic fixture corpus is here.

The main problem is not choosing a faster endpointer. YazSes already uses the correct end-of-turn signal for a held key: actual key release. The safety risk is allowing provisional text to cross into an external app before the definitive transcript has survived classification, filtering, target checks, and (where configured) staged review. LocalAgreement confirmation is model stability, not permission to mutate a user's document.

Evidence labels¶

• [C] Code-inspected: current source/tests at the exact YazSes snapshot above. An inferred failure is not a reproduced desktop incident.
• [H] Historical recorded measurement: published YazSes measurements; not re-executed in this study.
• [P] Primary external source: linked upstream repository at a named commit, or original paper. An upstream number is never a YazSes benchmark.
• [U] Unknown/unverified: requires deterministic replay, hardware, or user evaluation.

1. Existing pipeline and what it actually guarantees¶

Source anchors: streaming engine, streaming injector, daemon, staged buffer, agent use case, hesitation ADR, and Ghost Ahead spec.

1.1 High-priority code-level failure: staged + live-streamed partials¶

At daemon.py lines 3955–3976, the partial injector is constructed when [streaming] enabled regardless of [staged] enabled. At 6503–6527, the poll loop forwards partial deltas directly to that injector. At 5199–5209, the normal _stage_or_commit result of None returns from _on_hold_end without first calling stream_injector.cancel(). The final replacement call at 5283–5285 is never reached in that branch.

Finding [C]: the control flow admits provisional typing in a staged burst and has no matching cleanup in its ordinary stage-only return path. Not yet established [U]: how often this occurs in a real editor and the exact content observed. The existing tests/test_staged_daemon.py tests staging in isolation, not this combination.

Important: blindly adding cancel() at the return is not a complete safety repair. If focus or caret moved while the speech was being decoded, the cancel's backspaces can erase other text. The safe design boundary is to prevent external partial injection up front when staging or another precommit safety policy applies. Any final delivery should still be explicitly reviewed and target-guarded.

1.2 Prefix revision and dropped-delta corner cases¶

The engine sets stable_prefix = common_prefix(previous, current) then computes delta = stable_prefix[len(last_emitted):]. But last_emitted never decreases on a later contradictory hypothesis. Example, entirely synthetic:

1. Two outputs agree on alpha, so alpha is emitted.
2. Subsequent outputs change the accepted prefix to alpba (same length). No retraction is emitted.
3. The next accepted prefix grows to alpbaX. The code emits only X, leaving alphaX in the live target until the final re-decode/replacement.

Likewise, _pending_partial is one slot, not a queue: two new deltas before the poll consumes the first overwrite rather than concatenate. These are reasoned possible traces from the current code [C], not measured regression frequencies. The replay contract includes each trace and requires a target-independent oracle.

1.3 Existing #153 fix narrows, but does not close, the problem¶

Closed #153 documents a long-X11-dictation incident: a fixed ten-second xdotool timeout produced partial side effects plus fallback duplicate typing; selection and an in-flight poll were other hazards. The current code adds length-aware timeout handling and a lock/seal. The synthetic screen regression explicitly covers duplicate text and unintended deletion of a known prefix. That is useful regression evidence [C], not evidence for every real application or all scripts. The reported literal repeated "D" remains unexplained in that historical report; this research does not assign it a cause.

A separate test weakness: test_streaming_engine_emits_partial ends with assert partial is not None or True, which is tautological and cannot fail when emission is absent [C].

2. Primary research and architectural tradeoffs¶

2.1 Latency and error evidence¶

docs/configuration.md records an older real-time-fed benchmark (n=15): speech-end-to-final on tiny.en was 0.92 s batch versus 1.22 s with streaming; on base.en, 1.42 s versus 2.21 s. The base model produced no visible confirmed prefix before release on 9/15 cases. [H] These measurements are as published, not reproduced or extrapolatable: the cited historical bench script is not present in the current public repository. No new WER/CER, latency, throughput, CPU or memory measurements were made in this research.

The current streaming loop re-decodes the growing accumulated buffer every interval [C]. It has no source-level bound equivalent to Whisper-Streaming's sentence-trimmed window [P]. Worst-case decode workload and stability over 30–120-second prompts remain unmeasured [U].

2.2 Dependency, platform, licensing and egress ledger¶

3. Options, recommendation and gates¶

Research acceptance gates (for any later implementation request):

1. Synthetic replay suite contains zero premature staged target writes, cross-target/cross-boundary deletions, duplicate final commits, post-seal partial writes, and release-before-release violations across deterministic negative tests.
2. Trace-based assertions prove that prefix regressions, pending-delta replacement, late poll completion, decode failure, empty final, no editable target, and subsequent user edits all fail without damaging pre-existing text.
3. Benchmarks publish model weights hash, faster-whisper/CTranslate2 versions, host CPU/core count, OS/backend, dataset manifest, normalization, full results, p50/p95 tail and CPU/RSS; compare streaming against batch on the same input.
4. Human-in-the-loop desktop validation is explicitly opt-in and uses safe test editors rather than users' active terminals. Linux GNOME X11 + GNOME/KDE Wayland, then macOS/Windows, require individually labelled verified reports; never promote platform or default claims on pure simulation alone.
5. Maintainer reviews and accepts the companion proposed ADR, and separately authorizes any delivery milestone/task. Until then the existing default stays unchanged.

4. Open questions for follow-up¶

1. Does the staged+streaming leak arise on a deterministic daemon-level fake injector and on any real editor? Code-path hazard [C]; real reproduction [U].
2. Should streaming + staged be mutually exclusive, or should streaming inference feed a private preview only? The policy draft favours preview-only long term and suppression of target partials immediately.
3. Is a single shared _pending_partial intentionally a latest-only display policy? If so, a source-state versioned snapshot (not suffix-only deltas) may be necessary.
4. Which target APIs, if any, can prove caret ownership/selection boundaries reliably on each platform? Until proven, external multi-keystroke rollback must not be called atomic.
5. What are p50/p95 onset→first-useful-preview and release→safe-final latencies after 10, 30, 60, 120 s prompts with long silent gaps? No current measurement.
6. How should physical held-key release and separately enabled hands-free auto-stop be represented in one replay event model without collapsing their different permission semantics?

No user study was performed, no human transcript/audio was collected, no model was downloaded, and no runtime was changed. This is a reproducible research/design deliverable for the evergreen issue, not evidence that the proposed fixes ship or that all current failures have been reproduced.