Skip to content

Paper v2 status — results, gaps, and execution map

Updated: 2026-09-25
Programme issue: #510
Authorship/publication operations: #484

This page is the short operational view of the second YazSes paper. It does not replace the detailed result delta, claim ledger, Azure campaign record, or manuscript plan.

Current paper direction

The next paper should be an expanded empirical systems paper, not a feature changelog.

Working thesis:

A privacy-first offline voice-input system cannot be evaluated by one WER on clean speech. Harder audio, repeated decoding, decoder settings, hardware/ISA variation, and task-specific failure modes materially change the engineering conclusions.

Current evidence is already strong enough for a materially expanded arXiv revision after a final evidence freeze. A stronger conference/journal submission should add human-facing and external- validity studies.

Strongest new results since paper v1

Finding Current evidence
Shared local-ASR comparison Eight engine/checkpoint configurations measured through the shipping product seam. On the Azure clean-speech matrix, Parakeet TDT 0.6B v2 has the lowest point-estimate WER at 2.06%; Moonshine/tiny has the lowest measured RTF at 0.016. Intervals and host-specific timing caveats still apply.
Harder speech changes the picture Parakeet moves 2.06% → 2.88% from test-clean to test-other; base.en moves 4.01% → 9.46%; Moonshine/base 3.17% → 8.04%. Clean speech alone does not characterise robustness.
large-v3 repeated-decode failure Across the four controlled hard-split repeats, substitutions stay at 87, deletions at 15, hits at 3619, while insertions move 101 → 144; the archived matrix run reaches 184 and is reported beside that range, not folded into it. The measured WER movement is a continuation/insertion tail failure, not changing recognition substitutions.
Previous-text conditioning is checkpoint-dependent On the measured ladder it helps base.en, helps small.en slightly, is byte-identical at medium.en, and disabling it removes the observed large-v3 runaway mode. This is not evidence that disabling context universally improves WER.
Greedy decoding has a measurable cost on the default model base.en: beam 1 vs beam 5 is 4.39% vs 4.01% on clean speech and 10.56% vs 9.46% on hard speech; the hard-split paired comparison is significant in the archived analysis.
Cross-platform decode variation is measured On the 60-utterance common subset, tiny.en spans 3.39–3.88%, base.en 3.25–3.39%, while small.en is 2.05% on all four measured runners. Decode portability is still different from full end-to-end OS validation.
Synthetic onset silence does not recover missed speech No tested lead-in value establishes a corrected paired benefit over no lead-in. Audio not captured before activation cannot be reconstructed by prepending silence afterward.
Streaming is conditional, not automatically faster Manuscript-grade paper/results/streaming.json: tiny.en median final latency 1.377 → 2.371 s with streaming and 59.33% median text visible at release; base.en 4.024 → 7.612 s, 0% median visible, and 12/15 with no confirmed partial before release. A later docs rerun must be archived before its numbers enter the paper.
Real-meeting diarization changed the shipped default Full AMI test split: old threshold 0.5 gives 75.21% mean DER and extreme over-splitting; current meeting threshold 1.2 gives 26.71% mean DER, 27.37% time-weighted DER.
A speaker-count cap is not the same as better DER With max_speakers=4 at the current threshold, 7 recordings improve, 7 worsen and 2 are unchanged; sign test p=1.0. Count correctness improves from 2/16 to 15/16, but a DER benefit is not established.
One diarization threshold does not transfer across domains Approximate preferred regions differ: synthetic 0.8–0.9, VoxConverse ~0.9, AMI ~1.2.
Simple centroid repair failed No useful cosine threshold repairs enough split-speaker clusters without unacceptable wrong-person merges. This negative result should remain visible.
Provenance changed the science An apparent benchmark disagreement became the large-v3 reproducibility investigation because the contradicting run was still readable. Say it precisely: run 1 of the test-other matrix (4.86%) was overwritten and survives only as probes/logs/x86b-other_wer.log; the history/ retention mechanism was written because of that near-loss, not in time to prevent it.

September follow-up that narrows an August interpretation

A later tiny.en follow-up decoded the same 60 test-clean utterances five times and produced byte-identical hypotheses at 3.67% WER in all five runs.

Therefore:

  • large-v3 is the strong corpus-level repeated-decode instability result;
  • tiny.en should not be described as generally corpus-level unstable;
  • rare clip-level fallback instability can still exist, but the paper must keep that claim narrow.

The detailed wording is already corrected in RESULTS_DELTA.md.

What is ready now

  • August Azure campaign reconstructed with provenance and corpus roles.
  • v1 → v2 result delta documented.
  • Claim ledger separates supported, qualified, open and withdrawn claims.
  • Full manuscript architecture and threats-to-validity plan exists.
  • Authorship/consent/final-approval protocol exists.
  • Scientific follow-up work is split into issue-sized work packages.
  • September tiny.en follow-up incorporated into the interpretation.

Manuscript drafting

  • #532 — continue the public LaTeX paper-v2 manuscript against the claim ledger and frozen evidence.

The working draft now lives at manuscript/. Generated PDFs remain build artifacts and are not committed.

Required work for the arXiv-v2 evidence freeze

  • #495 — freeze and rerun the core ASR engine matrix.
  • #496 — replicate the large-v3 insertion/runaway result on a second corpus and CPU.
  • #507 — verify novelty and update related work.
  • #508 — generate manuscript tables/figures directly from committed result artifacts.
  • Freeze exact manuscript evidence commit, commands, lockfile and result manifest.
  • Re-run consistency/claim guards before final numbers are copied into the private manuscript.
  • #498 — controlled typing-vs-dictation human study.
  • #500 — consented spontaneous-microphone robustness evaluation.
  • #502 — far-field meeting diarization evaluation.
  • #504 — full end-to-end Windows/macOS validation.
  • #506 — CPU core-seconds and energy/battery measurements.

These are scientific evidence tasks. They are separate from authorship/publication approval.

Publication/governance track

Follow #484 and authorship/:

  1. freeze the contributor/candidate roster;
  2. collect explicit authorship opt-in and metadata;
  3. circulate one review draft to all confirmed authors;
  4. resolve requested changes;
  5. freeze source commit + final PDF hash;
  6. receive unanimous approval of the same final fingerprint;
  7. publish only the approved artifact.

Private emails, private consent messages, participant data and raw approval evidence stay outside public Git.

Evidence rules

For every central result, require:

claim → artifact → corpus/population → condition → uncertainty → caveat

Do not:

  • turn a point-estimate ranking into statistical superiority without evidence;
  • call LibriSpeech test-other “real-world dictation”;
  • attribute large-v3 instability to host load without a causal experiment;
  • claim disabling previous-text conditioning generally lowers WER;
  • call streaming universally faster;
  • infer full cross-platform interaction from decode-only artifacts;
  • claim the four-speaker cap improves DER at the current threshold;
  • call the current AMI DER state of the art;
  • use “first” or “novel” until #507 verifies the literature.

Definition of paper-v2 readiness

The manuscript is evidence-ready when:

  • the central result matrix is frozen and reproducible;
  • the large-v3 tail claim is independently replicated or explicitly kept workload-specific;
  • every headline figure/table regenerates from public JSON;
  • the claim ledger reflects the final evidence;
  • novelty wording has a current literature check;
  • threats to validity are explicit;
  • no withdrawn claim survives in the abstract/conclusion.

Publication readiness is a separate gate governed by the authorship protocol and #484.