# Paper v2 claim ledger This ledger is the control surface for manuscript claims. It prevents three common failures: 1. promoting an exploratory probe into a universal statement; 2. quoting a point estimate without the population/condition that produced it; 3. retaining an attractive claim after a later rerun contradicted it. Use the status labels literally: - **SUPPORTED** — current artifacts support the claim in the stated population/condition. - **SUPPORTED WITH QUALIFICATION** — a narrower version is defensible; wording must preserve the qualification. - **HYPOTHESIS / MECHANISM** — plausible and evidence-informed, but not established as a general effect. - **REFUTED / WITHDRAWN** — an earlier interpretation is contradicted or unsupported and must not reappear. - **OPEN** — important question with no adequate experiment yet. ## A. Core ASR claims ### C-ASR-01 — clean-speech engine comparison **Status:** SUPPORTED WITH QUALIFICATION **Claim:** On the measured 200-utterance LibriSpeech `test-clean` subset on the Azure Xeon CPU, Parakeet TDT 0.6B v2 had the lowest WER point estimate (2.06%), while Moonshine/tiny had the lowest RTF (0.016). **Evidence:** - `paper/results/probes/wer-vm-clean.json` (current uncontended eight-engine clean matrix; promote/freeze under #495 before submission) - `paper/results/wer.json` is the older v1 three-checkpoint laptop result and is not the source of this claim **Required qualification:** - Parakeet's interval overlaps `large-v3` and `small.en`; do not convert the point estimate into universal statistical superiority. - RTF is machine-specific. **Allowed manuscript wording:** > On the shared CPU harness, Parakeet produced the lowest clean-speech WER point estimate, whereas Moonshine/tiny was the fastest decoder; confidence intervals preclude treating the WER ranking as a universal ordering. **Do not write:** > Parakeet definitively beats all Whisper models. --- ### C-ASR-02 — hard audio changes the ranking/robustness picture **Status:** SUPPORTED **Claim:** Moving from `test-clean` to `test-other` increased WER for every measured engine, but by materially different factors; Parakeet had the smallest measured multiplier (about 1.4×), while several engines degraded by roughly 2–2.5×. **Evidence:** - `paper/results/wer.json` - `paper/results/wer-test-other.json` **Required qualification:** `test-other` is harder read speech, not spontaneous microphone dictation. --- ### C-ASR-03 — `large-v3` repeated WER variation is insertion-driven **Status:** SUPPORTED **Claim:** In repeated `large-v3` decodes of the measured `test-other` subset, substitutions, deletions, and hits remained fixed while insertion count changed substantially, so the observed WER spread was driven by extra emitted text rather than recognition substitutions. **Evidence:** - `paper/results/probes/largev3-instability-test-other.json` - related decode mechanism artifacts **Key numbers:** - substitutions = 87 across the compared repeats - deletions = 15 - hits = 3619 - dedicated four-repeat probe: insertions = 101–144 and WER = 5.46–6.61% - the earlier matrix run that triggered the probe reached 184 insertions and 7.69% WER; report it separately from the controlled repeat range **Required qualification:** this is a finding for the measured software/model/corpus conditions. It is not yet a claim about every Whisper `large-v3` implementation. --- ### C-ASR-04 — high machine load caused the `large-v3` instability **Status:** REFUTED / WITHDRAWN AS A CAUSAL CLAIM **Reason:** the 7.69% run was taken under unusually high load, but lower-load repeated runs still vary substantially. Load is a plausible moderator, not an established cause. **Allowed wording:** > The highest observed WER coincided with high host load, but similar-load repeats still varied; the experiment therefore does not identify system load as the cause. --- ### C-ASR-05 — previous-text conditioning is checkpoint-dependent **Status:** SUPPORTED WITH QUALIFICATION **Claim:** The measured effect of `condition_on_previous_text` changes across checkpoints: it helps `base.en`, has a small benefit on `small.en`, is byte-identical on `medium.en`, and is associated with rare runaway repetition on `large-v3` under the tested conditions. **Evidence:** - `paper/results/probes/decode-determinism-base.en-test-other.json` - `paper/results/probes/decode-determinism-small.en-test-other-baseline-no_context.json` - `paper/results/probes/decode-determinism-medium.en-test-other-baseline-no_context.json` - `paper/results/probes/decode-determinism-large-v3-test-other.json` - `paper/results/probes/decode-determinism-large-v3-test-other-no_context.json` - decode mechanism artifacts **Required qualification:** disabling conditioning on `large-v3` should be framed as preventing a rare catastrophic repetition mode, not as a statistically established average-WER improvement. **Do not write:** > Turning off context improves Whisper accuracy. --- ### C-ASR-06 — temperature fallback is the root cause and should be disabled **Status:** REFUTED / WITHDRAWN **Reason:** the `temperature=0.0` arm removes the rescue path and can make results dramatically worse. The fallback participates in variability but also rescues failed first-choice decodes. **Allowed wording:** > The fallback exposes stochasticity after a failed deterministic decode, but disabling it is not a remedy because the failed first-choice decode remains. --- ### C-ASR-07 — greedy decoding is an acceptable free speed-up **Status:** SUPPORTED WITH QUALIFICATION / GENERALLY FALSE ON DEFAULT MODEL **Claim:** On `base.en`, beam 1 is measurably worse than modest beam search on both clean and hard splits, for only an 11–16% speed improvement. **Evidence:** - `paper/results/beam-test-clean.json` - `paper/results/beam-test-other.json` - paired significance artifacts **Key paired results:** - clean beam 1 vs beam 5: +0.386 WER points, 95% CI [0.069, 0.726], p=0.0192 - hard beam 1 vs beam 5: +1.102 points, 95% CI [0.305, 2.196], p=0.001 **Qualification:** this is checkpoint-dependent. `small.en` on clean speech does not establish the same penalty. --- ## B. Reproducibility and platform claims ### C-REP-01 — one WER number is not portable across all CPUs **Status:** SUPPORTED **Claim:** The same checkpoint/subset can produce slightly different WER across OS/ISA environments; the size of that spread is model-dependent. **Evidence:** - `paper/results/platforms/*/wer.json` **Key figures on the 60-utterance cross-platform subset:** - `tiny.en`: 3.39–3.88% - `base.en`: 3.25–3.39% - `small.en`: 2.05% on all four measured runners **Allowed conclusion:** cross-platform numerical sensitivity decreases across these three checkpoints in this experiment. **Do not write:** all models are platform independent. --- ### C-REP-02 — cross-platform decode results prove full cross-platform operation **Status:** REFUTED AS AN INFERENCE **Reason:** benchmark decoding does not exercise global hotkeys, actual microphone capture permissions, text injection, GUI behaviour, daemon lifecycle, accessibility APIs, or recovery paths. **Open validation needed:** end-to-end scripted or human-in-the-loop validation on Windows and macOS. --- ### C-REP-03 — provenance/history changed the scientific conclusion **Status:** SUPPORTED AS A METHODOLOGICAL SYSTEMS CONTRIBUTION **Claim:** Retaining displaced benchmark files and recording exact commands/corpus identity exposed contradictions that would otherwise have been overwritten, notably the `large-v3` 4.86% vs 7.69% hard-split disagreement. **Evidence:** - `paper/results/probes/logs/x86b-other_wer.log` — matrix run 1, at 4.86%. This is the whole of run 1: the JSON was overwritten before the retention mechanism existed, so the disagreement is evidenced by a console log, not by `history/`. - `paper/results/wer-test-other.json` — matrix run 2, at 7.69% - `paper/results/history/` — the retention mechanism that postdates this loss, carrying six later displaced runs - `paper/results/MANIFEST.md` - benchmark provenance implementation **Caveat when writing this up:** the claim is that the *record* of the disagreement survived, not that the mechanism saved it. It did not — it was written because this one nearly was not. **Writing angle:** reproducibility infrastructure is not administrative metadata; it materially altered the interpretation. --- ## C. Latency and interaction claims ### C-LAT-01 — synthetic pre-speech silence recovers clipped first words **Status:** REFUTED **Evidence:** - `paper/results/onset.json` - `paper/results/onset-significance.json` **Finding:** no lead-in condition survives multiplicity correction as better than no lead-in; lost speech cannot be reconstructed by adding silence after the fact. **Important privacy implication:** genuine pre-key recovery requires buffering real audio before activation, which violates the current “not listening while key is up” boundary. --- ### C-LAT-02 — streaming universally reduces perceived latency **Status:** REFUTED / TOO BROAD **Evidence:** - `paper/results/streaming.json` - current `docs/benchmarks.md` rerun **Narrow supported claim:** streaming can expose substantial partial text on very fast checkpoints such as `tiny.en`, but on `base.en` the rolling CPU decode often fails to stay ahead of the microphone stream and makes final commit slower. **Do not write:** streaming is faster. --- ### C-LAT-03 — current streaming result is definitive across machines **Status:** OPEN / NEEDS REPLICATION **Reason:** n=15, one main machine, read speech. The effect direction is large but should be replicated on at least one second CPU and with spontaneous speech before being elevated to a broad interaction claim. --- ## D. Diarization claims ### C-DER-01 — old clustering default was unsuitable for real meetings **Status:** SUPPORTED **Claim:** On the full 16-recording AMI test split, the former 0.5 clustering threshold massively over-split speakers, producing 75.21% mean DER and an average +155.19 speaker-count error; raising the meeting threshold to 1.2 reduced mean DER to 26.71%. **Evidence:** - `paper/results/diarization-ami16_corpus-der.json` - ADR-v2-133 - related probe results **Preferred paper-comparable figure for current system:** 27.37% time-weighted DER, 20.51% at 250 ms collar. **Qualification:** AMI headset mix is cleaner than a far-field table microphone; 26–27% DER is not a claim of state-of-the-art quality. --- ### C-DER-02 — a four-speaker clustering cap improves DER at the current threshold **Status:** NOT ESTABLISHED **Evidence:** - `paper/results/diarization-ami16_corpus-maxspk4-vs-der-significance.json` **Results:** - 7 recordings better, 7 worse, 2 unchanged - sign test p=1.0 - mean ΔDER +2.71, CI [-1.64, +7.60] **Supported replacement claim:** `max_speakers=4` is a clustering ceiling, not an exact-count oracle. It makes the estimated count exact on 15/16 recordings instead of 2/16; the one remaining recording has three reference speakers and is over-counted by one. The sample does not resolve a DER benefit. --- ### C-DER-03 — one threshold is suitable for all diarization domains **Status:** REFUTED **Evidence:** synthetic, VoxConverse, and AMI sweeps. **Observed operating points:** - synthetic ~0.8–0.9 - VoxConverse ~0.9 - AMI ~1.2 **Allowed conclusion:** clustering calibration is domain-sensitive; feature-specific defaults are better justified than one global constant. --- ### C-DER-04 — centroid similarity can safely repair split speakers **Status:** REFUTED FOR THE TESTED SIMPLE HEURISTIC **Evidence:** - `paper/results/centroid-merge-ami16_corpus-meeting.json` - `paper/results/centroid-merge-ami16_corpus-recimport.json` - `paper/benchmark/analyze_centroid.py` **Conclusion:** no useful threshold repaired enough split-speaker clusters without unacceptable wrong-person merges. **Research value:** keep this as a negative result, not an omitted failed idea. --- ## E. HCI / user-value claims ### C-HCI-01 — YazSes is faster than typing in actual use **Status:** OPEN **Reason:** no controlled user study has measured composition + correction + final error cost in YazSes. The literature can motivate a hypothesis about speech bandwidth, but it cannot substitute for a YazSes experiment. **Needed experiment:** within-subject typing vs. dictation, same tasks, final corrected output, entry rate, correction time, error cost, subjective workload, and learning effects. --- ### C-HCI-02 — Dysfluency-Friendly Mode benefits people who stutter or have dysarthria **Status:** OPEN FOR USER BENEFIT; TECHNICAL FILTER GATE SUPPORTED **Supported technical result:** on the repository's small test set, false collapse = 0/33 and recall = 92.9% on 28 dysfluent clips. **Not supported:** clinical/accessibility benefit, usability, or generalization across speakers. **Needed study:** participant evaluation and/or external representative corpus. --- ### C-HCI-03 — hold-to-talk is superior to toggle/always-on interaction **Status:** HYPOTHESIS / DESIGN RATIONALE The HCI literature and privacy boundary support the design decision, but YazSes has not run a controlled comparison of activation modes. --- ## F. Candidate manuscript headline claims These are the strongest claims currently ready to anchor the abstract/results: 1. **A shared offline CPU harness produces materially different robustness conclusions on hard speech than clean speech alone.** 2. **Repeated `large-v3` failures in the measured dictation workload are driven by variable insertions/runaway continuation rather than substitutions.** 3. **Previous-text conditioning is checkpoint-dependent and can trade a small average benefit on smaller models for rare catastrophic repetition on a larger checkpoint.** 4. **Cross-platform numeric variation exists but is checkpoint-dependent, so third-decimal WER claims should not be transported blindly across hardware.** 5. **The original meeting clustering default failed catastrophically on real annotated meetings, and real-data measurement changed the shipped operating point.** 6. **A four-speaker clustering cap greatly improves count correctness, but does not establish a diarization-error benefit.** 7. **Several intuitive interventions—synthetic onset silence, generic CPU streaming, and simple centroid merge—do not survive direct measurement.** 8. **Preserving contradictory reruns and full provenance changed the scientific interpretation and should be treated as part of the evaluation method.** ## G. Claims that need a literature search before “novel/first” language Before submission, assign a related-work owner to verify: - reports of faster-whisper/CTranslate2 repeated decode instability; - previous-text conditioning causing repetition loops in Whisper-family models; - insertion-only variance under temperature fallback; - CPU cross-engine comparisons among Whisper, Parakeet TDT, and Moonshine; - diarization threshold domain transfer and speaker-count constraint interactions; - negative results on CPU local-agreement streaming in interactive dictation. Until then, use “we observe,” “we measure,” “in our experiments,” and “to our knowledge” only after explicit verification.