Skip to content

ADR-033 — A reproducibility gate for offline ASR comparisons (proposed)

Not to be confused with ADR-v2-033 (wake word), which is in the separate v2 series.

Status: Proposed (2026-11-03) · no runtime, model default, publication or implementation authorization
Decision owner: YazSes maintainer (pending review)
Research: #793, 2026-11-03 protocol
Context: ADR-030, ADR-021, ADR-011, ADR-019, engine/default decision #645, ASR epic #787

Context

YazSes currently has three offline STT backends (faster-whisper, Parakeet, Moonshine), a public community WAV/TXT benchmark script, and published historical multi-engine results. Its published benchmark page explicitly distinguishes the single-utterance community smoke test from the historical 200-utterance evaluation and says the primary historical harness and row-level artifacts are held outside the public repository. The community harness does not yet produce complete pinned, scored, per-sample results suitable for a third-party rerun.

The factory can transparently fall back to faster-whisper if an optional backend fails to load; this is appropriate for dictation availability but makes a benchmark invalid unless the resolved engine is asserted. The script can also skip an unreadable input and scores under a different text normalizer if whisper-normalizer is missing. Decode RTF and p50/p95 by themselves do not measure user release-to-final latency. These facts motivate a research evaluation policy, not an engine change.

Proposed decision (review required)

Establish protocol v1 in the linked research document as the common gate before YazSes publishes new cross-engine or default-engine comparative conclusions:

  1. Each model evaluation must pin exact YazSes commit, candidate and resolved engine/model identity, model weight digest, dependency/runtime versions, CPU and OS, decoder settings, corpus and split manifest, and normalization profile.
  2. A public candidate score must be reproducible from public benchmark source and licensed-public/synthetic per-sample result evidence. A private archive may be referenced as historical evidence but cannot replace this gate for new comparative claims.
  3. Per-sample failures and fallback are first-class data; they cannot disappear from the denominator or become scored output from a different engine.
  4. Report matched-corpus WER/CER and S/D/I, language/script preservation, repeatability, per-condition coverage, and speaker/group-block paired uncertainty. Report WER on the designated completed cohort and failures/coverage separately. Distinguish clean/read, spontaneous, multilingual, synthetic and user-study evidence.
  5. Report warmed decode separately from cold model load, key-release-to-final, safe-sink delivery time, p50/p95 and sampled peak RSS; label unmeasured or unsupported metrics explicitly. Only claim energy when instrumented; otherwise label CPU time a compute proxy.
  6. All benchmarking requires explicit authorized local dataset/model preparation, verified offline execution and no automated upload of user audio, transcripts or sensitive metadata. Engineering QA, synthetic work and participant research must keep their different evidentiary/consent classes.
  7. The method is read-only with respect to product architecture: no new default, model download, engine integration, telemetry, or release commitment. A change under #645 / #798 needs an independent accepted decision consistent with ADR-030.

Provisional engineering margins in the research report are open discussion inputs, not approved pass/fail thresholds and not claims about user preferences.

Alternatives

A. Continue with one WER leaderboard and informal laptop measurements

Reject. Incorrect normalizer/fallback identity, incomplete samples, corpus overlap and unreported latency tails can reverse the ranking. These measurements are useful exploratory signals, not a reproducible platform-agnostic decision.

B. Make the existing private historical harness the exclusive source of truth

Reject as a policy for future public comparisons. Existing historical data remains valuable and may be accurately quoted, but external researchers cannot independently inspect its executable version and per-sample result archive from a public YazSes checkout. No obligation to publish private archives, user data or restricted audio is created.

C. Replace the installed speech engine immediately based on present historical WER

Defer. #645 is the canonical default-engine delivery decision and ADR-030 requires measuring user-visible costs and failure honesty first. Historical Parakeet/Whisper comparisons do not settle every hardware, language or latency condition.

D. Add production telemetry to collect live error rates

Reject. Not necessary for public/synthetic reproducible research and contrary to the zero-telemetry offline boundary.

E. Adopt a pinned, offline, public-result benchmark gate (this proposal)

Recommend adopting as a documentation/research standard after review. Reuses shipping engine seams and existing tests; requires a small independent evaluation/analyzer implementation, not a feature expansion.

Consequences if accepted

Benefits: attributable comparisons, independent replication, safer model recommendations, honest unsupported cases, publication-ready provenance, a common measure across #792/#793/#796/#797/#798.

Costs: preparing manifests, verifying licences and weight hashes, multiple hardware runs, private-source exclusions, and statistical uncertainty that may yield “inconclusive.” Some past headline numbers remain historical rather than meeting this future standard.

Non-goals: no benchmark result produced by this ADR; no retroactive claim that existing published results were fabricated; no public participant corpus; no cloud inference; no new ASR API; no CI downloading gigabytes of models.

Acceptance questions for the decision owner

  • Do we agree to require immutable corpus/model provenance, exact resolved backend, public per-sample error counts and a public deterministic analyzer for future comparative claims?
  • Which English corpus and primary cost metric best represent the initial user decision: normalized WER, release-to-final p95, or an error-cost-weighted pre-registered pair?
  • Are provisional +0.5 percentage-point WER non-inferiority and 20% p95 improvement useful starting discussion margins, or should they remain entirely unspecified until user-workflow data exist?
  • Which actual CPU hardware will rerun the same pinned manifest, and what unsupported cells must be kept as N/A?

Follow-up if accepted

  • 793: freeze v1 schema, preregistration, synthetic pure analyzer and its tests, exact corpus sample ID manifests; run local English pilots and report negative results.

  • 792/#794/#795: candidate provenance/quantization/hardware details before adding arms.

  • 796: streaming and hesitation replay using distinct UX metrics.

  • 797: per-language/locale evaluation protocol, script preservation and leakage register.

  • 798/#645: deliberate adoption/defer/reject decision on new runtime/default only after measured matched evidence; record an ADR-030 amendment if the breadth freeze requires it.

Status and provenance

Research and architectural reasoning from current repository docs and code as of main commit 1598e65b708b86c2f8d5fa51749407e4a18740be, 2026-11-03. No benchmarks have been executed for this ADR; no user/participant audio acquired; no code or default changed. Status remains Proposed, not Accepted.