Small and quantized ASR models and GGUF-class runtimes — what can be compared fairly¶

Date: 2026-11-05
Status: Research note (not a decision). ADR-030 freeze unchanged; no default is changed.
Tracking: NEXT-ASR-03 #794, parent #787; siblings #792, #793, #795, #797, #798
Repository snapshot: main at a241a77b6c59d65fb9265a0475d42f1a97fdfe6a (YazSes 2.48.1)
Builds on, and does not repeat: the pinned evaluation protocol, the runtime and model provenance snapshot (NeMo-Speech.cpp, parakeet.cpp, Moonshine Voice, onnx-asr runtime rows and Nemotron/Parakeet licence rows live there), and Proposed ADR-033.

Labels: [Verified-primary-source] read on the upstream page on 2026-11-05; [Upstream-claim] a number or promise by the upstream author that YazSes has not reproduced; [Verified-in-code] read in this repository at the snapshot; [Assumption]; [Unknown]. Fetches were text summaries of each page, so every upstream page is "not read in full". No model was downloaded or run, and this note contains no measurement of its own. Upstream figures are quoted with the page they came from and are never YazSes results.

1. Answer to the issue question¶

Question (#794): do 4/8-bit or mixed-precision inference reduce download and RSS without unacceptable regression on names, code, accented speech or long prompts?

• Download size: yes, as an upstream claim. Every runtime read below publishes quantized files that are smaller than their float counterparts (section 3). [Upstream-claim]
• RSS: [Unknown] for YazSes. No source read reports peak RSS under a method that satisfies ADR-033 (process tree, sampled maximum, labelled method). The one upstream RSS figure found (faster-whisper README, small, int8, 1477 MB, "8 threads on an Intel Core i7-12700K", beam 5) is a different machine, model, decoder setting and method from YazSes's. [Upstream-claim]
• Regression on names, code, accented speech, long prompts: [Unknown]. None of the pages read measures those slices for a quantized-versus-float pair. The only quantized-versus-float accuracy table found is Moonshine's, on LibriSpeech test-clean (read speech). [Upstream-claim]
• The quantization recipe matters more than the bit-width label. Moonshine states that Tiny Streaming scored 7.57% before 2026-07-30 because one scale factor was used per weight tensor, and 4.83% after per-channel scales; its float reference is 4.52% (reproduced) / 4.49% (paper). [Upstream-claim] CTranslate2 says int8 quantizes "only embedding and linear layers" and claims "little to no degradation" without giving a measurement. [Verified-primary-source for the wording; Upstream-claim for the effect]
• Recommendation (adopt / defer / reject): defer any new engine; adopt a configuration-only quantization grid over engines YazSes already ships as the first measured study (section 6, option A). Whether that study is worth running is the owner's call.

2. What YazSes already has (so "new" means something)¶

Consequence. Any candidate that is not one of those three engines needs a factory.py branch, a config value, a feature-registry entry and an extra: all src/ changes, forbidden by the RESEARCH class under ADR-030. They are measurable only by an out-of-tree harness until a decision says otherwise. Two kinds of candidate need no code at all: a different compute_type, and a different CTranslate2 checkpoint named in [stt] model. [Verified-in-code; the second is an Assumption until a run resolves the model through the factory, per ADR-033's resolved-engine rule]

3. Candidate grid (new rows only)¶

Columns: licence is as stated by the upstream page, not legal advice. "Size" is an [Upstream-claim] and mixes MiB and MB as the page does. Format is what the page says it is.

Rows already covered in the provenance note (not repeated): NeMo-Speech.cpp (Q8_0 is the only quantization it names, CPU row "27 ms per 160 ms chunk", CPU model unnamed, per the repo page) and Moonshine Voice runtime/streaming behaviour.

3.1 How each would sit in the optional-extra / lazy-import / offline-cache model¶

This restates AGENTS.md rules 2, 3 and 6 against each row; it is a design note, not a plan to build.

• C1 whisper.cpp. [Assumption] Shape of an adapter: an SttEngine implementation under stt/, imported inside build_engine so a missing package raises ModuleNotFoundError and the factory falls back with its existing log line; a new optional extra and a stt-* feature entry. Offline cache: a single .bin file, hashable, verified before use; a Hub download would be a separate operator step (the evaluation protocol's rule), never a silent fetch. A native binary or third-party binding adds a platform matrix YazSes would own.
• C2 CTranslate2 types. [Verified-in-code] Nothing to add: compute_type already flows to the loader. The open item is only that an unsupported value raises ComputeTypeUnsupportedError with a message naming valid types, which makes an "unsupported" cell recordable as such rather than as an error rate.
• C3, C4, C5 Whisper variants. [Assumption] Same engine, different [stt] model; no extra needed. The language guard treats a custom name ending in .en as English-only and reports other custom names as unknown rather than refusing, so a multilingual claim for C5 would be unchecked by the guard. [Verified-in-code] language/capability.py
• C6 Moonshine. [Verified-in-code] The adapter accepts only names starting moonshine/ and otherwise uses moonshine/base; it ignores initial_prompt and returns no word timings. The streaming checkpoints are not reachable through this adapter.
• C7 sherpa-onnx. [Assumption] The diarization extra shows the pattern for naming sherpa-onnx-core explicitly (the pyproject.toml comment records an install failure when it was omitted), which any ASR use would inherit. Model files would be local-directory loads, matching the cache-first rule.
• C8 Vosk. [Assumption] Would be a separate native dependency family (not ONNX), so it carries the largest new-packaging cost per unit of expected benefit among the rows.
• C9 parakeet.cpp. [Assumption] As C1, with GGUF weights fetched from a Hub repo that the upstream page names; per-file licences must be recorded, since conversion does not change them.

4. Excluded or held back, and why¶

5. What is missing to compare each candidate fairly under ADR-033¶

Common to all rows (the ADR's own gate, restated only to say what is absent):

• Revision pin: no row has been resolved to an immutable model revision and file SHA-256; "latest" release names (whisper.cpp v1.9.5, parakeet.cpp v0.6.1, Moonshine Voice v0.1.5) are not full commit SHAs. [Unknown]
• Normaliser: upstream WERs use their own (Open ASR Leaderboard average for Moonshine English; LibriSpeech test-clean for Vosk and the distil usage snippet; Fleurs and CV17 for Vosk Persian). None is the ADR-033 pinned Whisper EnglishTextNormalizer profile. Upstream numbers must not be pooled with each other or with YazSes's. [Verified-primary-source for the dataset names; Assumption that the normalisers differ]
• Corpus and leakage: nothing is established about which candidates trained on LibriSpeech or Common Voice. [Unknown]
• Hardware: all upstream speed and memory figures came from other machines (e.g. an i7-12700K; "8 threads on a 20-core x86 host" for parakeet.cpp; an unnamed CPU for NeMo-Speech.cpp). Nothing exists for the 8 GB / 16 GB CPU-first laptops ADR-033 names. [Verified-primary-source for the stated hardware]

Per-row gaps:

Names, code and accent slices are absent from every row: no upstream page read states a measurement on spoken identifiers, accented English or prompts longer than 30 s for any quantized build. That is the issue's core question and it can only be answered by the P4 constructed-stress and P1/P2 slices in the evaluation protocol, plus a code/identifier set that does not yet exist (belongs to #793/#797). [Unknown]

5.1 Sketch of the pinned grid, if an owner authorises a run (design only, nothing executed)¶

Factors are separate columns so that no result credits the wrong cause:

Negative cases to keep as first-class rows (ADR-033 item 3): unsupported compute type on a CPU, load failure, factory fallback, empty output, catastrophic insertion, a model licence that blocks the arm.

Stop/go thresholds are not proposed here. The only placeholders in the repository are ADR-033's +0.5 percentage-point and 20% p95 margins, which that ADR itself marks as unapproved; this note neither adopts nor adjusts them.

6. Decision options for the owner (no recommendation to ship anything)¶

Reading of the evidence, labelled as such. [Assumption] A is the lowest-cost, highest-validity first step, B the next; C-E should wait for the #793 analyzer and a second-machine rerun. Owner decision needed on: (1) whether A/B are authorised as RESEARCH-class runs on named hardware, (2) which English slice is primary (ADR-033 acceptance question), (3) whether any candidate beyond MIT/Apache/CC-BY-4.0 licence classes is admissible.

7. Evidence that would change the ranking¶

• A same-checkpoint, same-machine int8 versus float32 result on the P4 stress and identifier slices showing a large regression (that would demote A's premise and raise the value of per-channel or mixed recipes, as Moonshine's own history suggests).
• A resolved confirmation that onnx-asr can run Parakeet unquantized (or that only int8 files exist), which changes whether A's Parakeet half is a valid pair.
• Per-model licences for sherpa-onnx families (promotes or removes D).
• An independent reproduction of the "WER 0 against NeMo" claim at q4_k/q5_k (promotes E), or a counter-example.
• A legal determination on Phonon/Redux weights (reopens the Hold row).
• A confirmed GGUF loader in whisper.cpp (upstream pages read describe ggml .bin only), which would merge C1 and C9 into one format family; absence of that is the current status. [Unknown beyond the pages read]
• Real user need: evidence from consented, non-telemetry channels (for example volunteered issue reports) that disk size or RSS blocks installs; none was consulted for this note, so the value of a footprint win is currently a hypothesis. [Unknown]

8. Primary sources read (2026-11-05, text summaries, not read in full)¶

whisper.cpp README, releases, models README, HF ggml files; CTranslate2 quantization doc; faster-whisper; Distil-Whisper cards for small.en and large-v3 and the CT2 conversion; whisper-large-v3-turbo; Moonshine repo, available models, accuracy, licence; sherpa-onnx and its pretrained model index; Vosk repo and models page; onnx-asr (README and docs home; the quantization parameter is not documented there) and the Parakeet v2 ONNX card (file list and sizes not visible); parakeet.cpp and its licences page; NeMo-Speech.cpp.

Not obtained: the PyPI page for useful-moonshine-onnx failed to load (licence taken from installed metadata instead); parakeet.cpp quantization.md, NeMo-Speech.cpp BENCHMARK.md, Moonshine's quantization doc, and the full GitHub release notes for whisper.cpp were not read. A web search found no evidence that whisper.cpp reads GGUF, but it returned mirrors and forks and is not conclusive.

Status line. Research note (not a decision). ADR-030 freeze unchanged; no default is changed. No model downloaded, no benchmark run, no src/ change; every number above is an upstream claim attributed to its page.