Benchmarks¶

Every number on this page was measured on real hardware with the shipping code, and every one can be reproduced with the commands at the bottom. Nothing here is estimated, extrapolated, or measured on a machine chosen to flatter the result.

These are the same measurements reported in the accompanying paper (arXiv:2607.28878).

Accuracy — word error rate¶

Dataset: LibriSpeech test-clean (Panayotov et al., ICASSP 2015), 200 utterances across 40 speakers, chosen by deterministic speaker-stratified round-robin so the sample is not cherry-picked. 1,808 seconds of audio in total. Text is normalised with Whisper's own EnglishTextNormalizer before scoring, which is the standard that makes these numbers comparable with other Whisper results.

LibriSpeech is read audiobook speech in clean conditions. Your dictation WER will be worse than this — a real microphone in a real room with spontaneous speech is a harder problem than the benchmark. Treat these as a comparison between models, not as a promise about your desk.

Speed — how long until the text appears¶

Decode time for a single utterance, measured end-to-end on 30 utterances (median duration 7.3 s):

Read this table before choosing a model. The honest summary:

• tiny.en feels instant, and costs you about 2.2 points of WER against small.en.
• base.en is the default because it is the best compromise: noticeably more accurate than tiny.en, and still returns a typical utterance in about a second and a half.
• small.en is the most accurate and is the right choice for transcribing files or meetings — but at a 5-second median for live dictation it is slow enough to break your flow. Use it for yazses transcribe and Meeting Mode, not for typing.

Change it with [stt] model in your config, or see performance tuning.

Everything that isn't the model is free¶

The rest of the pipeline is irrelevant to your latency. Measured over 2,000 repetitions each:

That is roughly 0.02 % of base.en decode time. All latency is the speech model; optimising anything else would be pointless.

Memory¶

The 820 MB baseline is the Python process with its dependencies loaded, before any model. This is the honest cost of a Python daemon — it is why the requirements say 4 GB minimum and 8 GB comfortable.

Voice-activity gate¶

The gate that decides whether a recording contains speech at all, tested against 200 LibriSpeech clips as positives and digital silence plus Gaussian noise at 0.1×, 0.25×, 0.5× and 0.75× the threshold as negatives:

The 2.4× margin is why yazses mic-level --set matters: the default threshold works because typical speech sits well above it, but a quiet voice or a low-gain microphone can fall below it — which shows up as Silent audio -- discarding in the log rather than as an error.

Command recognition¶

Whether a spoken phrase is correctly routed to type this text or run this action. Tested on 68 command phrases and 80 lines of ordinary dictation:

The zero false-positive rate is the number that matters. A dictation tool that occasionally interprets your sentence as "delete the last three words" is worse than one that occasionally fails to recognise a command — the remaining 7.4 % of commands get typed as text, which is annoying but harmless. The optional Tier-2 SLM router exists to catch those.

Dysfluency-Friendly Mode¶

The opt-in pass that collapses stutters and repeats (b-b-because → because), tested against a pre-registered gate defined before the evaluation was run (ADR-015):

The false-collapse rate is the safety-critical one: the feature must never mangle fluent speech. The sample is small — 61 clips — so treat the recall figure as indicative rather than precise.

Reproducing all of this¶

The harness lives in the repository, reuses the shipping code rather than a reimplementation, and writes JSON with a provenance block recording the exact hardware and library versions.

git clone https://github.com/MSKazemi/yazses
cd yazses
uv sync --group benchmark

# one-time: fetch LibriSpeech test-clean (346 MB)
cd paper/data
curl -fsSL -O https://www.openslr.org/resources/12/test-clean.tar.gz
tar -xzf test-clean.tar.gz
cd ../..

# run everything
uv run python paper/benchmark/run_all.py --wer-n 200 --lat-n 30 --vad-n 200

# or a single experiment
uv run python paper/benchmark/bench_wer.py 200
uv run python paper/benchmark/bench_latency.py 30

Results land in paper/results/*.json. If your numbers differ materially from this page, that is worth an issue — different CPUs are expected to differ, but a large gap on the same class of hardware means something is wrong and we would like to know.

What is not measured here¶

Being explicit about the gaps, because a benchmark that only reports its wins is not a benchmark:

• No real-world dictation WER. Everything above uses read audiobook speech. Spontaneous speech into a laptop microphone in a normal room is harder, and we do not have a licensed corpus to quantify it.
• No diarization accuracy. Meeting Mode's speaker separation has not been scored against a labelled multi-speaker corpus. It works well in practice; "well in practice" is not a number.
• No macOS or Windows figures. All measurements are Linux.
• Single machine. One CPU model, one run. There is no variance estimate across machines.

Contributions on any of these are welcome, and reporting results from your own hardware is a genuinely useful first contribution.