August 2026 Azure measurement campaign¶

This document reconstructs the measurement campaign behind most of the new paper-v2 evidence. It is written so an author or reviewer can tell what ran where, in what order, which artifacts are exploratory, and which conclusions are safe.

1. Campaign window and machines¶

The main rented-compute window was 2026-08-23 through 2026-08-24, followed by analysis and a small number of follow-up measurements through 2026-08-26.

The probe archive states that the exploratory runs were produced during a two-day Azure window on two rented 16-vCPU Xeon machines in westeurope. The stable artifacts identify the principal host as:

• Azure Standard_D16s_v6
• Intel Xeon Platinum 8573C
• 16 logical CPUs
• about 67.4 GB RAM
• Ubuntu 24.04.4 LTS
• Azure kernel
• CPU inference, int8
• faster-whisper 1.2.1
• CTranslate2 4.8.1 for the principal ASR matrix
• YazSes around 2.29.0–2.31.0 during the measurement window, as recorded per artifact

The exact version, load average, command, corpus identity, and library versions must be read from each artifact's provenance block. Do not replace those fields with one campaign-wide value.

2. Why Azure compute does not contradict the product privacy claim¶

YazSes' product claim is that normal dictation/transcription can run locally without sending a user's audio to a cloud service.

The research campaign used rented cloud compute to evaluate public benchmark corpora and synthetic test material. The paper should state this plainly so readers do not confuse “offline runtime architecture” with “all experiments were physically executed on the author's laptop.”

Recommended wording:

The evaluated product path is local/offline at runtime. For the expanded benchmark campaign we additionally rented CPU-only Azure VMs to run public and synthetic corpora under controlled, reproducible hardware conditions; no user dictation data were uploaded for these experiments.

Synthetic meeting audio generated with Azure Speech TTS is a benchmark fixture, not evidence about privacy of real user audio and not a substitute for human meeting evaluation.

3. Corpora used and the question each one answers¶

LibriSpeech test-clean¶

• 200 utterances in the main matrices
• deterministic speaker-stratified selection
• clean read audiobook speech
• useful for model comparison and repeated-decode mechanisms
• not a real-world dictation estimate

Question: Under controlled clean speech, how do local engines/configurations compare?

LibriSpeech test-other¶

• 200 utterances
• 33 speakers
• about 23.4 minutes
• same scoring/normalisation family, but harder audio than test-clean

Question: Does the clean-audio ranking and reproducibility survive a harder split?

It is a robustness stressor, not spontaneous microphone speech.

AMI test split¶

• 16 real meeting recordings: 15 with four reference speakers and one with three
• 543.7 minutes of recordings
• 30,714 s / 8.5 h of scored reference speech in the archived aggregate
• human RTTM annotations from the pyannote AMI diarization setup
• headset mix, cleaner than a single far-field table microphone

Question: Does Meeting Mode's diarization configuration work on real annotated meetings, and what type of error dominates?

VoxConverse subset used by the plausibility/threshold work¶

Used to test whether heuristics tuned on meetings transfer to crowded/broadcast-style speaker distributions.

Question: Is a clustering threshold or “implausible label” guard domain-general?

Synthetic meeting corpus¶

Generated dialogue + Azure neural TTS, used as a controlled regression fixture and as the earliest diarization probe.

Question: Does the harness detect gross regressions under known synthetic conditions?

It must never be averaged with AMI DER or presented as evidence of real-meeting accuracy.

4. Campaign chronology — how the questions evolved¶

The commit and artifact history is scientifically useful because it shows that the campaign changed hypotheses rather than simply confirming them.

Phase A — make the benchmark capable of falsifying documentation¶

Before the rented runs, the harness was extended so it could:

• compare engines rather than only YazSes against itself;
• identify the selected corpus and command in provenance;
• survive one engine failure without losing completed rows;
• run test-other, not only test-clean;
• score real annotated diarization, not only synthetic meetings.

This matters because several later “findings” are actually corrections to claims the old harness could not test.

Phase B — engine matrix and harder speech¶

The two main WER matrices established:

• eight engine/checkpoint rows on clean speech;
• the same family on test-other;
• confidence intervals;
• real-time factor on the Azure Xeon.

The hard split immediately showed that: - model degradation is not uniform; - clean-speech ordering is not enough; - large-v3 can move dramatically between identical runs.

Phase C — repeat the disagreement instead of choosing a convenient run¶

The disagreement was very nearly lost, and that is the point. The test-other matrix ran twice on the same instance; run 2 overwrote wer-test-other.json in place, so run 1 — the 4.86% large-v3 measurement — survives only as the console log paper/results/probes/logs/x86b-other_wer.log. _common.write_result was then changed to copy a displaced result into paper/results/history/ precisely so this cannot recur. The mechanism postdates the loss: cite the log for run 1, not history/, which holds no wer-test-other entry.

This is a core methodological contribution of the campaign: a contradictory rerun became the research question.

Follow-up probes measured: - repeated hypothesis sets; - insertion/substitution/deletion components; - temperature fallback behaviour; - thread-count hypotheses; - previous-text conditioning; - per-utterance concentration of the aggregate WER effect.

Phase D — 2×2 decoder mechanism and checkpoint ladder¶

The initial hypothesis “temperature fallback is the cause” was too simple. Disabling the fallback made some runs dramatically worse.

The more informative intervention was condition_on_previous_text=False, which: - removes the measured large-v3 runaway repetition mode; - slightly harms the default base.en; - has a smaller effect on small.en; - produces byte-identical output at medium.en.

That is why paper v2 should present the mechanism as an interaction among checkpoint behaviour, previous-text conditioning, and the fallback rescue path, not “sampling is bad.”

Phase E — test platform portability rather than assuming it¶

The benchmark workflow was dispatched to: - Linux x86-64, - Linux arm64, - macOS arm64, - Windows x86-64.

Separately, dependency resolution was probed across platform/extras combinations. This distinguishes: 1. “the decoder yields similar text on this ISA,” from 2. “this optional feature can actually be installed here.”

Those are different claims and should stay different in the paper.

Phase F — challenge product assumptions¶

The campaign then tested assumptions that had been encoded as defaults/documentation:

• beam width,
• pre-speech padding,
• streaming,
• diarization clustering threshold,
• a max_speakers=4 clustering cap,
• plausibility guards,
• centroid-based repair.

The result is intentionally mixed. Several defaults survived; several rationales did not.

5. Stable artifacts vs. probes¶

Tier 1 — manuscript-grade stable measurements¶

Prefer these for headline claims:

• paper/results/wer-test-other.json
• the final frozen clean eight-engine matrix to be produced under #495

The current uncontended clean eight-engine evidence is paper/results/probes/wer-vm-clean.json. It is reproducible and suitable for drafting, but it remains under probes/; do not misidentify the older paper/results/wer.json (the v1 three-checkpoint laptop run) as its source. - paper/results/beam-test-clean.json - paper/results/beam-test-other.json - paper/results/onset.json - paper/results/diarization-ami16_corpus-der.json - paper/results/diarization-ami16_corpus-maxspk4.json - paper/results/platform-resolution.json - paper/results/platforms/* - paper/results/plausibility-*.json - paper/results/streaming.json when its exact run conditions are stated

Tier 2 — derived statistical analyses¶

These are analyses of the same observations, not new measurements:

• paper/results/beam-*-significance*.json
• paper/results/onset-significance.json
• paper/results/diarization-*-significance.json

When writing, never count a measurement and its significance file as two independent experiments.

Tier 3 — exploratory/probe evidence¶

paper/results/probes/ contains the mechanism-finding work. Many of the most interesting reproducibility results live here.

A probe may become a central paper result if all of the following hold:

• the question and intervention are explicit;
• corpus and machine provenance are recoverable;
• the result is repeated or has an independent validation condition;
• the probe's conclusion is not contradicted by a later stable harness run;
• any exploratory multiple-testing risk is acknowledged;
• the paper labels it as a follow-up/mechanism experiment, not a preregistered primary endpoint.

Tier 4 — history and logs¶

• paper/results/history/: displaced result snapshots; scientifically important when disagreement itself is the finding.
• paper/results/probes/logs/: operational trace; use to reconstruct runs, not as the sole citation for a headline number.

6. Experiment-to-artifact map¶

7. Statistical/reporting rules for this campaign¶

WER¶

• Always name corpus/split and n.
• For cross-engine statements, prefer interval-aware language over rank language.
• Do not treat a tenth of a WER point across different hosts as a finding.
• For repeated unstable models, report a distribution/range rather than silently selecting one run.
• When a mean change is concentrated in a few utterances, say so.

Paired ASR comparisons¶

Use the paired bootstrap artifacts when the same utterances are decoded under two conditions. The pairing is the point: independent bootstrap samples answer a different, weaker question.

Onset / first-word accuracy¶

Use first-word paired outcomes and the multiplicity-corrected McNemar interpretation. Do not use whole-utterance WER to claim an onset effect the setting cannot directly cause.

Diarization¶

Report both: - time-weighted DER for comparison with conventional published diarization results; - per-recording mean when describing a meeting-level user experience.

Never compare a synthetic-corpus DER directly with AMI DER.

For paired meeting conditions, use per-recording paired inference; do not infer from two corpus means alone.

Multiple explorations¶

The campaign contains hypothesis-generating probes. The manuscript must distinguish: - primary/repeated result, - post-hoc mechanism probe, - product decision informed by evidence.

A product decision can be reasonable even when a mean effect is not statistically resolved; it must not be relabelled as a significant scientific effect.

8. Claims the campaign supports¶

The campaign supports these broad statements:

1. Audio difficulty changes model conclusions.
2. Some decoder/model combinations have meaningful run-to-run tail failures not visible in one average WER.
3. The measured large-v3 instability is insertion/continuation variance, not substitution variance.
4. A decoder setting can help one checkpoint and hurt another.
5. Cross-platform numerical variation is model-dependent.
6. Real-meeting diarization required a very different clustering operating point from the original default.
7. Speaker-count constraints and better diarization are not the same objective.
8. Several intuitive latency/robustness features fail when measured under the actual CPU budget.
9. Keeping contradictory reruns and full provenance materially changed the conclusions.

9. Claims the campaign does not support¶

Do not claim:

• a universal best ASR engine;
• statistical superiority of Parakeet over every other engine from the 200-utterance clean sample;
• real-world spontaneous dictation WER from LibriSpeech;
• that large-v3 instability is caused by Azure load;
• that disabling conditioning universally lowers WER;
• that beam 5 is uniquely optimal;
• that 300 ms of silence “recovers” a clipped first word;
• that the four-speaker cap improves DER at the new threshold;
• that 26–27% AMI DER is state-of-the-art or “good”;
• that cross-platform decode artifacts prove all end-to-end interaction paths;
• a productivity, accessibility, or usability advantage over typing without a user study.

10. Reproduction before manuscript freeze¶

Before submission, rerun or verify the central artifacts against a frozen commit/tag and record that freeze in the paper:

1. create a manuscript tag or exact commit SHA;
2. regenerate paper/results/MANIFEST.md;
3. ensure every central result carries argv, corpus identity, versions, and machine;
4. rerun the main WER matrices if dependency versions changed materially;
5. rerun the large-v3 mechanism on at least one additional independent corpus if feasible;
6. verify that the platform artifacts were produced by the same manuscript commit or explain differences;
7. keep all contradictory/displaced runs;
8. generate manuscript tables directly from JSON wherever possible rather than copying values by hand.