Multilingual, accent and code-switch evaluation design for offline ASR¶

Date: 2026-11-05 Status: Research note (not a decision). ADR-030 freeze unchanged. Tracking: #797 under #787; reuses #793 rules; coordinates #503, #505, #258. Builds on: pinned offline ASR protocol (normaliser, failure accounting, pinning, v1 row schema), proposed ADR-033 (Proposed, not Accepted), fa-en code-switch protocol fa-en-cs-v1, Persian benchmark spec, dysfluency protocol (rights-ledger pattern). Base: main at a241a77b. Scope: documentation only. No dataset or model was downloaded, no audio was processed, no score was computed. All web pages were read on 2026-11-05 through a text-summarising fetcher, so each is "not read in full" unless noted; treat every [Verified-primary-source] below as "the summary of that landing page said so", and re-read the page before relying on a number.

Labels: [Verified-primary-source] stated on the landing page/abstract/model card/source file read on 2026-11-05; [Upstream-claim] the upstream authors' own claim, not re-measured; [Verified-in-code] read in this repository at the base SHA; [Assumption] a design choice or inference needing review; [Unknown] not established by anything read.

1. Answer to #797 and recommendation¶

Adopt this note as the multilingual layer on top of the #793 protocol; defer any "supported language" claim, default change or new engine; reject (a) pooling languages into one WER, (b) scoring a refused or unsupported language as 0 or 100 % WER, © using a corpus's marketing "zero-shot" status as proof the engine never saw it. [Assumption: recommendation]

The user-visible failures the issue asks about are mostly not recognizer errors. They are (i) output in the wrong script or language that no WER hides, (ii) orthography damage after correct recognition (Han script, ZWNJ, Yeh/Kaf), (iii) a normaliser that silently rewrites words, and (iv) an engine that never refuses. This note makes each of them a separately reported number. [Assumption]

2. Current YazSes state (audit, [Verified-in-code] unless marked)¶

Engine fact that drives failure accounting: with a non-English [stt] language, an English-only checkpoint "does not fail"; faster-whisper replaces the language by en for a non-multilingual tokenizer, and the Distil-Whisper large checkpoints keep a multilingual tokenizer so nothing warns (capability.py docstring and rows). Parakeet v2 and Moonshine take no language. check() returns supported/unsupported/unknown and callers refuse only on unsupported. [Verified-in-code]

3. Candidate public evaluation sets (read 2026-11-05)¶

Licences are what the landing page printed. "Split" and "overlap" columns say what was stated, not what is true. Nothing here is approved for use; each needs the rights ledger of the dysfluency protocol §2 before any download.

Excluded after a search summary only (not read): single-speaker or non-commercial Persian sets (a CC BY-NC Bible set and a one-speaker CC BY set) fail the speaker-block and licence rules. There is no public Tamil code-switch or Persian-English speech set in what was read [Unknown beyond this search]. Search recall is not a survey.

4. Training-data leakage by engine family¶

A corpus licence says nothing about whether a pretrained model saw it. Rule from #793 §3 stands: a model scored on a corpus from its training family is labelled possibly in-domain, never "zero-shot".

Controls (all [Assumption]): keep a leakage register row per (engine, corpus); prefer corpora released after the model's training cut-off (Neyshekar-type sets) as the leakage-resistant slice; add a private-free fresh slice (§9) written for this study; report in-domain-possible and fresh slices side by side, never pooled. A large gap between them is evidence of exposure, not proof.

5. Per-language normalisation traps¶

The rule from #793 stands: score twice, raw and evaluation-normalised, pin the normaliser name and version per cell, and never apply a production cleaner to a reference. A lowercasing fallback is forbidden.

Whisper BasicTextNormalizer (source read on 2026-11-05 as a summary; verify by running it): NFKC, deletes bracketed text, and replaces every character whose Unicode category starts with M, S or P by a space. With remove_diacritics=True it deletes Mn and spaces Mc/Me. ZWNJ (category Cf) passes through. [Verified-primary-source] Consequences: Devanagari and Tamil vowel signs are Mc/Mn marks, so the normaliser can split an Indic word at its vowel sign, which changes both WER tokens and reference length for Hindi and Tamil. [Assumption: implied by the category rule; test with a Hindi and a Tamil sentence before use] Do not use it for hi/ta unless a test shows otherwise; pin NFC plus whitespace only and report CER over extended grapheme clusters beside WER.

6. Code-switch measurement¶

Reuse fa-en-cs-v1 unchanged for Persian-English and extend by version, not by edit: the 6 categories, script-based token tags (fa/en/mixed/neutral), zero-denominator = null not 0, pooled counts, seeded prompt-id split, and aggregate-only submission [Verified-in-repo]. A change is a new protocol id.

• MER for zh-en: tokens are Han characters and whitespace-delimited Latin words; ASCEND's abstract reports both CER (22.69 %) and "mixed error rate" (27.05 %) for its wav2vec 2.0 baselines [Upstream-claim; the page gives no definition, so this note defines it]. Persian-English MER stays whitespace-token (spec §8). Never compare MER across pairs. [Assumption]
• Per-language WER/CER: attribute each aligned reference token to its script tag and report the Han/Devanagari/Perso-Arabic part and the English part separately, plus overall MER. Script tag is a proxy, not language ID: English written in the native script counts as an error and is also counted as transliteration_rate. Romanised Hindi is tagged English by script, so Hindi-English needs a human-checked reference policy first. [Assumption]
• Boundaries: boundary error rate, recall and precision as in the spec; a hypothesis with no switch where the reference has several is a high-recall-of-nothing case, so always show precision on switch-free control prompts.
• Decoder setting is part of the arm: a fixed [stt] language, or auto-detect, is a different arm. Whisper takes one language per call, so a fixed zh arm can transliterate English and an auto arm can pick the wrong language; report both. [Assumption]
• What can be measured today: only the stock decoder, since [polyglot] is not wired. Any adapter result is a later protocol version. [Verified-in-code]
• #258 (code-switching vs accent as the dominant failure) is a hypothesis this note does not test. Testing it needs a within-speaker paired design: the same speakers read matched monolingual and mixed prompts, so accent and switching effects are separable. Public sets here cannot do that (ASCEND and MUCS have no monolingual twin per speaker). [Assumption]
• Sets: ASCEND (zh-en, CC BY-SA, ~23-26 speakers), MUCS sub-task 2 (hi-en), and fresh prompts (§9). No public ta-en or fa-en speech set was found. Common Voice Code Switching has no release.

7. Statistics: intervals and minimum sizes¶

1. Unit is the speaker block. Pooled edit counts; paired speaker-block bootstrap, 10,000 replicates, recorded seed, percentile 95 % (#793 §6; Liu & Peng, INTERSPEECH 2020 [Upstream-claim]). Bisani & Ney, ICASSP 2004 introduced bootstrap CIs for WER; a search summary (abstract not opened) says sentence-level resampling gave narrower intervals than speaker-level, which is the anti-conservative direction to avoid [Upstream-claim, second hand].
2. Proportions (exact match, ZWNJ preservation, script-collapse, refusal and coverage rates, boundary matches): Wilson or Jeffreys interval, not Wald; Brown, Cai & DasGupta, Statistical Science 16(2), 2001 recommend them for n up to 40 and report erratic Wald coverage [Upstream-claim, via search summary; landing page did not load].
3. Sample-size arithmetic (normal approximation, illustration only, no data behind it): a proportion near 0.5 needs about 385 items for ±5 points (1.96^2 x 0.25 / 0.05^2); one near 0.1 needs about 138 for ±5 points. Speakers needed for a WER difference scale as (1.96 x SD_speaker / half-width)^2: if the between-speaker SD of per-speaker WER were 15 points and the target half-width 3 points, that is about 96 speakers. The SD must come from a dev-half pilot; the 15 is a placeholder. [Assumption]
4. Floors proposed here (engineering, not from a source): at least 30 speakers and 2,000 reference tokens per reported cell for an inferential claim; below that, print the point value labelled "indicative" with n. At least 20 prompts per code-switch category, inherited from fa-en-cs-v1 §3. ASCEND (~23-26 speakers) and L2-ARCTIC (4 speakers per L1) are below the floor for accent or language contrasts. [Assumption]
5. If a corpus exposes no speaker ID (FLEURS: unknown), CIs are labelled utterance-level, anti-conservative, and no engine ranking is drawn from overlapping or near intervals.
6. Multiplicity: primary comparison = one baseline vs one candidate per language, declared before scoring; language x engine x slice grids are exploratory. Compare engines within a language only. Cross-language comparison of WER or CER is descriptive and never ranks languages. [Assumption]
7. Dev/test halves by seeded prompt-id hash as in fa-en-cs-v1 §6; speaker-level split wherever speakers are known.

8. Failure accounting for engines that refuse or mishandle a language¶

Extends #793 §5/§8 and the v1 row status field. A cell is first classified by the capability table verdict, not by what the engine printed.

Accuracy is compared only on the pre-registered completed set; coverage is a separate column. A language with no scored engine appears as an empty row with reasons, not as a missing row.

9. Private-free synthetic and consented evaluation (no real user audio)¶

AGENTS.md forbids sending real audio, transcripts, voiceprints or participant data to a third-party coding tool; this procedure keeps the coding agent on text, code and aggregates. [Verified-in-code, AGENTS.md]

1. Text-first fixtures (agent may work on these). Invented sentences dedicated CC0, per language and per code-switch category, as in fa-en-cs-v1 §4: no names, accounts or private messages. Scorer unit tests run on text pairs only (the ZWNJ, Han-script, mark-splitting, empty-output, unsupported-cell vectors of §5/§8). Counts in the fixture are test vectors, never results.
2. Synthetic audio. Optional: render the fixtures with a locally run text-to-speech model whose licence and training-data provenance were read first. No TTS model was evaluated in this note [Unknown]. Synthetic speech measures the pipeline, normaliser and failure accounting, not human accent or spontaneous behaviour, and must be labelled synthetic in study_mode.
3. Public audio only after the rights ledger (§3 caveats), downloaded by the human operator in a separate preparation step, then run offline per #793 §4.
4. Consented read prompts (human-run, local). Adult volunteers read the CC0 prompts; written consent first, using the fa-en-cs-v1 §5 template, which is still marked DRAFT, owner review required. Pseudonymous codes, option A (aggregate only) as default, audio kept on the collector's machine, deleted on a stated date. Only error counts per category leave the machine; the agent sees the aggregate JSON. Product QA consent is not research consent (ADR-v2-150 as cited in #793).
5. Accent slices without labelling people. Prefer public accent-labelled corpora (L2-ARCTIC locally only). For consented speakers, record self-declared first language as an optional field, never inferred, and never report a group with fewer than the §7 floor as a finding. One person recording alone is one speaker: no inferential claim.
6. Fresh slice. Prompts and recordings made for this study after the model snapshot are the leakage-resistant slice for §4.

10. Protocol checklist¶

• Pre-register languages, engines, corpora, slices, normaliser ids, bootstrap unit/seed, failure policy before scoring.
• Rights ledger per corpus (licence, owner, redistribution, speaker-identification limits); stop on any uncertainty.
• Capability verdict recorded per (engine, model, language) from check(), with source URL and date.
• Resolved engine/model and weight SHA-256 asserted; fallback invalidates the cell.
• Reference field and script pinned per corpus; sample scored under both FLEURS fields.
• Normaliser tested on a Hindi, a Tamil, a Persian-ZWNJ and a Han-script vector before use; Whisper BasicTextNormalizer not used for hi/ta without that test.
• Raw and normalised scores both reported; script-collapse and ZWNJ/Han-script preservation reported separately.
• Leakage register filled; possibly-in-domain cells labelled; fresh slice reported alongside.
• Speaker IDs present, else CIs labelled utterance-level; floors of §7 applied.
• Code-switch: MER, per-language error, boundary metrics, transliteration rate, fixed-language and auto-detect arms.
• Unsupported/unknown/refused cells shown as N/A with reasons; coverage column present.
• Public scorer and row-level counts for licensed-public or synthetic data; no private archive as sole source (ADR-033).
• No audio, transcript or participant data in any commit, issue or agent prompt.

11. What this protocol cannot show¶

• Whether any engine "supports" a language for real dictation: read or prompted speech is not spontaneous, noisy, microphone-varied use.
• Whether a model was or was not trained on a test item. Controls reduce and expose the risk; they cannot remove it.
• Per-accent or per-dialect performance from corpora with a handful of speakers per group, or from self-declared labels.
• Real user error cost: a lower CER does not mean fewer corrections, and typing into applications (Wayland, Windows Unicode injection) is a separate measurement.
• Cantonese, dialects, or any language whose corpus licence or script policy was not read here.
• Latency or memory: this note adds no timing method beyond #793.
• Anything about [polyglot] adapters, which do not exist yet.
• Cross-language ranking: different corpora, scripts and metrics make a WER of one language non-comparable with another.

12. Open questions for the maintainer¶

1. Is a CC BY-SA corpus (ASCEND, MUCS) acceptable as a local evaluation input given that ShareAlike reaches published manifests derived from it?
2. Is AISHELL-1's "Apache-2.0" or "academic use" reading controlling?
3. Does the maintainer accept IndicVoices' gating, and L2-ARCTIC's non-commercial licence, for local-only runs?
4. Should the private benchmark/ scorers be re-implemented publicly before any #797 number is quoted (ADR-033 point 2)?
5. Review and release of the DRAFT consent template before any recording.

Sources (read 2026-11-05, not read in full unless stated)¶

FLEURS card (first 100,000 characters of the raw README only) and abstract; Whisper abstract and HTML text; whisper-large-v3; parakeet-tdt-0.6b-v3; Whisper normalizer source; Common Voice terms and cv-dataset; OpenSLR 94, 33, 104; ASCEND; IndicVoices; Neyshekar; L2-ARCTIC; Golos. Statistics: Liu & Peng, Bisani & Ney, Brown et al. as linked in §7. Repository files cited by relative link. Not fetched successfully: Common Voice main dataset page, the HF Common Voice 17 card (empty, moved to MDC), the Brown et al. landing page.