Chinese support — test and validation plan¶

Status: Proposed release protocol
Purpose: Define what must be tested in CI, what must be benchmarked offline, what requires real desktop hardware, and what requires native-language review before first-class Mandarin support is claimed.

1. Validation philosophy¶

Chinese support has four independent failure classes:

1. Configuration failure — the application enters an impossible model/language state.
2. Recognition failure — speech is decoded inaccurately or unstably.
3. Semantic-control failure — a command is misclassified or normal prose executes an action.
4. Delivery failure — correct Han text is recognized but cannot be injected into the target application.

A single “transcribed one Chinese sentence” test covers only a small part of class 2.

The release gate therefore combines deterministic CI tests, reproducible model benchmarks, real-platform injection tests, and native-speaker review.

2. Test layers¶

3. Unit-test plan¶

3.1 Language profile aliases¶

Create tests/test_language_profiles.py.

Required cases:

en -> en
EN -> en
zh-CN -> zh-CN
zh-cn -> zh-CN
zh-Hans -> zh-CN
zh_hans -> zh-CN
zh-TW -> zh-TW
zh-Hant -> zh-TW
zh_hant -> zh-TW

Refusals:

• empty request;
• unknown profile;
• zh-HK;
• yue until implemented.

For zh-HK, assert the error text mentions both: - Mandarin + Traditional option (zh-TW); - Cantonese is separate/not P1.

Do not test only exception type; the remediation is part of the user contract.

3.2 Resolver: default English -> Simplified Mandarin¶

Given default Config(), assert plan contains exactly relevant mutations:

• stt.model: base.en -> small;
• stt.language: en -> zh;
• stt.chinese_script: "" -> simplified.

Assert it does not alter:

• device;
• compute type;
• beam;
• vocabulary;
• microphone;
• hotkey;
• injection backend;
• unrelated command/app profile.

3.3 Resolver: default English -> Traditional Mandarin¶

Same, script target = traditional.

3.4 Resolver preserves compatible model¶

Examples:

• current small + en -> zh-CN keeps small;
• current large-v3 + en -> zh-CN keeps large-v3 if capability layer declares it compatible;
• current small + zh -> en keeps small.

3.5 Recommended-model mode¶

• zh-CN --recommended-model resolves to supported Chinese baseline;
• en --recommended-model resolves to English baseline;
• explicit --model and recommended mode conflict.

3.6 Requirement detection¶

Mock cache/dependency probes:

• small cached + OpenCC installed => no unsatisfied requirements;
• model absent => one model requirement;
• OpenCC absent => one optional-extra requirement;
• both absent => both reported;
• probes perform no network.

3.7 Status derivation¶

Test:

• exact en;
• exact zh-CN;
• exact zh-TW;
• custom compatible Chinese model;
• zh + base.en incoherent;
• Chinese script set with en/base.en;
• command language override en while speech zh;
• command language auto -> zh.

4. Transaction tests¶

Create tests/test_language_apply.py.

Use a temporary config file and injected collaborators for downloads/install/restart.

4.1 No prerequisite side effects in dry-run¶

Assert: - no package install; - no model download; - no file write; - no restart.

4.2 Model download failure rolls back fully¶

Initial file checksum before call == checksum after failure.

Also parse final config to ensure no hidden partial write.

4.3 Optional dependency failure rolls back fully¶

Same invariant.

4.4 Candidate parse/validation failure¶

Inject invalid candidate rendering or validator failure. Original survives byte-for-byte.

4.5 Atomic successful commit¶

Observe config file externally only after complete candidate is ready.

At no point should a watcher see language=zh while model remains base.en.

A unit test can assert implementation uses same-directory temp + os.replace/equivalent. An integration test can watch the file if practical.

4.6 Concurrent edit¶

Simulate:

1. plan generated from version A;
2. another process changes unrelated setting to version B;
3. apply starts.

Required behavior: re-read and re-resolve so unrelated change is preserved; do not overwrite from stale in-memory config.

If project locking makes step 2 impossible during application, test lock acquisition/release instead.

4.7 Restart behavior¶

• zero mutations => no restart;
• successful mutation => one restart;
• --no-restart => no restart, clear result flag;
• restart failure => committed config remains syntactically valid and error is distinguishable from config failure.

5. Command grammar tests¶

5.1 First refactor gate: English parity¶

Before adding Chinese rules, extract current English grammar behind registry and run:

• all existing grammar tests;
• contract/vectors/grammar.json;
• generated contract-vector comparison if project tooling supports it.

No behavior changes should be mixed into the extraction PR.

5.2 Chinese fixture file¶

Add a language-specific vector file, for example:

contract/vectors/grammar-zh.json

Schema should include:

{
  "id": "zh-save-simplified",
  "language": "zh",
  "text": "保存文件",
  "expected": {
    "intent": "edit",
    "action": "save",
    "args": {}
  }
}

Each semantic action has:

• Simplified natural phrase;
• Traditional natural phrase where wording/script differs;
• ASR punctuation variant;
• spacing variant around digits/Latin identifiers;
• negative near-match/prose example.

5.3 Command precision first¶

Command false positives are more damaging than missed commands because a missed command is typed text, while a false positive changes application state.

Build a negative corpus of ordinary Mandarin sentences containing words such as 保存, 删除, 复制, 运行 in non-command contexts.

Release criterion for deterministic Tier-1 fixtures: zero false positives on the curated negative set.

5.4 Numeral tests¶

Test at least:

• 第1行, 第 1 行;
• 第一行;
• 第十行;
• 第十一行;
• 第二十五行;
• 两行 where grammatically accepted;
• out-of-range/malformed numeral stays dictation or non-match.

5.5 Chinese punctuation¶

Outer punctuation accepted:

• 。
• ！
• ？
• full-width variants produced by ASR.

Interior punctuation/filename characters must not be blindly stripped.

5.6 Terminal/safety commands¶

For every Chinese phrase mapping to run_command or other terminal action:

• prove same CommandIntent.action;
• prove it traverses existing command-safety gate;
• negative tests for prose;
• no catch-all regex broader than the English equivalent.

If this cannot be established in P1, omit those Chinese terminal rules and document partial parity.

6. STT wiring tests¶

Generalize existing test_stt_language.py where useful.

With fake WhisperModel:

• language="zh" reaches transcribe;
• reaches transcribe_words;
• reaches decode_window;
• translate task still omits fixed source language;
• initial prompt survives;
• Han wrapper converts final text/words/windows.

Profile-to-engine integration:

default config
 -> resolve zh-CN
 -> candidate SttConfig
 -> build_engine(fake model)
 -> transcribe kwargs contains language=zh
 -> output normalization is Simplified

English regression: - default engine still sees language=en; - default model remains base.en; - build_engine(SttConfig()) behavior unchanged.

7. Import and dependency regression tests¶

A fresh default English process should not import:

• opencc;
• funasr;
• qwen_asr;
• torch solely due to language support.

Possible test:

• start subprocess importing core/CLI status;
• inspect sys.modules or use import sentinel;
• assert language resolver modules remain stdlib/project-only.

This protects startup time and package independence.

8. CLI integration tests¶

Use Click test runner / project CLI harness.

language list¶

• stable IDs;
• zh-CN/zh-TW visible;
• zh-HK not available;
• no model import/download.

language status¶

• coherent exit;
• incoherent exit;
• JSON schema;
• custom-model indicator.

language set --dry-run¶

• exact plan;
• no write.

Real apply with fakes¶

• writes coherent config;
• restart called;
• output names one-time artifact;
• --no-download refuses cache miss.

Error usability¶

Snapshot/substring tests for: - base.en override with zh; - network/model failure; - optional package unavailable; - ambiguous zh-HK.

9. Settings UI/controller tests¶

Qt-free controller/model tests first.

• language profile choices;
• current custom-compatible state;
• plan preview;
• confirmation requirement for downloads;
• cancel = zero mutation;
• apply delegates to shared resolver;
• model dropdown compatibility after profile selection;
• selecting UI language never calls STT language setter.

Minimal Qt smoke test verifies widgets reflect controller data; business logic remains Qt-free.

10. File-transcription validation¶

Mock integration¶

With fake engine, verify:

• config zh applies when no CLI override;
• --language en overrides config for one operation;
• override does not rewrite global config;
• script conversion is applied when appropriate;
• metadata records effective values.

Real benchmark¶

Take a subset of the Mandarin benchmark corpus through the actual file CLI and compare output to direct engine harness. Differences should be explainable by file preprocessing only.

11. Meeting validation¶

Both live and post-pass must record/use effective Mandarin configuration.

Tests:

• fake live path returns Han;
• fake post-pass returns Han;
• quality gate still functions on Chinese transcript;
• no English-specific word-token assumption causes false degeneration where CJK spacing differs.

This is a confirmed blocker in current main, not only an audit hypothesis. meeting/quality.py::tokenize() uses [^\\W\\d_]+; with ordinary unsegmented Chinese, a whole Han run between punctuation marks becomes one token. store.live_word_count() calls the same tokenizer. The existing THIN_MAX_WPM and n-gram thresholds were calibrated on English meetings, so a healthy Mandarin meeting can appear artificially sparse and the live/batch second-opinion ratio is not a language-independent count. Before claiming Chinese Meeting Mode, implement a CJK-valid quality unit (language-aware segmentation or a separately justified character-rate design), re-fit thresholds on healthy + failed Mandarin meetings, and add tests proving healthy Chinese is not marked thin while repetition collapse still fires.

Long meeting real test: - >=30 min Mandarin recording or legally usable corpus composition; - no repetition collapse; - live and batch artifacts; - resource monitoring; - script consistency.

12. Postprocessing audit¶

Search every postprocess/filter for English/whitespace assumptions.

Required review categories:

• filler-word removal;
• sentence casing/capitalization;
• spacing between bursts;
• punctuation cleanup;
• vocabulary correction;
• prosody formatting;
• number/date normalization;
• code formatting;
• meeting word counting;
• transcript quality metrics.

For each component classify:

Create a machine-readable or Markdown compatibility matrix from this review.

13. Injection validation matrix¶

Use a known test string containing:

你好，世界。简体中文測試123，YazSes。

For Traditional target, use reviewed equivalent after conversion.

Test:

Linux X11¶

• default injection backend;
• clipboard backend;
• browser text field;
• GTK text field;
• Qt text field;
• VS Code/electron.

Linux Wayland¶

• ydotool/current backend;
• clipboard fallback;
• same app categories where available.

macOS¶

• native injection backend;
• clipboard fallback;
• TextEdit/browser/editor.

Windows¶

• native injection backend;
• clipboard fallback;
• Notepad/browser/editor.

Record: - exact bytes/text expected vs actual; - lost characters; - reordered punctuation; - line break behavior; - latency for 10/100/1000 Han characters.

No backend gets a green status from “ASCII worked.”

14. Mandarin benchmark protocol¶

14.1 Corpus requirements¶

At least two different evaluation conditions:

• clean/read or clean controlled speech;
• spontaneous/conversational or accented speech.

Prefer an additional noise/far-field set.

Do not mix train/dev material used by a candidate with a supposedly independent evaluation without disclosure.

14.2 Corpus manifest¶

Commit metadata, not restricted audio:

{
  "corpus": "...",
  "version": "...",
  "split": "...",
  "license": "...",
  "sha256_manifest": "...",
  "n_utterances": 500,
  "duration_hours": 1.2,
  "reference_script": "simplified"
}

14.3 Text normalization¶

Publish scoring normalization code.

Score at least: - Unicode normalized consistently; - punctuation-preserving CER; - punctuation-stripped CER; - raw script; - requested script after OpenCC; - digits policy explicitly stated.

Never silently convert references in a way that makes errors disappear without reporting raw results too.

14.4 Error metrics¶

Per model: - substitutions; - deletions; - insertions; - CER; - sentence exact match; - hallucination/repetition count; - empty decode count.

Subsets: - short (<3s); - medium; - long; - numbers; - Latin technical terms; - accent/noise if labels available.

14.5 Determinism¶

Decode each selected stress subset multiple times for models/settings known to vary.

Report: - distinct hypothesis count; - CER range; - repetition incidents.

15. Performance benchmark¶

Reference machine record:

• CPU model;
• physical/logical cores;
• RAM;
• OS/kernel;
• power mode;
• Python;
• CTranslate2/faster-whisper version;
• compute type;
• thread count.

Measure: - cold model load; - warm model load; - ⅖/10/30s utterances; - p50/p95 decode time; - RTF; - release-to-text; - peak RSS; - CPU-seconds; - disk model size.

Run enough samples to avoid single-utterance timing claims.

16. Alternative-model benchmark¶

Same audio and scoring pipeline for:

• Whisper small baseline;
• large-v3-turbo;
• large-v3;
• SenseVoiceSmall prototype when available;
• Paraformer candidate when available;
• Qwen3-ASR-0.6B when available.

Do not change VAD/segmentation differently per model without recording it as a distinct pipeline arm.

A model’s upstream benchmark is background context only; promotion is based on YazSes harness results.

17. Native-speaker acceptance protocol¶

At least:

• one Simplified Chinese reviewer;
• one Traditional Chinese reviewer independently.

Prefer more than one speaker/accent.

Review checklist:

Dictation¶

• common prose;
• punctuation;
• numbers;
• names/English technical terms;
• correction experience;
• script consistency.

Commands¶

• phrase naturalness;
• alternatives speakers actually say;
• ambiguous commands;
• Traditional wording, not mechanical character conversion only.

UX¶

• language switch wording;
• Mandarin vs Chinese terminology;
• zh-TW wording;
• zh-HK/Cantonese explanation;
• diagnostics understandable.

Reviewer result should be documented without publishing private audio.

18. English regression gate¶

Run full existing suite.

Additionally: - default Config values exact; - default model artifact unchanged; - startup import regression; - representative English grammar vectors; - English language set no-op behavior; - Chinese profile code absent from hot decode path when unused.

Performance smoke: - compare baseline English release-to-text before/after implementation on a fixed small sample; - investigate >5% systematic regression in median wall time attributable to code change.

19. Packaging validation¶

For each supported distribution path that can exercise Settings/language setup:

• pip/pipx/uv;
• Snap;
• Flatpak where applicable;
• macOS app/package;
• Windows app/installer;
• distro packages as appropriate.

Questions: - can OpenCC extra be installed? - if not, is it bundled or does the profile give exact remediation? - can model download write to cache? - can air-gapped provisioning work? - does writable config transaction work in sandbox? - does restart work? - are non-ASCII CLI/UI strings rendered?

A packaging path may support Chinese with a documented fallback, but cannot silently enable an unavailable extra.

20. Release evidence artifact¶

Suggested output directory:

design/chinese/results/<release-or-date>/
    README.md
    environment.json
    corpus-manifest.json
    recognition.json
    performance.json
    commands.json
    injection-matrix.md
    native-review.md

If repository policy prefers paper/results for machine artifacts, put JSON there and link from this directory. Follow the established public/private visibility contract.

21. Release decision checklist¶

Chinese may move from experimental/manual to supported only when:

• profile resolver tests pass;
• config transaction failure tests pass;
• English contract parity passes;
• Chinese command positive/negative vectors pass;
• STT wiring all surfaces passes;
• postprocess compatibility audit complete;
• file transcription real test passes;
• meeting audit/real test passes or meeting Chinese is explicitly excluded;
• model benchmark complete;
• performance gate complete;
• X11 result recorded;
• Wayland result recorded;
• macOS result recorded;
• Windows result recorded;
• fallback documented for any red backend;
• Simplified native review complete;
• Traditional native review complete;
• English regression complete;
• exact model/tool licenses recorded;
• offline-after-cache test passes;
• support documentation accurately names remaining limits.

Unchecked items are not “follow-up polish” if they affect the scope being claimed; they either block release or narrow the documented support matrix.