Mandarin validation plan, evidence schema and release gates¶

Status: Proposed P1 verification plan (2026-10-29), not a record of completed testing. Governing ADR: v2-144; evidence owners: #497 foundation, #499 commands, #501 compatibility, #503 ASR, #505 release. Release scope: Mandarin + Simplified/Traditional output, one verified surface at a time.

Verification rules¶

• Treat unit test, packaged integration, real model, real OS/hardware, native speaker, and release signoff as distinct gates. A passing mock or pre-existing source file must never be presented as real-world product validation.
• No human reviewer may be replaced by machine translation, synthetic Chinese phrases, an LLM's confidence or an English maintainer's intuition.
• Default English and the offline/privacy contracts are non-negotiable blocking regressions.
• Every P1 support row needs a named owner, test/protocol, artifact path, date, model revision, OS details if relevant, result, limitations and reviewer signoff. Missing evidence means “experimental/unverified”.
• Run the complete repository gates from AGENTS.md with an up-to-date checkout before merging production code. If a gate cannot run, state why and do not claim it passed.

Layer 1: model-free CI (required for every PR)¶

Ordinary offline tests must be deterministic and contain neither private speech nor model downloads. Qt-dependent tests legitimately skip where Qt extra is absent; a successful skip is not a GUI smoke result. Negative cases that expose current bugs should first be committed as a red reproducer in a targeted PR, then fixed in a separate code review where practical.

CI commands¶

uv sync
uv run python -m pytest \
  tests/test_language_profiles.py \
  tests/test_language_service.py \
  tests/test_configedit_atomic.py \
  tests/test_cli_language.py \
  tests/test_cli_language_set.py \
  tests/test_chinese_command_grammar.py \
  tests/test_mandarin_command_numerals.py \
  tests/test_mandarin_command_safety.py \
  tests/test_han_script.py \
  tests/test_cjk_text.py \
  tests/test_cjk_transcripts.py \
  tests/test_meeting_cjk_word_count.py \
  tests/test_bench_mandarin.py -v
uv run python -m pytest tests/ -q
uv run ruff check src tests scripts
uv run mypy src

When config/CLI/schema changes, regenerate docs with uv run python scripts/gen-docs.py or make docs as directed by AGENTS.md; do not edit generated reference docs by hand. Any new source dependency needs the project's optional-extra, import and egress gates.

Layer 2: language-profile installation and lifecycle¶

For each claimed distribution/channel and the three profiles en, zh-CN, zh-TW:

1. Start from a fresh English config, record installed package/version and base base.en state.
2. language set zh-CN --dry-run: verify zero file, package, network, cache, daemon changes.
3. language set zh-CN --no-download --no-install without artifacts: verify refusal and byte-identical config.
4. Explicitly approve preparing the script extra and multilingual model; verify cached artifact and coherent one-shot config.
5. Start/restart the daemon, dictate a fixed test sentence, verify status/doctor; separate configured from running-effective model.
6. Switch to zh-TW with existing small: only script changes; no redundant model download.
7. Switch to English in preserve mode: keep compatible multilingual model; optionally --recommended-model to return to base.en.
8. Concurrent CLI + Settings edit: preserve unrelated changes, one writer wins safely, no partial profile.
9. Fail model download, dependency install, config write and process restart separately: record config bytes, cache state, daemon state and messages.
10. Remove network after cache provisioning and repeat dictation/file/meeting supported flows; no hidden fetch or telemetry.

Important: package installs and downloads may remain after a failed later step. That is an expected outcome of the current config-only transaction; the UI must report it accurately.

Layer 3: real recognition and quality (mandatory for a “dictation supported” label)¶

Use the model protocol. Run real authorized Mandarin audio with the shipping adapter and pinned weights. At minimum cover zh-CN and zh-TW references, short/long speech, mixed Han+Latin, numbers/names, accent/noise, silence, word timestamps, streaming and file transcription. Record cer_raw, cer_normalized, cer_script_normalized, p50/p95 latency, RTF, RSS and core-seconds.

Set quantitative accept/reject SLOs before evaluating the holdout. Suggested governance: the release manager pre-registers hardware classes, expected max latency and memory, minimum speaker/sample counts and tolerable absolute CER with native reviewers; the repository currently lacks enough representative Mandarin data to defend one universal absolute CER threshold. Do not pick a threshold after viewing model results.

A model-specific advantage cannot compensate for failed Unicode injection, unsafe commands or data egress. Report all failures, not only successful clips.

Layer 4: native phrase and usability review¶

At least one independent native reviewer for Simplified Mandarin and one for Traditional Mandarin must execute:

• Installation, first model/cache preparation, dry-run and offline-after-cache, language switch both ways;
• Natural phrase round-trip for every shipped commands/grammars/zh.py::PHRASE_REVIEW action family (all currently provisional); NUMERAL_REVIEW for counts and line numbers;
• Negative set: ordinary prose, quoted instructions, sentence-long occurrences of command verbs, navigation/file-name phrases, disfluencies, homophones and incomplete fragments;
• Dictation in a text editor, browser input and office/document field, including Simplified and Traditional punctuation/IME interactions;
• Return to English without losing previous model/settings and without unexpected command execution.

A phrase is accepted only after native reviewer disposition and matching positive/negative tests. One English speaker's literal translations are insufficient. Use the privacy-safe protocol at human-validation-data-protocol.md. Store approval as a dated decision + reviewer consent; do not require publication of legal names or private emails.

Command false-positive policy¶

A false positive for a destructive/high-impact command is a release blocker on that action family. No broad “Chinese commands supported” claim while any P1 phrase remains unreviewed. A deliberate narrow safe-core subset may be supported with all other families explicitly excluded. The English Tier-2 classifier must remain disabled for Mandarin until a separate safety ADR and benchmark clear it.

Layer 5: desktop Unicode and physical application matrix¶

There are no submitted platform reports in docs/research/mandarin-platform/ as of this audit; keep their status unverified.

Fixed public strings: 你好，世界。今天测试 YazSes 123。; 你好，世界。今天測試 YazSes 123。; 版本 v2.0，文件 test.py，编号 12345。; plus a non-BMP emoji and a burst with a Han prefix/suffix around Latin text. Each report must say exact observed output, not only “worked”; record whether mixed-mode typing duplicated a prefix, lost focus, or required clipboard.

A failed backend is not a reason to hide evidence: mark unsupported or restrict the product's support claim to a verified fallback. Do not classify an OS supported based solely on a test mocking its injector.

Layer 6: cross-surface matrix and regression¶

Evidence artifact schema¶

Store aggregate results under tracked public design/docs/result locations only when safe. Suggested machine-readable report:

{
  "schema": "yazses.mandarin.release/1",
  "date_utc": "YYYY-MM-DD",
  "yazses_commit": "<immutable SHA>",
  "profile": "zh-CN",
  "surface": "hold-to-talk",
  "platform": {
    "os": "<name+version>",
    "session": "<X11|Wayland|macOS|Windows>",
    "app_versions": ["<target app version>"]
  },
  "model": {
    "id": "small",
    "revision": "<immutable model revision>",
    "license_id": "MIT",
    "weights_sha256": "<sha256>"
  },
  "corpus_sha256": "<authorized corpus manifest SHA256 or null>",
  "evidence": ["<relative public artifact path>"],
  "result": "pass|fail|blocked|not-run",
  "limitations": ["<named limitation>"],
  "review": {
    "native_simplified": "pending|approved|not-applicable",
    "native_traditional": "pending|approved|not-applicable",
    "maintainer": "pending|approved"
  }
}

This is a proposed release-report format, not an implemented validator or a claim that those files exist. Raw audio, private transcripts, personal home paths, emails and biometric data must not be committed.

Release checklist and signoff (ADR-v2-144)¶

• All governing ADR decisions reviewed; scope and every deferred capability named.
• CI gates G01–G12 green against exact release commit; full English tests green; lint and import budgets green.
• Model/corpus license provenance pinned and benchmark JSON validated with a frozen split.
• Release hardware SLOs pre-registered and met for each claimed device tier.
• Simplified and Traditional native reviewer approval recorded separately.
• Every advertised platform/session injection report exists and shows tested applications.
• Real short-file and streaming finalization complete; meeting only if separately cleared.
• No undesired network egress; fully offline after provisioning.
• Installer, error-recovery, dry-run, rollback-to-English, and no-restart states verified.
• Product docs clearly distinguish Mandarin and script, commands preview scope, Cantonese exclusion.
• Each surface marked supported/experimental/excluded with citations to exact evidence.
• Maintainer signs/date-stamps release decision and known exceptions; no blanket “Chinese supported” without that matrix.

Until all relevant conditions pass, the release statement is: “YazSes has experimental offline Mandarin dictation with Simplified/Traditional output; availability and quality vary by tested model, application and platform.” Do not imply the same validation for Cantonese, arbitrary code switching, every Mandarin accent or every desktop target.