Persian–English Code-Switch Evaluation Protocol (fa-en-cs-v1)¶

Status: Proposed — protocol only. No prompt has been collected, no recording made, and no model output scored under it. Work package: FA-09 (#518), part of the Persian programme (#483). Companions: persian-benchmark-and-validation.md §5 "Code-switch" (the metric list this makes precise) and §8 (sampling); polyglot-switch.md (the feature being measured). Scorer: paper/benchmark/codeswitch_metrics.py, tested by tests/test_codeswitch_metrics.py on synthetic text.

1. Why the protocol comes first¶

[polyglot] pair = "fa-en" is accepted by the config today, but the router stays dormant until a user supplies [polyglot] adapter_path (src/yazses/polyglot/router.py, PolyglotRouter.active). FA-10 is meant to measure a baseline before any adapter is trained. That measurement only means something if the questions, the categories, the split and the metric definitions are fixed before anyone looks at a model's output. Otherwise the set gets tuned to the model, and the number describes the tuning.

So this document freezes all of them as protocol version fa-en-cs-v1. The rules:

1. Nothing in §3–§8 changes after a result has been inspected. A change, however small, is a new protocol version (fa-en-cs-v2), recorded in §11 with the reason, and the scorer's PROTOCOL_VERSION changes with it. A v1 number and a v2 number are not compared.
2. The held-out half (§6) is not used to choose prompts, thresholds ([polyglot] mer_gate), decoder settings or adapters. Use the dev half for those.
3. A prompt found to be wrong after scoring (a typo in the reference, say) is fixed in a new version of the prompt file, and both scores are reported. It is never edited out silently.

When this file landed, paper/results/ held no fa-en code-switch result.

2. Scope¶

Measured: how well a transcript keeps Persian and English apart and intact when one utterance uses both, scored from text: a reference transcript and a hypothesis per prompt.

Not measured here: acoustic robustness, latency or RTF (bench_persian.py covers those for monolingual Persian), injection or bidi display (persian-text-and-rtl.md), and a decoder's own span labels. The scorer reads script, not a decoder's language tags (§7).

Not claimed: that YazSes handles fa-en code-switching today. Without an adapter the router is inactive, and the [polyglot] config itself records that stock Whisper cannot code-switch (PolyglotConfig docstring in src/yazses/config.py).

3. Content categories¶

Every prompt belongs to exactly one category. These are the six from #518, and the scorer rejects any other value (CATEGORIES in the scorer).

Authoring rules:

• Each reference is written the way a careful bilingual writer would type it: English terms in Latin script with their usual casing, Persian in Persian script with ZWNJ where the word takes one (می‌خواهم). Persian (Yeh U+06CC, Keheh U+06A9) letters, not Arabic ones. Digits in the script the prompt uses. A prompt whose correct written form is disputed (Latin or Persian spelling of a loanword, say) is dropped, because the protocol would score a matter of taste as an error.
• Before any held-out result is quoted per category, each category has at least 20 held-out prompts. This is a protocol rule, not a power calculation; below it a per-category rate is reported with its n and marked indicative. The overall rate carries the per-category n beside it.
• A prompt holds no personal data: no real person's name except public product or organisation names, no address, no account identifier, nothing from a private message.

4. Sourcing¶

In order of preference:

1. Prompts written for this set by a contributor who dedicates them to the public domain (CC0 1.0, https://creativecommons.org/publicdomain/zero/1.0/). This is the default. The prompts are invented sentences of the kinds in §3, so no third party's text is involved and the prompt file can be committed.
2. Openly licensed public text (CC0, CC BY 4.0, CC BY-SA 4.0), adapted into prompts. Each prompt records its source URL, the revision or access date, and the licence. ShareAlike carries over to the prompt file, so CC BY-SA material makes that file CC BY-SA. It goes in a separate file, not mixed into a CC0 one.
3. A public code-switch speech corpus, if one exists with a clear licence. The protocol does not assume one does. If one is adopted, it is pinned (version, split, sample ids, as in persian-benchmark-and-validation.md §2) and reported as its own corpus, never pooled with read-prompt results.

Never used: private chats, emails, work documents, dictation logs, the learning corpus, voiceprints, or any text or audio a person produced without knowing it would be evaluated. AGENTS.md also forbids uploading real user audio or transcripts to a third-party coding agent. That covers any recording made under §5.

The prompt text may be committed when it meets source 1 or 2. Audio is never committed by default, and neither is a hypothesis transcript of a recorded speaker. Both stay on the collector's machine, under the manifest rules bench_persian.py already enforces (licence and consent values required, speaker and session only as opaque pseudonyms, audio read only from the manifest directory or --audio-root; persian-benchmark-and-validation.md §6.1).

5. Recordings, consent and data sharing¶

Recordings are read prompts: an adult volunteer reads prompts from §3 aloud. Nobody records spontaneous speech about their own life for this set.

Every recording session needs written informed consent obtained before recording, using the template below. The participant's sharing choice (A, B or C) is recorded in the manifest's consent field as aggregate-only, share-with-maintainer or publish-cc-by-4.0. The default, when nothing else is chosen, is A.

DRAFT — owner review required¶

This template is a draft. It has not been reviewed by the project owner or by anyone qualified in data-protection law, and it must not be used to collect a recording until that review is done and this banner is removed. Voice recordings are personal data. The collector is responsible for checking this text against the law that applies to them and to the participant (for example the GDPR where either is in the EU) and for adapting it. This draft is not legal advice.

---

YazSes Persian–English code-switch evaluation — participant information and consent

What this is. YazSes is an open-source, offline voice-typing program. We are measuring how accurately it writes sentences that mix Persian and English. You will read [N] short prepared sentences aloud. The sentences are written by the project and contain nothing about you.

Who is collecting. [collector name and contact], a volunteer contributor to YazSes. This collection is [not] / [is] carried out on behalf of an organisation: [name, if any].

What is recorded. Audio of you reading the sentences, plus your choice below. We do not record your name with the audio. Recordings are labelled with a random code. We ask for: [none / age range / device and microphone type].

Where it is kept. On the collector's own computer, [encrypted disk / location]. It is not uploaded to any online service, including AI tools.

How it is used. Speech-recognition software runs over the audio on that computer, and the output is compared with the sentences you read. Only numbers leave the computer: error counts and rates per sentence category, with no audio, no text and no sentence identifiers.

Your sharing choice. Choose one:

• A — Aggregate only (default). My audio stays on the collector's computer and is deleted by [date]. Only aggregate numbers are shared or published.
• B — Share with the maintainer for evaluation only. In addition to A, the audio may be passed privately to the YazSes maintainer, only to re-run this evaluation, and deleted by [date]. It is never published.
• C — Publish under CC BY 4.0. My recordings may be published under the Creative Commons Attribution 4.0 licence (https://creativecommons.org/licenses/by/4.0/), credited as [pseudonym / "anonymous contributor"]. I understand that published audio cannot be fully withdrawn once others have copied it, and that a voice can identify me.

Voluntary and withdrawable. Taking part is voluntary. You can stop at any time during recording. You can withdraw later by contacting [contact] with your recording code: we will delete audio we still hold and leave you out of any result not yet published. Numbers already published as aggregates cannot be recalculated without you, and for option C, copies already made by others cannot be recalled.

Eligibility. I am 18 or older, and I am reading only the provided sentences.

Recording code: ______ Date: ______ Signature or recorded verbal consent: ______

6. Held-out split¶

The split is decided by the prompt id alone:

point = int.from_bytes(sha256(SEED + "\x00" + prompt_id)[:8], "big") / 2**64
split = "heldout" if point < HELDOUT_FRACTION else "dev"

with SEED = "yazses-fa-en-cs-v1" and HELDOUT_FRACTION = 0.5, frozen with the protocol (split_of in the scorer; a known-answer test pins the assignment of the fixture ids).

• Reproducible without coordination. Anyone with the prompt ids gets the same split. Adding, removing or reordering prompts never moves an existing prompt.
• Split by prompt, not by recording. Every recording of a prompt lands on the same side, so a prompt seen during tuning is never scored as held-out.
• Speakers are not split. The same speaker can appear on both sides. A held-out result measures generalisation to unseen prompts, not to unseen speakers, and is reported as such.
• Equal halves. The dev half is for tuning (§1 rule 2) and the held-out half for reporting. Both need enough prompts per category for §3's minimum. With only one collection effort to spend, halves are the simplest choice that serves both.
• Prompt ids are opaque: ^[a-z0-9][a-z0-9-]{0,63}$, naming a prompt and never a speaker, session or person. The scorer refuses any other id.
• Published with every result: the split name, seed, fraction, prompt count and prompt_digest, a SHA-256 over the sorted held-out ids. A reader with the prompt file can confirm the same prompts were scored without anyone sharing a transcript.

7. Normalisation, tokens and script tags¶

1. Normalise reference and hypothesis with fa-eval-v1 (persian-benchmark-and- validation.md §6, persian_metrics.normalize_fa_eval_v1): NFC, Arabic Yeh/Kaf to Persian, whitespace collapsed. ZWNJ, digits and punctuation are kept. Matching is case-sensitive.
2. Split on whitespace. ZWNJ stays inside a word.
3. Tag each token from its letters (Unicode category L*, script from the character name):
4. fa: an ARABIC-script letter and no LATIN letter;
5. en: a LATIN letter and no ARABIC-script letter;
6. mixed: letters from both scripts (PRها);
7. neutral: no letter at all (3.12, ۱۴۰۳, ،).

The tag is a script tag, not language identification. English written in Persian script (گیت‌هاب) is tagged fa, and Persian written in Latin script (Finglish, kheili) is tagged en. That is on purpose: both are what a code-switch decoder gets wrong, and both have to show up as errors. mixed tokens count toward MER and are reported as mixed_ref_tokens, but they make no boundary and take no part in English-preservation or Persian CER. Their count shows how much of a set falls outside those two metrics.

8. Metrics¶

Each metric is a sum of counts over utterances divided by a sum of denominators. They are never averaged per utterance. A metric with a zero denominator is null, never 0. Alignment is minimum-edit over tokens. When two alignments cost the same, the diagonal (match or substitution) wins, then deletion.

Every metric is reported overall and per category, with n and the raw counts, so a reader can recompute any rate and judge its size.

9. Submitting results: aggregate only¶

A contributor scores locally and shares only the aggregate block:

uv run python paper/benchmark/codeswitch_metrics.py rows.jsonl --split heldout --out agg.json

rows.jsonl holds {id, category, reference, hypothesis} per prompt. It stays on the contributor's machine. The output (schema: yazses.bench.codeswitch/1) is the only thing submitted. It contains the protocol and normaliser versions, the split block from §6, and counts and rates overall and per category. It contains no text, prompt ids, audio paths, speaker codes or per-utterance rows, and a test asserts that for the synthetic fixture. Aggregate-only is the CLI's only output mode. Anyone who needs per-utterance scores for their own debugging calls score_utterance locally.

A submission also states, in prose, the model, decoder settings, YazSes commit and machine, as persian-benchmark-and-validation.md §7 asks of every Persian result, plus the corpus source from §4 and how many speakers there were. Results from different sources or protocol versions are reported separately and never pooled.

10. What this commit does not do¶

• It collects no prompts and makes no recordings. The fixture is twelve invented sentences, enough to test the scorer and far short of §3's minimum.
• The consent template is a draft awaiting owner review (§5).
• It runs no model. Decoding a manifest and handing the hypotheses to this scorer is FA-10's measurement, not FA-09's.
• It does not score a decoder's own LangSpan labels (polyglot-switch.md). If the decoder ever exposes them, scoring them is a protocol addition (v2).

11. Version history¶