Release validation checklist and evidence standard¶
CI proves that the code runs on a runner. It cannot hold a key, grant a permission prompt or hear a microphone, and the defects that reached users on macOS and Windows were all of that kind. This page is the shared rule for when a platform or install route may be called verified, and the checklist a tester follows to produce that evidence.
Status: provisional
This is the first version, written from the proposals in #387. Nothing here is a threshold measured on many machines; the staleness rule and the check list will change after they have been used for a few releases. The results live in the validation matrix; the Mac campaign built on this page is macOS validation campaign.
You do not need to run every check on every platform. A report that covers three rows honestly is worth more than one that claims twelve. Say "not tested" for anything you skipped — a blank is read as a gap to fill, not as a pass.
The evidence vocabulary¶
Every cell in the matrix carries exactly one of these words.
| Status | Meaning | Needs |
|---|---|---|
| Verified (HW) | "Verified on hardware": a named human ran the check on a real machine and it worked | A complete evidence record |
| Failed (HW) | A named human ran it on a real machine and it did not work | The same record, plus what happened instead |
| CI only | "Verified by automated test only": a job passed; no person has tried it | A link to the run, and what the job actually does |
| Reported | "Reported by user, not reproduced": someone said it works or fails, but the record is incomplete | The report; the missing fields go in the Gap column |
| Blocked | Could not be tried because an earlier step fails or no suitable device exists | What blocks it and which issue tracks that |
| UNVERIFIED | "Not yet tested": nobody has tried it | Nothing; this is the honest default |
Two rules keep the words honest:
- A virtual machine or a CI runner is never "hardware". A runner has no microphone and no held key, and a VM usually has no real audio path or permission prompt. Those results are CI only, or Reported if a person ran them in a VM and says so.
- "Verified" belongs to the version that was tested. A cell records the YazSes version and the date. A later release does not inherit it; the cell is simply older evidence until someone repeats the check. When a release changes the area a cell covers, that cell is due for a re-test.
Both a last positive and a last negative record are worth keeping for the same check: when did this last work, and when did it last fail? That is more useful to a troubleshooter than one status word, and the status word is derived from those two records. A failure stays visible after the fix ships and is replaced by a positive record only when someone confirms the fix on hardware. (This last point is a provisional choice; see open questions.)
The evidence record¶
For a result to count as Verified (HW) or Failed (HW), the report contains all of the following. A report that lacks one is still welcome; it is graded Reported and the missing item is named as the gap.
| Field | What to write | How to get it |
|---|---|---|
| Who | Your GitHub handle | — |
| Date | The day you ran it | — |
| Device | Model and CPU/chip (for example "Mac mini, Apple M2", "Dell laptop, Intel, Windows 11 Home") | Settings, About |
| OS build | Full version, not "latest" | macOS sw_vers; Windows winver; Linux cat /etc/os-release |
| Session | Linux only: X11 or Wayland, and the desktop | echo $XDG_SESSION_TYPE $XDG_CURRENT_DESKTOP |
| Install route | One of: .dmg, Homebrew cask, pipx, uv tool, .exe installer, Scoop, winget, snap, Flatpak, apt/.deb, install script, source | what you actually ran |
| Fresh or upgrade | Fresh install, or upgraded from which version | — |
| YazSes version | Exact | yazses --version |
| Command output | yazses doctor pasted as text, and yazses verify if you got that far | see Privacy |
| Observed result | What appeared, and in which application | one line per check |
| Not tested | Every check below you skipped or could not run | say so explicitly |
"It works" with none of this is a claim, not evidence. The Windows 11 report in #183 is a good model for the shape: numbered steps, the model, the compute type, the injection backend and the four applications tried.
Privacy rules¶
Nothing leaves your machine unless you attach it, so read what you attach.
- Never attach audio, a transcript of what you said, voiceprints or learning-corpus files. The checks below need none of them. If you want to show that text appeared, name the application and say "a short test sentence appeared"; do not paste a dictated paragraph.
- Remove your username, hostname and home path from
doctoroutput, logs and screenshots.yazses reportwrites a redacted diagnostic file locally and uploads nothing; read it before attaching. SHOWCASE entries are scanned for home paths and emails byscripts/check-compatibility.py. - Crop screenshots to the dialog or settings pane in question. Do not include other windows, notifications or browser tabs.
- A camera-based or microphone-based test must use a short synthetic phrase you are willing to publish ("testing one two three"), not a conversation.
What to run on every release¶
Groups follow the structure proposed by the tester who volunteered for this role in #387. Run the ones that fit your device; each line is do this, attach that.
Install¶
| # | Check | Attach |
|---|---|---|
| INS-1 | Install by one documented route, exactly as its guide says (Linux, macOS, Windows). Every documented route needs its own test; do not assume one covers another | the route, the version, any error shown |
| INS-2 | Negative test for a known limit the guide states (for example, "needs administrator rights", "unsigned: right-click, Open"). Confirm the limit and the message are still as documented | the message, verbatim |
| INS-3 | Uninstall by the route in Uninstalling and confirm nothing is left running | ps / Task Manager result, or "none" |
First use¶
| # | Check | Attach |
|---|---|---|
| USE-1 | First launch: does it start, and does something visible show (tray or menu-bar icon)? | yes/no and where |
| USE-2 | Permissions: were you asked, was the prompt findable, did granting it work? Name each permission separately (macOS: Accessibility, Input Monitoring, Microphone; Windows: microphone privacy switch; Wayland: portal prompt) | one line per permission |
| USE-3 | yazses doctor, then yazses verify | both outputs, redacted |
| USE-4 | Documentation check: did the guide's troubleshooting section match what you saw? | the page and the mismatch |
| USE-5 | First successful dictation: hold the key, say a short test sentence, release | which application the text landed in |
Everyday dictation¶
| # | Check | Attach |
|---|---|---|
| DIC-1 | Hold-to-talk into a plain text editor | application name, result |
| DIC-2 | Into a browser text box and a terminal | per application |
| DIC-3 | Start, restart and stop: yazses start, yazses status, yazses restart, yazses stop. After a restart, does status show the daemon, and does the key still work? (If you changed a setting first, it should now apply.) | the four outputs |
| DIC-4 | Overflow: a small text field and more speech than fits | what happened to the rest |
| DIC-5 | Background noise and rapid speech, one short phrase each | whether text appeared; no transcript |
| DIC-6 | Long session, a few minutes, still running at the end | yazses status at the end |
Meeting mode¶
| # | Check | Attach |
|---|---|---|
| MTG-1 | yazses meeting start, a short call from a source you control, then yazses meeting stop (needs [meeting] enabled = true; see meeting notes). Does a transcript file result? | file exists and its size; not its content |
| MTG-2 | Speaker change markers present when two voices alternate | yes/no |
| MTG-3 | Delete the meeting files by the route the meeting guide describes, and confirm they are gone | what you ran, the result |
Meeting-mode checks involve other people's speech. Use only your own voice or a consenting participant, and never attach the output.
Upgrade and regression¶
| # | Check | Attach |
|---|---|---|
| UPG-1 | Upgrade by the route's standard command from a previous release | from-version, to-version, errors |
| UPG-2 | Remove and reinstall the new version | same |
| UPG-3 | After either, repeat USE-3 and USE-5. A permission that stopped applying after an upgrade counts as a Failed here | doctor output |
| REG-1 | For every bug fixed in this release that was reported on a platform you own, run the reporter's original steps | the bug number and the result |
The "closed bugs" group starts empty. Add a line per closed platform bug so the next release re-runs it; a fix that has only ever passed in CI is not yet a regression check.
How to report¶
Open an issue from the platform's existing test issue (#182 for Apple silicon, #216 for Intel Mac, #183 for Windows 11, #217 for Windows 10, or the Linux distro issue listed in the task finder), or reply on one that already exists. Paste this block and fill it in.
**Tester:** @your-handle **Date:** YYYY-MM-DD
**Device:** model, CPU/chip
**OS build:** full version (Linux: session + desktop)
**Install route:** … **Fresh or upgrade:** …
**YazSes version:** (output of `yazses --version`)
| Check | Result (worked / failed / not tested) | Note (no dictated text) |
|---|---|---|
| INS-1 | | |
| USE-1 | | |
| USE-2 (each permission) | | |
| USE-3 doctor / verify | | output below |
| USE-5 first dictation | | application: |
| DIC-3 start/restart/stop | | |
| UPG-1 upgrade | | from → to |
<details><summary>yazses doctor (redacted)</summary>
…paste…
</details>
**Not tested / could not verify:** …
A maintainer then adds the result to the validation matrix with a link to your comment. You do not have to edit the matrix yourself.
Open questions¶
These were put to the tester in #387 and are unanswered at the time of writing. The provisional answers above stand until they reply.
- Mobile rows. iOS, iPadOS and Android appear in the proposed setup codes. YazSes does not support them (see platform support and the mobile notes), so they are not in the matrix. They can be reserved when a mobile build exists.
- After a fix. A failure record is kept visible until a positive hardware record replaces it, rather than clearing when the fix merges. This costs a longer table and avoids a cell turning green on a fix nobody has run.
- A searchable dashboard was proposed. The matrix is a plain table first; a dashboard can be generated from it once the table has been used for a few releases.