Architecture¶

This page explains how YazSes turns a held key and a few spoken words into text in your focused application — entirely on your own machine. It is written for users and curious contributors who want to understand the moving parts, not a line-by-line internal reference.

For what each capability does and how to turn it on, see the features reference. For every configuration key, see the configuration reference. For the commands mentioned below, see the CLI reference.

How to read this page¶

Two things are marked throughout, and they are never blurred together:

BUILT — wired into the shipping build, with tests. You can run it today.

DESIGNED — decided and documented, and deliberately not in the running system. Some of these are waiting on hardware, some on a model, and one is deferred on purpose so a promise stays kept.

The figures below are generated from the code and the measurements, not drawn by hand — see how these figures are made for the provenance and how to regenerate them yourself. Every figure is followed by the same numbers as a table, so nothing here is only reachable by looking at a picture.

High-level overview¶

YazSes is a small background program — a daemon — that runs on your desktop and listens for a single hold-to-talk key. The model is deliberately simple:

• Hold the hotkey and start speaking.
• Release it when you finish.
• Your words appear in whatever window has focus, usually within about a second and a half on the default model.

Everything happens offline, on-device. Speech is transcribed locally by faster-whisper running on your CPU (int8, no GPU required). There is no cloud service, no API key, and no account. Nothing leaves your machine unless you explicitly point YazSes at a remote host over SSH (see the remote path). Because it is push-to-talk, YazSes is not always-listening: no audio is captured while the key is up.

A thin command-line tool (yazses) talks to the running daemon to start it, stop it, query its status, and tune it. The daemon does the actual work; the CLI is a remote control.

Subsystem map¶

The settings window closes the loop over IPC. Config is read at daemon startup, so a settings app that only writes config leaves the user looking at values that are not in effect. After Apply the window offers a restart, runs the same path as yazses restart, and then polls status over IPC until the daemon answers — success is defined as the daemon replying, not as a command exiting zero, because a daemon that dies while loading a model would otherwise be reported as restarted. Declining keeps a persistent "restart pending" hint. Decisions live in settingsui/restart.py (Qt-free, IPC injected); the dialog is dumb.

A row you cannot reach cannot explain itself. yazses features has had --on/--tier/--category since it was written; the window had none of it, so the only way to a capability was to scroll past every other one. settingsui/search.py is the filter — pure, matching name, slug, category and description text, with on:/off:/tier: tokens taking the same words the CLI flags take rather than inventing a second vocabulary. It changes visibility and nothing else: a hidden row keeps its staged edit, and both Apply and Restore defaults act on the whole model, never on what happens to be on screen.

A switch that cannot explain itself is not a setting. With 152 capabilities, a checkbox and a tier tell the user nothing about what a row does, when to reach for it, or what ticking it writes. settingsui/help.py renders that from the same registry entries yazses features info prints — one source, three renderings: a one-line summary that is always visible (a category has to stay scannable), the full card as a hover tooltip, and the same card behind a ? button. The button is not redundant with the tooltip: hover is unreachable by keyboard, unavailable on touch, and never announced by a screen reader, so each row also carries an accessible description in plain text. The card names the exact [section] key = value writes a toggle performs, which is what keeps the window auditable against a config file the user may also be editing by hand.

Restore defaults stages rather than writes. "Default" has exactly one definition — features.default_state(), the same set system/firstrun.py seeds a new install with — read by both the window's button and yazses features reset, so the two cannot drift. The window computes the difference against what it is showing (staged edits included), names every capability it would touch before anything happens, and leaves it staged for Apply; the CLI writes only the rows that differ rather than rewriting all ~200 keys and churning the file's comments. Neither can turn an experimental capability on, because the defaults are by definition the advised set — a reset can only ever switch one off.

Staged dictation puts the review before the injection. staged/buffer.py is a pure state machine — chunks in, commit/discard/undo out — held by the daemon and consulted in _on_hold_end before anything is typed. Chunks are kept separately rather than concatenated so "scratch that" can remove exactly the last burst, which is what it means. A commit returns the buffer to the ordinary injection path instead of typing it directly, so the no-text-target guard and the injector selection still apply; a failed inject puts the text back rather than losing work the user already reviewed. The spoken commit grammar is opt-in and anchored at both ends, for the same reason commands/revise.py anchors "scratch that": a suffix match would turn "git commit" into a commit, which is precisely the accident the feature exists to prevent.

Decode latency is summarised, not just logged. The daemon has always timed every decode; stt/latency.py now keeps a bounded per-model window of those times and yazses status reports p50/p95 from it. Percentiles rather than a mean because decode time is right-skewed and the slow tail is what you actually wait through, per model because that is what makes "which model should I run" answerable, and with the sample count attached because a p95 over six utterances is not a p95. The window is in memory and bounded — a diagnostic that required the opt-in learning corpus would be missing exactly when it is wanted, and nothing about it touches disk, audio or transcript text.

One way in, on every OS. The window is reachable from the tray's Settings… entry on Linux, macOS and Windows, from yazses settings, and on Windows from a Start-menu shortcut. The three trays draw the label from one tray/menu.py constant rather than each spelling it themselves, so the menus cannot drift into offering a different product per platform. The Start-menu shortcut launches the single PyInstaller binary with --settings, which __main__.py dispatches alongside --daemon/--tray/--cli — a mode flag the bundle does not know falls through to the CLI and exits 2, and the windowed binary has no console to say so on, which is why the flag and its branch are asserted against each other in tests rather than trusted to stay in step across two files.

Offline Command Mode rewrites the selection locally. Select text, hold the command key, say "make this shorter" — commands/rewrite.py parses the request (whole-utterance grammar, so the same words dictated as prose are unaffected), rewrite/engine.py reads the selection through the clipboard's new PRIMARY-first read path, and the existing local GGUF from [filters.disfluency] produces the replacement. Voice-editing of selected text is the most-praised capability in the paid dictation tools and is cloud-only in every product that has it; this one never leaves the machine.

The guards are the feature, because a rewrite replaces writing the user already has. Getting it wrong is not a worse suggestion, it is lost text. So each instruction carries its own plausibility band — "make this shorter" returning something longer has not followed the instruction, and "fix the grammar" that halves the text has deleted content — and a model that answers with a preamble ("Sure, here is…") is refused as having treated the prompt as chat. Every rejection leaves the selection untouched, and the original is put on the clipboard before the model runs, so the undo path exists before the risky step rather than after it. Off by default under [commands] rewrite.

A squeeze can now carry which mode, not just when. platform/emg/pressure.py turns raw EMG into a level — relaxed / light / hard — using the Teager–Kaiser energy operator rather than a bare amplitude threshold, because TKEO responds to amplitude and frequency and therefore ignores the large, slow artefacts (baseline wander, cable sway, a shifting electrode) that a rectified threshold reads as a contraction. Light squeeze dictates, hard squeeze commands, mirroring the command key so all three mode switches behave identically. Thresholds are calibrated from the user's own relaxed baseline and maximum contraction, because muscle amplitude varies by an order of magnitude between people and fixed microvolt numbers are meaningless. platform/emg/brainflow_source.py adds buyable hardware (OpenBCI, Muse, …) through BrainFlow, keeping the serial YESP path as the DIY reference; a missing device or dependency is a logged no-op, never a crash. (#103)

Gaze calibration refines itself. gaze/implicit.py treats a mouse click as ground truth for where the user was looking and folds it into the existing affine map with recursive least squares — the same estimator fit_calibration uses, updated one sample at a time, so cost per click is constant and no sample history is kept. It is gated on eye-agreement confidence and a residual bound, carries a forgetting factor so it tracks a moved laptop lid rather than averaging both positions, and refined_if_better only replaces the wizard's map when the candidate wins on held-out samples (ADR-014's rule). Capture stays opt-in and on-device per ADR-011/012; the module itself opens no camera and listens for no clicks, which is why it is testable without a desktop.

Three STT engines now sit behind one seam. faster-whisper (default), parakeet (accuracy), and moonshine (#74 — built for short segments on CPU, which is the shape of hold-to-talk, and needs onnxruntime + tokenizers plus librosa, numba and huggingface-hub, but no torch). stt/factory.py selects on [stt] engine and falls back to faster-whisper with a warning whenever an engine's optional dependency is absent, its model fails to load, or the engine name is unknown, so a bad engine choice never stops dictation. The one exception is language: a [stt] language the resolved engine and model cannot transcribe is refused before any model loads (LanguageUnsupportedError), because every fallback ends on an English-only checkpoint and would type English for non-English speech. The daemon then holds in the ERROR state, with IPC alive, and says what to change.

That fallback is deliberately quiet at the decode layer (a log line, never a crash), which on a real machine meant it was quiet everywhere: [stt] engine = "parakeet" with onnx_asr missing degraded silently to a much smaller, less accurate model with nothing surfaced anywhere a user would see it. yazses doctor's _configured_engine_check closes that — it reports WARN with the exact features enable fix whenever the configured non-default engine's dependency cannot actually be found, distinct from _stt_engine_check above it (which only asks whether faster-whisper itself, the universal fallback, can load — OK in both the healthy and the silently-downgraded case). Doctor reads the environment, so it cannot see an engine whose package is installed but whose model fails at load; the running daemon can. Since #644 the built engine says what it is (engine_name, model_name), the factory stamps a fallback with its cause, and yazses status prints the engine and checkpoint that actually decode, with a FALLBACK: line naming the engine asked for and why. The dependency loss that triggered this was system/deps.py::install_packages installing a feature's extra via a bare uv pip install, which never touches a uv tool install's own receipt — so uv tool upgrade (what the tray's "Install update" and yazses update both run for that install method) reconciled the venv back to a receipt that never listed it, and removed it. install_packages now detects a uv tool-managed environment and records the extra in its receipt instead, merged with whatever was already there, so a second feature enabled later cannot silently drop the first.

That protects installs made after the fix, not the ones made before it. So the upgrade side closes the rest: when an enabled feature's libraries are installed but missing from the receipt, yazses update and the tray's "Install update" issue uv tool install --upgrade --with <recorded> --with <missing> <target> instead of uv tool upgrade — the same version move, with the requirements recorded so the next plain upgrade keeps them too (system/deps.py::uv_tool_upgrade_command). Only working features are carried; a dormant one is reported, not silently downloaded during an update. The receipt round-trip also keeps environment markers, which an earlier version dropped — turning the Intel-Mac-only onnxruntime<1.24 into a ceiling for every platform the next time a feature was enabled.

Neither Parakeet nor Moonshine supports initial_prompt, so the personal dictionary reaches them a different way: postprocess/vocab_correct.py (#73) recovers mis-heard vocabulary after decoding, which is why it was built engine-agnostic rather than as a Whisper prompt trick.

Moonshine carries a hard upstream constraint the adapter absorbs rather than propagates: audio must be 2-D and between 0.1 s and 64 s, enforced with bare asserts. Both bounds are reachable — a stray key tap is under 0.1 s, a dictated paragraph is over 64 s — so short buffers return empty and long ones are split on the silence gate, instead of an AssertionError surfacing as a crash.

Noise suppression has a backend that can be installed. The denoise seam (ADR-v2-015) shipped with only a deepfilternet adapter, which no environment can satisfy: its latest release pins numpy<2.0 while this project needs numpy>=2.4.6, and those ranges are disjoint on every Python version — so the feature was designed, wired and permanently unusable. denoise/spectral.py (spectral gating over noisereduce) is the installable backend and the new default. It is weaker than DeepFilterNet and the docs say so: it removes steady broadband noise — fans, air conditioning, road hum — and does little against a second speaker, which is the Cocktail Filter's job. Selecting deepfilternet still parses and degrades to a passthrough, with no remedy offered, because advising an extra that can never install is worse than saying "unavailable".

The control plane never touches the pipeline — the CLI, the settings window and the tray all speak to the daemon over the same JSON-RPC channel, which is why yazses status reports the truth rather than a guess, and why the tray can be killed and relaunched without disturbing dictation.

Both front ends also share one definition of enabling a capability. Twenty-four of the 152 rows need an optional Python package (gaze needs mediapipe, cocktail needs speechbrain, and so on), and yazses features enable has always installed those before telling you to restart. The settings window now does the same, on a worker thread, through the identical system/deps.py call — so the two cannot drift on what "enabled" means. The decisions (what to install, what to say when it fails) are pure and Qt-free in settingsui/deps.py; only the thread and the widgets live in settingsui/app.py.

When an install fails, the config key stands. A failed install is usually transient — a network blip, a slow mirror, a wheel building — and silently un-toggling the switch a user just moved discards their intent and leaves no trace of why. Instead the capability is recorded as on and reported as dormant until its packages arrive, which is a state yazses doctor and yazses features already model and a user can fix by retrying. Enabling records intent; satisfying it is a separate, retryable step, and the gap between the two is always visible.

"Always visible" is enforced, not assumed. system/dormant.py derives, from the feature registry, every enabled capability whose libraries cannot be imported — however it got that way: a failed install, --no-install, a hand-edited config.toml, or an upgrade that rebuilt the environment without the extra. A camera feature also counts when its libraries import but OpenCV is half-deleted (system/opencv_health.py: cv2 left as a folder without its native module after one of two OpenCV builds was removed). That finding gets the exact repair command and no install button, since installing would change nothing. The daemon logs each at start and shows one rate-limited notification with the fix; yazses doctor prints a row per feature; yazses features draws it as ◐ ON with the command beside it. And a failed features enable, features reset or handsfree enable exits non-zero and says dormant, rather than printing "Apply it: yazses restart" over the installer's error.

Two static guards keep a new feature from making the base install heavy or broken: tests/test_no_module_imports_an_extra_at_module_scope.py scans every module (not only the daemon's import graph) for a module-scope import of any extra's packages, derived from scripts/check_dependency_budget.py::EXTRA_MODULES, transitively; and tests/test_every_extra_has_an_install_path.py fails when an extra exists that no feature installs and no stated exemption explains.

A feature's libraries arrive when it is needed, by one action, and never half. Every surface that finds a dormant feature now offers its install instead of a sentence: the daemon's start-up notification carries an Install now button, yazses start/restart in a terminal ask once (default no), and yazses features install (or features repair, which lists every one first and ends with a per-feature summary) does all of them (system/ondemand.py). Consent is unchanged — nothing downloads without that click, yes or command — and an environment that can never take the libraries (a snap, a frozen bundle, the distribution's own Python) gets the reason and no button. The install itself (system/deps.py::install_packages, shared by the CLI, the settings window and the toast) is a transaction (system/install_txn.py):

• it goes to the channel that records it — the uv tool receipt, pipx inject (a package pip-installed into a pipx venv is dropped by pipx reinstall), or the venv; a source checkout is told that uv sync will remove it and how to keep it;
• YazSes's own requirements are passed as --constraint (a dry run first, for pipx, which would otherwise store the temporary path and replay it on reinstall), so a feature that would downgrade numpy or ctranslate2 is refused before anything moves;
• the installed set is snapshotted; a failed, interrupted, or base-breaking install is undone, and the report says whether the undo was complete;
• features enable and the toast then check that the feature's modules import.

The registry → extra → installer contract is closed in both directions by tests: tests/test_every_import_is_declared.py fails when any module imports a third-party package that neither the base install nor an extra declares, and tests/test_feature_imports_are_installable.py fails when a feature's own package imports an extra's module that its _FEATURE_DEPS entry would not install.

The four blocks in the bottom band are the ones people most often assume are already there. Each is a recorded decision rather than an omission:

The same subsystem map ships in three other formats under docs/diagrams/: Mermaid (renders on GitHub), ASCII for terminals and code review, and a self-contained HTML page.

The dictation pipeline¶

Each time you hold the key, speak, and release, your audio flows through a fixed sequence of stages. The figure below colours each stage by what it actually costs, measured rather than assumed.

Stage by stage:

1. Hotkey. A hold-to-talk key press starts a recording; releasing it ends the burst. The trigger is usually a keyboard key, but it can also be an EMG muscle-sensor (experimental, off by default) connected over USB serial — a hands-free / silent-input option (src/yazses/platform/emg/).
2. Audio capture. The microphone is recorded into an in-memory buffer for the duration of the hold (src/yazses/audio/recorder.py).
3. VAD gate. A calibrated voice-activity check discards audio that is mostly silence, so an accidental tap doesn't produce a spurious transcript (src/yazses/audio/vad_calibrated.py). If you see "Silent audio -- discarding" in the logs, your speech fell below the threshold — run yazses mic-level --set.
4. Pre-speech padding. A short silent lead-in is prepended before decoding so faster-whisper doesn't clip your first word (src/yazses/audio/padding.py).
5. STT decode. The buffered audio is transcribed on the CPU by a pluggable engine behind the SttEngine protocol (src/yazses/stt/base.py, selected by [stt] engine): faster-whisper by default, or NVIDIA Parakeet TDT (yazses features enable stt-parakeet) — 2.06 % word error rate against whisper-large-v3's 3.23 % on 200 LibriSpeech utterances, at roughly twice whisper-small's CPU speed, and an eighth of large-v3's insertions (benchmarks). On the Whisper path an initial_prompt biases the model toward the right words — your personal vocabulary, and for English and auto-detect the app name (the English app-name phrase is withheld when [stt] language is another language). (An editor-context prefix was designed and is not wired: the daemon never constructs LspContextProvider, so nothing from your editor reaches the prompt.)
6. Text cleanup. clean_text() strips Whisper artefacts such as [BLANK_AUDIO] and stray leading punctuation (src/yazses/postprocess/cleaner.py).
7. Disfluency filter. A three-pass filter removes fillers, de-duplicates repeated words, and rolls back self-corrections (src/yazses/stt/filters/disfluency.py).
8. Command classification. A fast Tier 1 regex grammar decides whether the utterance is plain dictation or a command like "undo that" or "go to line 42" (src/yazses/commands/grammar.py). A Tier 2 small-language-model router (slm_router.py) handles the unsure case: when Tier 1 returns plain dictation and [commands] slm_model_path names a local GGUF model, the router gets a second look and its answer is accepted above slm_confidence_threshold. Off unless that model path is set — Tier 1 alone decides every utterance otherwise, which is what every install without a model does. Few-shot examples mined by yazses tune are loaded from few_shots.toml if present. A router that fails or raises falls back to Tier 1 rather than losing the burst.
9. Dispatch and inject. Plain dictation goes to text injection; commands go to a key sequence (src/yazses/commands/dispatch.py). For dictation only, two optional finishing steps run first: an offline LLM cleanup pass that lightly reformats the text (src/yazses/postprocess/llm_cleanup.py, off by default), and continuation spacing so back-to-back bursts don't glue together (src/yazses/postprocess/spacing.py).

The measured cost of each stage that is separately timed:

The design consequence is worth stating plainly: all latency is the speech model. Micro-optimising the filters, the grammar, or the injector would not be measurable. What is measurable is which model you choose — so that is the knob the docs push you toward, and the only one worth turning.

Most of these stages are configurable or can be turned off; see the configuration reference.

What a burst costs¶

base.en is the default because it is the compromise that survives contact with real use: noticeably more accurate than tiny.en, and still back inside a second and a half. small.en is the most accurate and the right choice for yazses transcribe and Meeting Mode — but a 5-second median breaks the flow of live dictation, which is a different job.

These are read audiobook clips in clean conditions, so your dictation WER will be worse; treat them as a comparison between models, not a promise about your desk. Full method, hardware and the recipe each number was measured with are on the benchmarks page.

Your machine, not this one¶

Every number above is one laptop. yazses status reports p50 and p95 decode latency per model from your own dictation — percentiles rather than a mean, because decode time is right-skewed and the slow tail is what you actually wait through, and with the sample count attached because a p95 over six utterances is not a p95. stt/latency.py keeps a bounded in-memory window per model; nothing is written to disk and no audio or text is involved, so it does not depend on the opt-in learning corpus and is available exactly when it is wanted.

Wall-clock is not the only cost. A decode taking a second of your time spends around five seconds of CPU, because the work spreads across cores — the right trade on mains power and the wrong one on battery. [stt] cpu_threads caps it (0, the default, leaves the library alone). Measured on this machine, capping at 4 changed nothing; the trade only begins below that. See CPU and battery.

The daemon and its states¶

The orchestrator is a single long-lived process, yazses-daemon (src/yazses/core/daemon.py). It wires the whole pipeline together, owns the hotkey listener, and runs an IPC server for the CLI and tray.

Internally it is a state machine. The ordinary dictation loop is the spine down the middle; everything else is a mode you enter deliberately and return from.

• LOADING — the daemon is up but the speech model is still loading. With the opt-in [stt] idle_unload_s (off at 0), a model released while idle is reloaded on the next hold, and the daemon shows LOADING between recording and decoding while that reload finishes.
• IDLE — ready and waiting for the hotkey.
• RECORDING — the key is held; audio is being captured.
• TRANSCRIBING — the key was released; the engine is decoding.
• INJECTING — the result is being typed into the focused app.
• MEETING — hands-free whole-meeting capture, driven over IPC rather than by the hotkey; stopping it runs the batch diarization pass through TRANSCRIBING.
• REMOTE_SETUP / REMOTE_ACTIVE — establishing or using the SSH forwarding path.
• ENROLLING — the accessibility / mic calibration wizard is running.
• READBACK — speaking a transcript back via offline TTS.

Two more states exist in the type and are worth being precise about, because a tidier diagram would misrepresent them. ERROR has two sources. The daemon enters it only when start-up fails in a way the user can fix — the model cannot be downloaded or loaded, the CPU refuses the compute type, or the configured language is one the engine cannot transcribe. It then stays up with no model loaded and no hotkey hooked, so the tray, yazses status and the log can all say why. The tray also shows ERROR on its own when the daemon stops answering IPC, which is why the icon can go red while the daemon itself has simply died. PAUSED is defined, and Meeting Mode and enrollment both accept it as a legal starting point, but nothing in the daemon currently sets it.

CLI ↔ daemon IPC¶

The CLI and system tray never touch the pipeline directly — they talk to the daemon over a small JSON-RPC 2.0 channel (newline-delimited JSON). On Linux and macOS this is a Unix domain socket; on Windows it is a named pipe (src/yazses/ipc/). This is how yazses status and yazses mark-wrong reach a running daemon, and how the daemon reports live state (model, PID, ready/loading, mic level) back to callers.

yazses stop is the exception: it does not go through IPC. It calls the platform LifecycleBackend.stop_daemon, which on Linux and macOS is a plain SIGTERM to the daemon's PID. Only the Windows backend asks the daemon over IPC first (the shutdown method), waits for the process to exit, and terminates it if it did not acknowledge or is still alive. The tray's Quit and the in-app update hand-off also send shutdown over IPC, and so does a newly started daemon that finds an older version still running after an upgrade.

Platform abstraction¶

YazSes runs on Linux, macOS, and Windows from one codebase. It does this by defining a set of Protocol interfaces — abstract contracts — that each OS implements in its own way. The contracts live in src/yazses/platform/base.py; the implementations live in src/yazses/platform/linux/, .../macos/, and .../windows/. A factory (platform/factory.py) inspects the running OS and returns a Platform bundle of the right concrete backends, so the daemon and CLI are written once against the interfaces and never branch on the operating system.

These are the core seven. platform/base.py defines seven more. Four are built on demand by functions in platform/factory.py, with one implementation per OS family: HostBackend, FileOpener, UrlOpener (the "send to" hand-off) and PackRuntime (the add-on pack loader). Three are optional fields of the platform bundle, left None where an OS has no implementation yet: FocusedAppProvider (macOS and Windows only), UpdateLauncher (Windows only) and AccessibilityBackend (Linux only).

The EMG hotkey backend (platform/emg/backend.py) is a platform-independent HotkeyBackend implementation — it reads squeeze events from a USB serial device (YESP protocol). Since v2.14.0 the daemon constructs it through a pluggable activation-source seam (core/daemon.py::_build_activation_sources) whenever [emg] device_port is set: a squeeze drives the command-key callbacks by default (mode = "command"), or plain hold-to-talk dictation (mode = "full_text"). Each activation source runs in its own background thread beside the keyboard hook and is stopped at shutdown; the same seam is where future non-keyboard triggers (wake word, switch access) plug in.

The webcam face-gesture switch is the third source through that seam (facegesture/, v2.37). MediaPipe's FaceLandmarker — the model Glance-Type already downloads — reports 52 blendshape activations per frame, and one of them becomes the key you hold: the mic opens while your jaw is open (or your brows are raised) and closes when it relaxes. facegesture/detector.py is the whole activation policy and is pure, so it is tested frame by frame with no camera; facegesture/backend.py is the capture loop and duck-types HotkeyBackend like the two EMG transports, taking the same two callbacks and the same mode decision.

Two parameters carry it. Hysteresis: a blendshape score is continuous and noisy, so a gesture held at a single threshold crosses it several times a second and each crossing would be a separate recording — the mic opens at hold_threshold and closes only below release_threshold. A frame-count debounce: talking, laughing and yawning all spike jawOpen briefly, so min_hold_frames is what separates a held gesture from the Midas-touch problem, and its latency is affordable here because the audio path already prepends [accessibility] pre_speech_padding_ms of buffered audio — the words spoken during the debounce are still in the recording. A frame with no reading (no face, a dropped frame) counts toward release, never toward hold: the alternative to releasing when the user leaves the camera is a microphone that stays open until they come back.

It matters most for the users who cannot use the other two: it needs no hardware beyond the webcam already in the lid, where EMG needs an armband and the keyboard hook needs a key you can press. eViacam has been unmaintained since ~2019 and Google's Project Gameface was archived in 2025, so this class of Linux switch access currently has no maintained free implementation (#102).

Bluetooth is the second transport through the same seam. platform/emg/ble_backend.py (BLEEMGBackend) speaks the same YESP protocol over the Nordic UART Service and duck-types the same HotkeyBackend with the same two callbacks, so [emg] ble_address builds a source exactly as device_port does, and [emg] mode routes both identically — deliberately one decision shared rather than one per transport, since a squeeze that dictates over Bluetooth and runs a command over USB is the kind of drift nobody would think to test. Setting both is allowed and logs a warning: they are two physical devices, and choosing one for the user would be a guess.

⚠ Until v2.36 ble_address was read by nothing. It was a documented config key that _build_activation_sources never looked at, and yazses doctor printed it as a flat OK — so a paired armband was configured, reported healthy, and never connected. There is no symptom to notice, because a hotkey that fails is silent and so is one you are not pressing. doctor now reports WARN when bleak is absent, since an address without the transport library cannot connect.

Reaching the app grid¶

A .deb or snap can write into /usr/share/applications; a pipx or uv tool install owns nothing outside its own virtualenv, which left those users with a Settings window reachable only by typing a command. system/launcher.py is the per-user equivalent — it writes the .desktop entry and the icon set into $XDG_DATA_HOME, which is what XDG intends and needs no privileged step (yazses settings --install-launcher). The Exec= line is the bare command rather than an absolute path: hard-coding today's interpreter is how a launcher breaks at the next upgrade.

Adding a new operating system¶

Because every OS-specific concern is behind a Protocol, porting YazSes to a new platform is well-scoped: implement each interface in src/yazses/platform/<os>/, then register the new sys.platform value in platform/factory.py. No changes to the daemon, pipeline, or CLI are required.

Text injection backends¶

Getting characters into the focused window is surprisingly OS- and display-server-specific, so on Linux YazSes ships several injector backends and picks one automatically (src/yazses/inject/):

• ydotool — types via the kernel uinput device. This works everywhere, including on Wayland and inside terminals, which is why it is the default where available.
• xdotool — the classic X11 typing tool; fine on plain X11 sessions.
• wtype — a Wayland typing tool, but it is blocked on GNOME and KDE Wayland, so it can't be relied on there.
• clipboard — copies the text and pastes it. A pragmatic fallback, but a no-op inside terminals (paste means something different there) and it clobbers your clipboard.
• portal (inject/portal.py) — types through org.freedesktop.portal.RemoteDesktop, the one Wayland route that survives strict snap confinement: it needs no /dev/uinput device node and no snap connect, since access rides on the desktop plug the snap already declares. Selected on the snap build's Wayland path, after ydotool and before wtype; types by keysym so it stays layout-independent.
• unicode (inject/unicode.py, opt-in) — resolves each character through libxkbcommon's XKB state and emits key events via a private /dev/uinput keyboard, to type the non-ASCII characters (accented letters, most non-Latin scripts) the ydotool path silently drops on Wayland. Ships non-default pending real hardware validation.

At startup, auto.py probes the environment and selects the best available backend; you can override the choice with the [injection] backend config key (auto | type | clipboard | wtype | portal | unicode).

Why Wayland needs ydotool. Wayland deliberately isolates applications from one another for security, so an app generally can't synthesise input into a different window. wtype relies on a protocol that GNOME and KDE do not expose, so it fails silently there. ydotool sidesteps this by injecting at the kernel level through uinput (via its helper daemon ydotoold), which is why it is the one reliable path for keystroke injection on modern Wayland desktops. yazses setup provisions ydotoold for you; see the Linux install guide.

Why the keycodes are a committed table. ydotool key accepts only numeric <keycode>:<state> tokens and silently ignores symbolic names — ydotool key ctrl+v emits no events at all — so a combo like ctrl+v has to be turned into 29:1 47:1 47:0 29:0 somewhere. That resolution used to happen by importing evdev inside ydotool_key_args. evdev is a C extension compiled against <linux/input.h>, Linux-only, and deliberately not installed on the BSDs — which the BSD backend reaches, since it composes the Linux injectors. The result was that every spoken command, every backspace correction and the clipboard paste raised ModuleNotFoundError there. src/yazses/inject/keycodes.py now holds the whole keyboard range (codes 1–255) as a table; codes above 255 (media, brightness, vendor keys) still resolve through evdev where it exists, so nothing that worked on Linux stopped working. Freezing them is safe because input-event-codes.h is a kernel ABI whose numbers cannot change without breaking every existing binary, and tests/test_ydotool_keycodes.py re-derives the table from evdev on any machine that has it, so it cannot drift unnoticed.

The remote path¶

YazSes can dictate into an application running on a different machine over SSH (src/yazses/remote/, yazses remote <host>). The speech is still captured and transcribed locally — only the final text is sent onward, so your audio never leaves your machine.

At a high level: the local daemon opens an SSH reverse tunnel (remote/forwarder.py), a lightweight injection agent runs on the remote host (yazses-agent, remote/agent.py), and the daemon's normal injector is swapped for a proxy (remote/local_proxy.py) that forwards typed text over the tunnel to that agent. From your point of view it feels identical to local dictation; the last mile just happens over the network.

What is built, and what is designed¶

YazSes carries a large catalogue of capabilities, and it is deliberately honest about which of them are real. Every capability is a row in one registry (src/yazses/system/features.py), and each row knows whether it is actually wired into the build. yazses features prints that distinction, and features enable refuses a capability that isn't wired rather than writing a config key nothing will read.

It also prints what a capability costs. system/depsize.py reads a committed table (system/feature_sizes.json, regenerated by scripts/gen-feature-sizes.py) and reports the download size before anything is fetched — the marginal size, so a feature whose packages you already have reports as free rather than as its full weight. The table holds fully resolved dependency closures, not wheel sizes, because the two differ by orders of magnitude where it matters: speechbrain is a few MB of its own and about 3.4 GB resolved, since it pulls PyTorch and the NVIDIA CUDA stack that a CPU-only tool never uses. Model files are added on top of the pip figure, because the generator resolves packages and a model is not one: the catalogue priced read-back at 12 MB while tts/download.py said in its own docstring that the voice it fetches is 340 MB, and the loud-download warning could never fire on it. _MODEL_FETCHERS holds the two together and tests/test_feature_size_counts_model_files.py derives both sides from the tree, so a new download.py cannot ship unpriced. Sizes come from a committed file rather than a live query so the catalogue renders offline and instantly (ADR-011), and an unknown size prints nothing rather than a guess. See ADR-018, which also records why third-party plug-ins are declined: a plug-in would sit on the dictation hot path with the microphone, the transcript and the injector.

Two readings of that chart matter.

The core is dense and the frontier is thin. The categories a daily user touches — core dictation, formatting, editing — are almost entirely wired. The category furthest from done is multilingual (5 of 7 designed: polyglot, langroute, compose, interpret, pronunciation). Accessibility and input modalities is now mostly wired (20 of 25); the five still designed there are loadguard, spatialvad, lipread, sign and the [tts] clone_voice option. Each is specified so that when it is wired the work is integration, not invention.

A designed row is a contribution-shaped hole. Several of these have a tested, dependency-free pure core and no caller — the design is done and the wiring is not. Five such capabilities were wired by outside contributors in a single week, which is why this number moves. If you want to help, this chart is the map: see find a task that fits you.

The optional learning loop¶

YazSes can improve its accuracy for your voice and vocabulary over time — but only if you opt in. This is off by default.

When enabled ([learning] enabled = true), the daemon records one event per hold-release into an encrypted, machine-bound local corpus (src/yazses/learning/). The corpus is a local SQLite database at ~/.local/share/yazses/; the transcript text and any stored audio are encrypted with a key derived from the machine, while only coarse metadata is kept in the clear. Capture happens on a background thread so it never slows down dictation, and nothing is ever uploaded.

Later, yazses tune analyses that corpus offline and proposes concrete improvements — new vocabulary entries, a better VAD threshold, a different model, disfluency tweaks — which you review and approve before anything is written to your config. Each proposal is checked against held-out data it wasn't derived from, so a change has to survive evidence it hasn't already seen. You can inspect or delete the corpus at any time with yazses corpus status, and flag a bad transcription with yazses mark-wrong to feed the loop a correction signal.

The v2 cognitive layer¶

Beyond core dictation, YazSes includes an experimental v2 cognitive layer: a set of opt-in perceptual and personalization features such as a target-speaker "cocktail" filter, look-to-pane gaze targeting, corpus-driven vocabulary biasing, and code-switch language routing. Every one of them is OFF by default, and their heavy machine-learning dependencies (speaker embeddings, gaze models, and so on) are isolated behind optional install extras — the base install never pulls them in, and the pure logic stays dependency-free and dormant until you explicitly enable a feature (manage them with yazses features enable/disable). See the v2 features preview.

Staying up: the reliability layer¶

A dictation daemon is only useful if it is running and correct at the moment you reach for the key, and the failures that matter most are the ones with no visible symptom. Five mechanisms exist for that, all on by default.

Config can never stop startup. configcheck.py reads the config dataclasses' own annotations and repairs what it can ("0.004" → 0.004), falls back to documented defaults otherwise, drops unknown keys and sections, and survives unparseable TOML. Nothing is swallowed: every decision becomes a ConfigProblem, listed at startup and shown by yazses doctor as a Config validity check. Previously a single quoted number loaded cleanly and then failed every dictation burst inside numpy, naming neither the file nor the key.

One authority on "is it running". A PID file survives kill -9 and is not recreated if deleted under a live daemon; the single-instance lock is held by the OS for the process lifetime and so is exact in both directions. pid.is_running() consults the lock first, so yazses status and yazses start cannot give opposite answers — a state this project reached in practice.

It comes back. yazses autostart enable installs a systemd user service pointing at the current install (platform/linux/autostart.py is its single source of truth) with Restart=on-failure and a StartLimit crash-loop bound. A clean exit — what yazses stop produces — is deliberately not restarted.

The gate follows your voice. audio/adaptive_vad.py watches outcomes: a run of discards with no successful transcription between them means the silence threshold sits above the speaker, so it is lowered to pass those bursts, persisted, and announced. Only downward, because a gate that is too high fails invisibly while one that is too low only adds noise the user can see.

The icon stays. The tray was once launched at startup and never checked, so a crash left dictation running unobserved. tray/supervisor.py re-checks every 20 s using the tray lock as the liveness signal, bounded at five relaunches.

And so does the daemon — supervision runs both ways. systemd and launchd restart a crashed daemon; Windows autostart is an HKCU\Run value that fires once at login, so there a crash meant no dictation until the next sign-in while the tray watched and did nothing. The tray's poller now restarts it, under the same shape as the supervisor above: bounded attempts, spaced by a cooldown, then stop and say how to see the real error. The one distinction worth the code is dead versus wedged — a daemon that is alive but not answering still holds the single-instance lock, so replacing it is impossible and red is the honest answer. The other is crashed versus stopped: yazses stop leaves a note first (system/stop_request.py), and a tray that finds the daemon gone and a note newer than itself closes instead of restarting it, so a stop stays a stop.

The user-facing inventory of every one of these — what triggers it, what bounds it, and the failures deliberately left alone — is Reliability: what recovers by itself. Keep the two in step: that page states the actual constants (MAX_DAEMON_RELAUNCHES, RELAUNCH_COOLDOWN_S, silent_streak_threshold), so changing one here without changing it there publishes a number that is no longer true.

A self-healing action nobody sees is only half of one. system/notify.py speaks notify-send, which exists on Linux and nowhere else, so on Windows and macOS the mic auto-heal and the VAD retune repaired capture and told the user nothing. The daemon now queues those toasts onto its status reply — the channel the tray already polls — and the tray shows them with the OS's own notifier. Deliberately not a new IPC direction: the daemon never calls the tray, so a tray that has quit costs at most a bounded queue.

And it says the same thing on every OS. The badge's colour and tooltip come from one pure function, tray/menu.py::icon_spec, fed by one TrayModel→status bridge, status_from_model. That is not decoration: the Windows tray once kept a private colour table of its own, so the same daemon state was a different colour on Windows than on Linux, command mode and "no text field focused" had no colour at all, and five of the twelve tray states — Meeting Mode among them — fell through to idle blue. A second implementation of a policy is a second policy.

One renderer draws the mark everywhere. brandmark.py redraws contrib/icons/yazses.svg in Pillow, and the live tray badge, the packaged .ico/.icns (scripts/gen-icons.py) and the Microsoft Store listing artwork (scripts/gen-store-art.py) all come out of it, so the shortcut icon, the tray glyph and the store page cannot diverge. Each committed image is bound back to its generator by a test, comparing decoded pixels within a measured tolerance (scripts/imagediff.py) rather than exactly — the renderer supersamples in floating point, so an exact comparison answers "was this file produced on this architecture", which is not the question the guard is asking. The build now fails when the icon file is absent. It used to fall back to icon=None, which is why every Windows release shipped PyInstaller's default artwork on the desktop shortcut: the asset had never existed, and a silent fallback meant no build, test, or CI run ever mentioned it.

Two commands expose the result: yazses verify runs the real chain and names the first broken link, and yazses report writes a redacted diagnostic bundle locally — never uploaded, per the privacy posture below.

Where this is going¶

The architecture is built so the next era is an extension rather than a rewrite. Three seams carry most of the future work, and each already exists in the shipping code:

• The activation seam. _build_activation_sources treats "what starts a burst" as a plug-in. A keyboard key and an EMG squeeze are already two implementations of the same contract; a wake word, a switch, or a brain-computer interface would be a third without touching the pipeline.
• The engine seam. SttEngine made Parakeet and Moonshine config changes rather than forks. Anything that turns audio into words plugs in here.
• The platform seam. Every OS-specific concern is a Protocol, which is what makes the Android programme a port rather than a rewrite.

What is deliberately not being built is as much a part of the design as what is. Cloud escalation is fully specified and stays unimplemented so the offline promise cannot erode by increments; personal speech adapters wait on a measured win against held-out data rather than shipping on the strength of the idea. See the roadmap for the full account, including the speculative directions — semantic recall over your own corpus is the one the project exists for.

Privacy posture¶

YazSes is offline by design. Push-to-talk means it only records while you hold the key; transcription runs locally on your CPU; and no audio, text, or telemetry is sent anywhere by default. The only outbound path is one you turn on yourself — the SSH remote injection above, and even then only the final text is forwarded, not your audio. The learning corpus is opt-in, encrypted, and local.

For the full commitment and details, see the privacy statement.

How these figures are made¶

The five figures on this page are generated, not drawn:

uv run python scripts/gen-arch-figures.py

The generator reads the live capability registry (yazses.system.features — the same one yazses features prints), the generated command index, and the measured timings parsed out of the benchmarks page. It fails rather than guessing: if a benchmark table is reworded it stops, and if a label would overflow the shape holding it the figure is refused. So a figure claiming a capability is wired is making the same claim the CLI makes, and a figure quoting a latency is quoting a number recorded on the benchmarks page.

The figures are plain SVG with no script and no external reference — nothing is fetched from a third party when you load this page, which is the same commitment the privacy statement makes about the software itself. They animate from the site's stylesheet and stop animating entirely if your system asks for reduced motion; every value they show is also in a table on this page.