YazSes Architecture¶

Last updated: 2026-05-18 (v1.0.0-dev.5)

v1.0 Rust-core Architecture (archived — archive/rust-hci-v1 branch)¶

This Rust architecture is not on main anymore. It was moved to the archive/rust-hci-v1 branch and paused in early stages. main is the Python app (Part 1; see docs/cli-reference.md and the README). Kept here for reference.

Workspace crates¶

v1.0 Pipeline¶

Hold-key / EMG squeeze (yazses-inputs / InputBackend)
  → HoldStart event → DaemonState: Idle → Recording
  → AudioCapture (cpal) + VadGate (RMS; or SileroVad with --features silero)
  → PaddingBuffer prepends pre-speech ring buffer
  → HoldEnd event → DaemonState: Recording → Transcribing
  → EditorBridge.get_context() [async, parallel with audio]
      └─ 5-tier WindowDetector (Hyprland → Sway → WLR-toplevel → X11 EWMH → Null)
      └─ NeovimBridge (nvim-rs, $NVIM socket) / VSCodeBridge (TCP push, port 57843)
      └─ EditorContext.to_initial_prompt(224 BPE tokens) → ASR initial_prompt
      └─ EditorContext.to_llm_block() → LLM <editor_context> block
  → STTRouter (duration ≤ 4 s → streaming / > 4 s → long-form)
      ├─ MoonshineV2Backend  (PyO3 0.28, moonshine-voice, ~9 ms P50)
      └─ WhisperBackend      (whisper-rs 0.16, whisper.cpp; state reused across calls)
  → TranscriptReady event (stays in Transcribing)
  → LLMBackend.complete(LLMRequest { system_prompt, messages, grammar, editor_context, tier: Tier::Fast })
      ├─ LlamaCppBackend             (llama-cpp-2, GGUF, prompt caching on LSP prefix; --features llama-cpp)
      ├─ OllamaBackend               (reqwest, localhost:11434; --features ollama)
      └─ OpenAICompatibleBackend     (opt-in only; --features openai-compatible; never default)
  → LLMOutput: ToolCall | Text
  → CleanupEngine.clean(text, mode)   (dictation/type_text branch only; off by default;
                                       YAZSES_CLEANUP_* env; guards + fallback — ADR-013)
  → ToolCallReady event → DaemonState: Transcribing → Injecting
  → Dispatcher.dispatch(call) — 20 tools, all implemented:
      ├─ type_text / key_sequence        (xdotool/wtype/ydotool; trailing space appended)
      ├─ commit_to_memory / recall / forget_last  (PersonalMemory)
      ├─ open_file                       (xdg-open)
      ├─ git_commit                      (git subprocess, SHA extracted)
      ├─ goto_symbol                     (nvim --server $NVIM --remote-send)
      ├─ app_launch                      (direct spawn → xdg-open fallback)
      ├─ window_focus                    (wmctrl → xdotool fallback)
      ├─ volume_set                      (wpctl → pactl fallback)
      ├─ media_play_pause                (playerctl → xdotool XF86AudioPlay)
      ├─ screenshot_named                (grim → scrot → gnome-screenshot)
      ├─ note_quick                      (async append ~/notes.md with ISO timestamp)
      ├─ time_set_timer                  (tokio::spawn + sleep + notify-send)
      ├─ dismiss_notification            (dunstctl → notify-send fallback)
      ├─ mode_switch                     (logged; daemon mode wiring next release)
      ├─ send_message                    (stub — platform messaging future)
      └─ clarify / cancel_request        (immediate returns)
  → DispatchComplete event → DaemonState: Injecting → Idle
  → LatencyTracker.record(elapsed_ms)   [P50/P95 over last 100 turns; visible in `yazses status`]

State machine¶

Loading ──ModelsLoaded──→ Idle ──HoldStart──→ Recording ──HoldEnd──→ Transcribing
                           ↑                                              │
                     DispatchComplete                            TranscriptReady (loop)
                           │                                    ToolCallReady ──→ Injecting
                        Injecting ◀─────────────────────────────────────┘
                           │
                     DispatchComplete → Idle

Idle → EnrollStart → Enrolling → EnrollComplete → Idle
Idle → RemoteStart → RemoteSetup → RemoteConnected → RemoteActive → RemoteStop → Idle
Any state + ErrorOccurred → Error → ErrorResolved → Idle

IPC methods (v1.0 additions)¶

Latency budget (v1.0.0-dev.5)¶

Hold-key down ─ 0 ms
├─ Input event dispatch          5 ms
├─ Editor bridge query (async)   ≤ 100 ms (warm cache)
├─ Audio capture + VAD           continuous, 30 ms/frame
├─ STT: Moonshine v2             9 ms P50   (streaming path, ≤ 4 s utterance)
├─ STT: Whisper-large-v3-turbo   200–500 ms (long-form path, > 4 s)
├─ LLM prompt-eval               50 ms (cached prefix) / 200 ms (cold)
├─ LLM decode ≤ 50 tokens        200 ms at ~25 tok/s
├─ Tool dispatch                 10–100 ms (OS subprocess)
Tool action visible ────────── ~750 ms P50 (streaming path)

LatencyTracker maintains a 100-sample VecDeque; P50 and P95 are reported by yazses status.

New features in v1.0.0-dev.5¶

Test coverage¶

94 Rust unit tests across all workspace crates (up from 85 in dev.4).

v0.4 Python Pipeline (legacy, preserved)¶

System overview¶

┌────────────────────────┐   ┌──────────────────┐
│ Hotkey hook            │──▶│ Audio recorder   │
│ (per-OS API)           │   │ PortAudio/16kHz  │
│ OR EMGBackend (v0.4.0) │   │ PreSpeechRingBuf │
└────────────────────────┘   └────────┬─────────┘
                                      │
                    ┌─────────────────▼──────────────────────────┐
                    │ STT pipeline                                 │
                    │  1. vad_calibrated (silence gate)           │
                    │  2. faster-whisper (CPU / int8)             │
                    │     ↑ initial_prompt from LspContextProvider│ ← v0.4.0
                    │  3. clean_text                              │
                    │  4. disfluency filter                       │
                    │  5. grammar.classify()                      │
                    │     Tier 1: regex rules (< 5 ms)            │
                    │     Tier 2: SLMRouter (llama-cpp, optional) │ ← v0.4.0
                    └───────────────────┬────────────────────────┘
                                        │
                       ┌──────────────────────────────────────────┐
                       │ Dispatcher                                 │
                       │  DICTATE  → llm_cleanup (optional)         │ ← ADR-013
                       │           → continuation spacing           │
                       │           → injector.inject(text)          │
                       │  COMMAND  → inject_key_sequence()           │
                       └────────────────┬───────────────────────────┘
                                        │
             ┌──────────────────────────▼─────────────────────────┐
             │ Injector (local or remote)                           │
             │  Local:  xdotool / ydotool / wtype / SendInput      │
             │  Remote: RemoteInjectorProxy → SSH → agent          │
             └─────────────────────────────────────────────────────┘

Daemon owns the state machine and IPC server:

  LOADING → IDLE ↔ RECORDING → TRANSCRIBING → INJECTING → IDLE
                 ↕               (ERROR)
  REMOTE_SETUP → REMOTE_ACTIVE
  ENROLLING
  PAUSED

DICTATE-path post-process chain (core/daemon.py::_on_hold_end). After clean_dictation, the cleaned transcript flows through a chain of opt-in, OFF-by-default text transforms, each guarded by its own config flag (if self._config.<x>.enabled: text = transform(text)). Order: self-repair (ADR-058) → inline compute (086) → phonetic corrector (027) → voice punctuation → entity ITN (045) → redaction (046) → symbols (055) → unit convert (056) → temporal (057) → grammar repair (050) → diacritize (122) → transliteration (116) → semantic line breaks (111) → structured markup (067) → SafeGlyph warn (123) → auto-pairing (088) → prosody (002). Adding a new pure-text feature = one guarded stanza here plus a test_v2_daemon_wiring.py case that enables the flag and asserts the effect. (Many other v2 cores are built + tested but not yet wired into a runtime path — see an internal planning note.)

Module map¶

src/yazses/core/¶

Guards between the transcript and the injector¶

Three run in daemon.py::_on_hold_end, in this order, and the order is load-bearing.

Each is judged on how rarely it fires. A guard that interrupts a house number teaches the user to dismiss it, and a dismissed guard costs attention while catching nothing — so each holds only on a specific, checkable signal, never a heuristic.

cmdsafety judges the command text and deliberately not the focused window: focus detection needs AT-SPI or X11, so on Wayland without AT-SPI the window class is empty, and a guard that silently stops protecting on a whole display server is worse than none.

Feedback surfaces¶

src/yazses/platform/¶

src/yazses/inject/ (Linux sub-backends)¶

src/yazses/stt/¶

src/yazses/recimport/ (Wave O, ADR-v2-125 — Diarized Recording Import)¶

CLI-only (yazses transcribe <file>); no daemon/IPC/hotkey involvement. Off by default; diarization is lazy behind the diarization extra. pipeline.py::transcribe_file is pure orchestration with injected backends, so every stage is unit-testable without models.

Privacy (ADR-011/012): fully offline; diarization labels are transient; voiceprint naming is opt-in, consent-gated, on-device, and never auto-enrolls third parties. Cloud escalation is designed but deferred (ADR-v2-126).

src/yazses/commands/¶

src/yazses/remote/¶

src/yazses/audio/¶

src/yazses/accessibility/¶

src/yazses/learning/ (v0.5.0, ADR-012 — opt-in self-improvement loop)¶

Off by default. When [learning] enabled = true, the daemon writes one event per hold-release (every text stage + optional audio, including discards) to a local encrypted corpus, off the dictation hot path. yazses tune turns it into reviewable config diffs.

src/yazses/hotkeys/¶

Optional dedicated command key. When [hotkey] command_key is set (≠ the dictation key), the daemon builds a second hotkey backend (_make_command_hotkey) and runs it in a background thread; the dictation key keeps the main thread. Holding the command key sets a per-burst _command_mode flag (_on_command_hold_start), which _on_hold_end consumes: the utterance is always parsed as a command and never typed as literal text — an unmatched phrase is discarded (discard_reason="command_unmatched"). Live streaming is suppressed in command mode so command words are never partially injected.

src/yazses/ipc/¶

src/yazses/postprocess/¶

src/yazses/tts/¶

Read-Back Loop (spec-read-back-loop): offline TTS that speaks the transcript back. Permissive engines only (Kokoro Apache-2.0 default). All deps in the optional tts extra, imported only when [tts] enabled.

Wired in core/daemon.py: built at startup; after a dictation injection, _maybe_read_back truncates to max_readback_chars and _speak_readback enters the READBACK state and speaks on a background thread. Commands are never read back. yazses say / readback_speak IPC speak on demand. status exposes read_back + tts_backend.

v2 perceptual & personalization layer (off by default; design/v2-cognitive-layer/)¶

Four advanced features. Each follows the optional-extra + dormant-factory pattern; the dependency-free cores are fully tested, the model/sensor/training parts are behind extras and gated.

New config sections: [voiceprint], [cocktail], [personalize], [gaze], [polyglot] (all off by default). New extras: voiceprint (speechbrain); gaze deps are manual-install (l2cs pins an old torch). doctor reports each enabled extra's importability.

src/yazses/overlay/ (voice-activity overlay, yazses-overlay)¶

Standalone process (separate from the daemon, which is blocked by the hotkey loop). A thin IPC client that polls status and renders "sonar" rings near the cursor that pulse with audio_level. Pure-logic modules are unit-tested; the Qt layer (PySide6, optional overlay extra) is thin and smoke-tested offscreen.

src/yazses/system/¶

Host-side helpers and the CLI-facing config/diagnostics surface (no daemon needed).

CLI commands added in the v1.1.x line (all write comment-preserving config, then prompt yazses restart): yazses restart (stop all daemons incl. detached, start one — start now restarts instead of duplicating), yazses features [enable/disable], yazses vocab [add/list/remove], yazses hotkey [show/set/command] (the command subcommand binds the dedicated command key; see src/yazses/hotkeys/ above).

IPC methods¶

Registered in core/daemon.py::_start_ipc_server. "Reached from" is the set of callers in src/ that actually send the method, not the set that could -- this table listed nine of the twenty-one, and three of those nine were described as CLI -> daemon when no CLI path had ever sent them. tests/test_ipc_methods_are_documented.py derives both columns from the tree, so a method added to the daemon, or a caller added or removed, fails here.

The last three are registered, implemented, and unreachable: no CLI command, tray entry or MCP tool sends them, so they answer a hand-written JSON-RPC client and nothing else. They are recorded as unreachable rather than deleted, because two of them are the only runtime streaming toggle that exists. Before wiring one up, note that neither writes config.toml, so the change lasts until the next restart and no surface reports it.

Configuration reference¶

All fields have defaults. config.toml only needs the sections you want to override.

[stt]
model = "tiny.en"          # tiny.en | base.en | medium.en | large-v3
device = "cpu"
compute_type = "int8"

[hotkey]
key = "auto"               # auto | space | right_ctrl | right_option | …
hold_threshold_ms = 500
source = "default"         # default | evdev (Linux footpedal)
evdev_device = ""          # e.g. /dev/input/event5

[audio]
sample_rate = 16000
channels = 1
max_record_seconds = 90

[general]
log_level = "INFO"

[streaming]
enabled = true
partial_interval_ms = 300
partial_marker = ""

[filters.disfluency]
enabled = true
filler_words = ["um", "uh", "er", "like", …]
self_correction_triggers = ["scratch that", "delete that", …]
collapse_repetitions = false   # ADR-015 opt-in: b-b-because → because, the the the → the
collapse_prolongations = false # ADR-015 opt-in: sooo → so
llm_enabled = false
llm_endpoint = "http://localhost:11434"
llm_allow_remote_endpoint = false  # cleanup refuses a non-loopback endpoint without this

[accessibility]
vad_threshold = 0.01       # calibrate with `yazses enroll`
min_silence_ms = 500
pre_speech_padding_ms = 200
vad_source = "default"
dysfluency_friendly = false # ADR-015 preset: enables both collapse_* + widens onset padding

[commands]
enabled = true
profile = "auto"           # auto | default | vscode | vim

[injection]
backend = "auto"               # auto | xdotool | ydotool | wtype | clipboard
fallback_to_clipboard = true
continuation_window_ms = 30000 # 0 disables; bursts within this window get a
                               # separating leading space so words don't glue
                               # together at the boundary (postprocess/spacing.py)

[remote]
default_host = ""
ssh_port = 22
agent_port = 9875
key_file = ""

Adding a new platform¶

1. Create src/yazses/platform/<os>/ with implementations of all Protocol interfaces.
2. Register the new sys.platform string in platform/factory.py.
3. No changes needed in the daemon, CLI, or any other module.

Key architectural decisions (ADRs)¶

v0.3.0 ADRs¶

Full v0.3.0 ADR files: research/yazses-innovation/output/adrs/

v0.4.0 ADRs¶

Full v0.4.0 ADR files: docs/adr/

2026-08 — packaging, egress and direction¶

Each of these is a numbered ADR in design/adr/, and each records what would reverse it.

v0.4.0 Configuration reference additions¶

[commands]
# Tier 2 SLM routing (optional; needs yazses[slm] + GGUF model file)
slm_model_path = ""                  # e.g. ~/.cache/yazses/models/tinyllama.gguf
slm_confidence_threshold = 0.75

# LSP code context injection (optional; needs yazses[lsp])
lsp_enabled = false
lsp_editor = "auto"                  # auto | neovim | vscode

[emg]
# EMG silent speech backend (optional; needs yazses[emg] + device)
device_port = ""                     # e.g. /dev/ttyUSB0, COM3
baud_rate = 115200
mode = "command"                     # command | full_text
# command_map: device label → grammar phrase
# [emg.command_map]
# save = "save file"
# undo = "undo"

v2.0.0 — Voice-First Interaction Layer (v2.0.0-dev)¶

v2 extends the daemon from dictation into a broader on-device interaction layer. Every feature is off by default and guarded, so the v1 pipeline is unchanged when all are off. Full rationale in design/adr/adr-v2-000..013 and the integration map in an internal design note.

New packages (src/yazses/)¶

Daemon integration points (core/daemon.py)¶

• _effective_initial_prompt — Context-Primed terms + Personal Adapter corpus-mined bias.
• decode path — Confidence Ink count (metadata only); Whisper-Mode effective VAD threshold.
• _on_hold_end command mode — Ambient Scratch capture, Spoken Edit (_try_spoken_edit).
• IPC — recall, scratch methods; _handle_status exposes confidence_enabled + low_confidence_last.
• startup — PolyglotRouter seam (dormant unless a CS adapter is set).

Waves D–O (v2.1–v2.11 developer preview)¶

The table above lists the founding Wave A–C packages (ADRs v2-000..013). Waves D–O add ~100 more off-by-default capabilities following the same pattern — a pure, 100%-covered core module under src/yazses/<feature>/, an enabled = False dataclass in config.py, a system/features.py registry entry, and heavy backends (if any) lazy-imported behind an optional extra. As of v2.11.0-dev.1: 135 capabilities, 1489 tests green, ADRs adr-v2-001..126.

Wave O (v2.11) opens the first file-ingestion entry point: yazses transcribe <file> (src/yazses/recimport/) — decode any audio format offline (PyAV via faster_whisper.decode_audio, no new dep) → transcribe_words() → optional speaker diarization (SherpaDiarizer, sherpa-onnx int8 ONNX behind the diarization extra) → pure max-overlap word↔turn alignment (align.py) → speaker naming (naming.py, reuses voiceprint/) → render.py sidecar (txt/md/srt/vtt/json). Unlike the rest of v2 this is a CLI-only path (no daemon/IPC/hotkey change). ADR-v2-125; cloud escalation designed but deferred (ADR-v2-126).

Rather than duplicate the full catalog here, the authoritative, always-current sources are: docs/v2-features.md (user-facing catalog), design/adr/adr-v2-* (per-feature rationale), and the per-wave SoA research notes (internal). Each is updated in lockstep with the code every dev tag.

Design invariants¶

On-device only, no telemetry, no transcript persistence beyond the opt-in encrypted corpus (ADR-011/012). Heavy backends (SLM+MCP for agent, pyatspi for pilot, gaze webcam, EMG serial) are lazy-imported behind opt-in extras/config; the pure planner/ router/parse layers carry no heavy deps and are fully unit-tested.