YazSes Architecture¶

Last updated: 2026-05-18 (v1.0.0-dev.5)

v1.0 Rust-core Architecture (archived — archive/rust-hci-v1 branch)¶

This Rust architecture is not on main anymore. It was moved to the archive/rust-hci-v1 branch and paused in early stages. main is the Python app (Part 1; see docs/cli-reference.md and the README). Kept here for reference.

Workspace crates¶

v1.0 Pipeline¶

Hold-key / EMG squeeze (yazses-inputs / InputBackend)
  → HoldStart event → DaemonState: Idle → Recording
  → AudioCapture (cpal) + VadGate (RMS; or SileroVad with --features silero)
  → PaddingBuffer prepends pre-speech ring buffer
  → HoldEnd event → DaemonState: Recording → Transcribing
  → EditorBridge.get_context() [async, parallel with audio]
      └─ 5-tier WindowDetector (Hyprland → Sway → WLR-toplevel → X11 EWMH → Null)
      └─ NeovimBridge (nvim-rs, $NVIM socket) / VSCodeBridge (TCP push, port 57843)
      └─ EditorContext.to_initial_prompt(224 BPE tokens) → ASR initial_prompt
      └─ EditorContext.to_llm_block() → LLM <editor_context> block
  → STTRouter (duration ≤ 4 s → streaming / > 4 s → long-form)
      ├─ MoonshineV2Backend  (PyO3 0.28, moonshine-voice, ~9 ms P50)
      └─ WhisperBackend      (whisper-rs 0.16, whisper.cpp; state reused across calls)
  → TranscriptReady event (stays in Transcribing)
  → LLMBackend.complete(LLMRequest { system_prompt, messages, grammar, editor_context, tier: Tier::Fast })
      ├─ LlamaCppBackend             (llama-cpp-2, GGUF, prompt caching on LSP prefix; --features llama-cpp)
      ├─ OllamaBackend               (reqwest, localhost:11434; --features ollama)
      └─ OpenAICompatibleBackend     (opt-in only; --features openai-compatible; never default)
  → LLMOutput: ToolCall | Text
  → CleanupEngine.clean(text, mode)   (dictation/type_text branch only; off by default;
                                       YAZSES_CLEANUP_* env; guards + fallback — ADR-013)
  → ToolCallReady event → DaemonState: Transcribing → Injecting
  → Dispatcher.dispatch(call) — 20 tools, all implemented:
      ├─ type_text / key_sequence        (xdotool/wtype/ydotool; trailing space appended)
      ├─ commit_to_memory / recall / forget_last  (PersonalMemory)
      ├─ open_file                       (xdg-open)
      ├─ git_commit                      (git subprocess, SHA extracted)
      ├─ goto_symbol                     (nvim --server $NVIM --remote-send)
      ├─ app_launch                      (direct spawn → xdg-open fallback)
      ├─ window_focus                    (wmctrl → xdotool fallback)
      ├─ volume_set                      (wpctl → pactl fallback)
      ├─ media_play_pause                (playerctl → xdotool XF86AudioPlay)
      ├─ screenshot_named                (grim → scrot → gnome-screenshot)
      ├─ note_quick                      (async append ~/notes.md with ISO timestamp)
      ├─ time_set_timer                  (tokio::spawn + sleep + notify-send)
      ├─ dismiss_notification            (dunstctl → notify-send fallback)
      ├─ mode_switch                     (logged; daemon mode wiring next release)
      ├─ send_message                    (stub — platform messaging future)
      └─ clarify / cancel_request        (immediate returns)
  → DispatchComplete event → DaemonState: Injecting → Idle
  → LatencyTracker.record(elapsed_ms)   [P50/P95 over last 100 turns; visible in `yazses status`]

State machine¶

Loading ──ModelsLoaded──→ Idle ──HoldStart──→ Recording ──HoldEnd──→ Transcribing
                           ↑                                              │
                     DispatchComplete                            TranscriptReady (loop)
                           │                                    ToolCallReady ──→ Injecting
                        Injecting ◀─────────────────────────────────────┘
                           │
                     DispatchComplete → Idle

Idle → EnrollStart → Enrolling → EnrollComplete → Idle
Idle → RemoteStart → RemoteSetup → RemoteConnected → RemoteActive → RemoteStop → Idle
Any state + ErrorOccurred → Error → ErrorResolved → Idle

IPC methods (v1.0 additions)¶

Latency budget (v1.0.0-dev.5)¶

Hold-key down ─ 0 ms
├─ Input event dispatch          5 ms
├─ Editor bridge query (async)   ≤ 100 ms (warm cache)
├─ Audio capture + VAD           continuous, 30 ms/frame
├─ STT: Moonshine v2             9 ms P50   (streaming path, ≤ 4 s utterance)
├─ STT: Whisper-large-v3-turbo   200–500 ms (long-form path, > 4 s)
├─ LLM prompt-eval               50 ms (cached prefix) / 200 ms (cold)
├─ LLM decode ≤ 50 tokens        200 ms at ~25 tok/s
├─ Tool dispatch                 10–100 ms (OS subprocess)
Tool action visible ────────── ~750 ms P50 (streaming path)

LatencyTracker maintains a 100-sample VecDeque; P50 and P95 are reported by yazses status.

New features in v1.0.0-dev.5¶

Test coverage¶

94 Rust unit tests across all workspace crates (up from 85 in dev.4).

v0.4 Python Pipeline (legacy, preserved)¶

System overview¶

┌────────────────────────┐   ┌──────────────────┐
│ Hotkey hook            │──▶│ Audio recorder   │
│ (per-OS API)           │   │ PortAudio/16kHz  │
│ OR EMGBackend (v0.4.0) │   │ PreSpeechRingBuf │
└────────────────────────┘   └────────┬─────────┘
                                      │
                    ┌─────────────────▼──────────────────────────┐
                    │ STT pipeline                                 │
                    │  1. vad_calibrated (silence gate)           │
                    │  2. faster-whisper (CPU / int8)             │
                    │     ↑ initial_prompt from LspContextProvider│ ← v0.4.0
                    │  3. clean_text                              │
                    │  4. disfluency filter                       │
                    │  5. grammar.classify()                      │
                    │     Tier 1: regex rules (< 5 ms)            │
                    │     Tier 2: SLMRouter (llama-cpp, optional) │ ← v0.4.0
                    └───────────────────┬────────────────────────┘
                                        │
                       ┌──────────────────────────────────────────┐
                       │ Dispatcher                                 │
                       │  DICTATE  → llm_cleanup (optional)         │ ← ADR-013
                       │           → continuation spacing           │
                       │           → injector.inject(text)          │
                       │  COMMAND  → inject_key_sequence()           │
                       └────────────────┬───────────────────────────┘
                                        │
             ┌──────────────────────────▼─────────────────────────┐
             │ Injector (local or remote)                           │
             │  Local:  xdotool / ydotool / wtype / SendInput      │
             │  Remote: RemoteInjectorProxy → SSH → agent          │
             └─────────────────────────────────────────────────────┘

Daemon owns the state machine and IPC server:

  LOADING → IDLE ↔ RECORDING → TRANSCRIBING → INJECTING → IDLE
                 ↕               (ERROR)
  REMOTE_SETUP → REMOTE_ACTIVE
  ENROLLING
  PAUSED

DICTATE-path post-process chain (core/daemon.py::_on_hold_end). After clean_dictation, the cleaned transcript flows through a chain of opt-in, OFF-by-default text transforms, each guarded by its own config flag (if self._config.<x>.enabled: text = transform(text)). Order: self-repair (ADR-058) → inline compute (086) → phonetic corrector (027) → voice punctuation → entity ITN (045) → redaction (046) → symbols (055) → unit convert (056) → temporal (057) → grammar repair (050) → diacritize (122) → transliteration (116) → semantic line breaks (111) → structured markup (067) → SafeGlyph warn (123) → auto-pairing (088) → prosody (002). Adding a new pure-text feature = one guarded stanza here plus a test_v2_daemon_wiring.py case that enables the flag and asserts the effect. (Many other v2 cores are built + tested but not yet wired into a runtime path — see an internal planning note.)

Module map¶

src/yazses/core/¶

src/yazses/platform/¶

src/yazses/inject/ (Linux sub-backends)¶

src/yazses/stt/¶

src/yazses/recimport/ (Wave O, ADR-v2-125 — Diarized Recording Import)¶

CLI-only (yazses transcribe <file>); no daemon/IPC/hotkey involvement. Off by default; diarization is lazy behind the diarization extra. pipeline.py::transcribe_file is pure orchestration with injected backends, so every stage is unit-testable without models.

Privacy (ADR-011/012): fully offline; diarization labels are transient; voiceprint naming is opt-in, consent-gated, on-device, and never auto-enrolls third parties. Cloud escalation is designed but deferred (ADR-v2-126).

src/yazses/commands/¶

src/yazses/remote/¶

src/yazses/audio/¶

src/yazses/accessibility/¶

src/yazses/learning/ (v0.5.0, ADR-012 — opt-in self-improvement loop)¶

Off by default. When [learning] enabled = true, the daemon writes one event per hold-release (every text stage + optional audio, including discards) to a local encrypted corpus, off the dictation hot path. yazses tune turns it into reviewable config diffs.

src/yazses/hotkeys/¶

Optional dedicated command key. When [hotkey] command_key is set (≠ the dictation key), the daemon builds a second hotkey backend (_make_command_hotkey) and runs it in a background thread; the dictation key keeps the main thread. Holding the command key sets a per-burst _command_mode flag (_on_command_hold_start), which _on_hold_end consumes: the utterance is always parsed as a command and never typed as literal text — an unmatched phrase is discarded (discard_reason="command_unmatched"). Live streaming is suppressed in command mode so command words are never partially injected.

src/yazses/ipc/¶

src/yazses/postprocess/¶

src/yazses/tts/¶

Read-Back Loop (spec-read-back-loop): offline TTS that speaks the transcript back. Permissive engines only (Kokoro Apache-2.0 default). All deps in the optional tts extra, imported only when [tts] enabled.

Wired in core/daemon.py: built at startup; after a dictation injection, _maybe_read_back truncates to max_readback_chars and _speak_readback enters the READBACK state and speaks on a background thread. Commands are never read back. yazses say / readback_speak IPC speak on demand. status exposes read_back + tts_backend.

v2 perceptual & personalization layer (off by default; design/v2-cognitive-layer/)¶

Four advanced features. Each follows the optional-extra + dormant-factory pattern; the dependency-free cores are fully tested, the model/sensor/training parts are behind extras and gated.

New config sections: [voiceprint], [cocktail], [personalize], [gaze], [polyglot] (all off by default). New extras: voiceprint (speechbrain); gaze deps are manual-install (l2cs pins an old torch). doctor reports each enabled extra's importability.

src/yazses/overlay/ (voice-activity overlay, yazses-overlay)¶

Standalone process (separate from the daemon, which is blocked by the hotkey loop). A thin IPC client that polls status and renders "sonar" rings near the cursor that pulse with audio_level. Pure-logic modules are unit-tested; the Qt layer (PySide6, optional overlay extra) is thin and smoke-tested offscreen.

src/yazses/system/¶

Host-side helpers and the CLI-facing config/diagnostics surface (no daemon needed).

CLI commands added in the v1.1.x line (all write comment-preserving config, then prompt yazses restart): yazses restart (stop all daemons incl. detached, start one — start now restarts instead of duplicating), yazses features [enable/disable], yazses vocab [add/list/remove], yazses hotkey [show/set/command] (the command subcommand binds the dedicated command key; see src/yazses/hotkeys/ above).

IPC methods¶

Configuration reference¶

All fields have defaults. config.toml only needs the sections you want to override.

[stt]
model = "tiny.en"          # tiny.en | base.en | medium.en | large-v3
device = "cpu"
compute_type = "int8"

[hotkey]
key = "auto"               # auto | space | right_ctrl | right_option | …
hold_threshold_ms = 500
source = "default"         # default | evdev (Linux footpedal)
evdev_device = ""          # e.g. /dev/input/event5

[audio]
sample_rate = 16000
channels = 1
max_record_seconds = 90

[general]
log_level = "INFO"

[streaming]
enabled = true
partial_interval_ms = 300
partial_marker = ""

[filters.disfluency]
enabled = true
filler_words = ["um", "uh", "er", "like", …]
self_correction_triggers = ["scratch that", "delete that", …]
collapse_repetitions = false   # ADR-015 opt-in: b-b-because → because, the the the → the
collapse_prolongations = false # ADR-015 opt-in: sooo → so
llm_enabled = false
llm_endpoint = "http://localhost:11434"
llm_allow_remote_endpoint = false  # cleanup refuses a non-loopback endpoint without this

[accessibility]
vad_threshold = 0.01       # calibrate with `yazses enroll`
min_silence_ms = 500
pre_speech_padding_ms = 200
vad_source = "default"
dysfluency_friendly = false # ADR-015 preset: enables both collapse_* + widens onset padding

[commands]
enabled = true
profile = "auto"           # auto | default | vscode | vim

[injection]
backend = "auto"               # auto | xdotool | ydotool | wtype | clipboard
fallback_to_clipboard = true
continuation_window_ms = 30000 # 0 disables; bursts within this window get a
                               # separating leading space so words don't glue
                               # together at the boundary (postprocess/spacing.py)

[remote]
default_host = ""
ssh_port = 22
agent_port = 9875
key_file = ""

Adding a new platform¶

1. Create src/yazses/platform/<os>/ with implementations of all Protocol interfaces.
2. Register the new sys.platform string in platform/factory.py.
3. No changes needed in the daemon, CLI, or any other module.

Key architectural decisions (ADRs)¶

v0.3.0 ADRs¶

Full v0.3.0 ADR files: research/yazses-innovation/output/adrs/

v0.4.0 ADRs¶

Full v0.4.0 ADR files: docs/adr/

v0.4.0 Configuration reference additions¶

[commands]
# Tier 2 SLM routing (optional; needs yazses[slm] + GGUF model file)
slm_model_path = ""                  # e.g. ~/.cache/yazses/models/tinyllama.gguf
slm_confidence_threshold = 0.75

# LSP code context injection (optional; needs yazses[lsp])
lsp_enabled = false
lsp_editor = "auto"                  # auto | neovim | vscode

[emg]
# EMG silent speech backend (optional; needs yazses[emg] + device)
device_port = ""                     # e.g. /dev/ttyUSB0, COM3
baud_rate = 115200
mode = "command"                     # command | full_text
# command_map: device label → grammar phrase
# [emg.command_map]
# save = "save file"
# undo = "undo"

v2.0.0 — Voice-First Interaction Layer (v2.0.0-dev)¶

v2 extends the daemon from dictation into a broader on-device interaction layer. Every feature is off by default and guarded, so the v1 pipeline is unchanged when all are off. Full rationale in design/adr/adr-v2-000..013 and the integration map in an internal design note.

New packages (src/yazses/)¶

Daemon integration points (core/daemon.py)¶

• _effective_initial_prompt — Context-Primed terms + Personal Adapter corpus-mined bias.
• decode path — Confidence Ink count (metadata only); Whisper-Mode effective VAD threshold.
• _on_hold_end command mode — Ambient Scratch capture, Spoken Edit (_try_spoken_edit).
• IPC — recall, scratch methods; _handle_status exposes confidence_enabled + low_confidence_last.
• startup — PolyglotRouter seam (dormant unless a CS adapter is set).

Waves D–O (v2.1–v2.11 developer preview)¶

The table above lists the founding Wave A–C packages (ADRs v2-000..013). Waves D–O add ~100 more off-by-default capabilities following the same pattern — a pure, 100%-covered core module under src/yazses/<feature>/, an enabled = False dataclass in config.py, a system/features.py registry entry, and heavy backends (if any) lazy-imported behind an optional extra. As of v2.11.0-dev.1: 135 capabilities, 1489 tests green, ADRs adr-v2-001..126.

Wave O (v2.11) opens the first file-ingestion entry point: yazses transcribe <file> (src/yazses/recimport/) — decode any audio format offline (PyAV via faster_whisper.decode_audio, no new dep) → transcribe_words() → optional speaker diarization (SherpaDiarizer, sherpa-onnx int8 ONNX behind the diarization extra) → pure max-overlap word↔turn alignment (align.py) → speaker naming (naming.py, reuses voiceprint/) → render.py sidecar (txt/md/srt/vtt/json). Unlike the rest of v2 this is a CLI-only path (no daemon/IPC/hotkey change). ADR-v2-125; cloud escalation designed but deferred (ADR-v2-126).

Rather than duplicate the full catalog here, the authoritative, always-current sources are: docs/v2-features.md (user-facing catalog), design/adr/adr-v2-* (per-feature rationale), and the per-wave SoA research notes (internal). Each is updated in lockstep with the code every dev tag.

Design invariants¶

On-device only, no telemetry, no transcript persistence beyond the opt-in encrypted corpus (ADR-011/012). Heavy backends (SLM+MCP for agent, pyatspi for pilot, gaze webcam, EMG serial) are lazy-imported behind opt-in extras/config; the pure planner/ router/parse layers carry no heavy deps and are fully unit-tested.