Skip to content

ADR: Use llama-cpp-python (in-process GGUF) for Tier 2 SLM inference

Context

YazSes v0.4 introduces a Tier 2 SLM intent routing layer that classifies voice command transcripts that fail the existing regex grammar (Tier 1). This layer must operate on CPU hardware without a GPU, must not require cloud connectivity, must integrate with the existing Python daemon process, and must complete inference within a 300 ms budget per utterance.

The two primary implementation approaches are: (1) an in-process Python inference library that loads the model into the daemon process memory space, or (2) an out-of-process inference server (e.g., Ollama, LM Studio) that the daemon queries over a local socket. A third approach is a cloud API, but this is excluded by the offline-first constraint.

The daemon currently has no external service dependencies at runtime. Introducing a dependency on a running local inference server would change the daemon startup contract (requires a separate process to be running) and add 5–20 ms of IPC overhead per inference call.

The cap-001 feasibility case is validated by src-003, which demonstrates TinyLlama-1.1B at 4-bit quantisation running on edge hardware without cloud connectivity for speech-to-action intent classification. The same GGUF format used in that paper is the standard interchange format for llama-cpp-python.

Decision

We will use llama-cpp-python >= 0.3.0 as the in-process inference backend for Tier 2 SLM classification, loading a GGUF-quantised model (Phi-3-mini-4k-Q4_K_M or TinyLlama-1.1B-Q4_K_M) into the daemon process at startup.

In practice this means: the SLMRouter module imports llama_cpp.Llama at startup; the model is loaded once into the daemon's RAM and kept resident for the daemon's lifetime; each Tier 2 classification call invokes Llama.create_completion() synchronously within the post-transcription pipeline stage. The model is loaded only if CommandsConfig.slm_model_path resolves to an existing GGUF file; if the file is absent, Tier 2 is disabled and the daemon runs in Tier 1 (regex-only) mode. [HYPOTHESIS]

Consequences

Positive

  • Zero runtime service dependency — the daemon starts without requiring any external process. [HYPOTHESIS]
  • No IPC overhead per inference call — model state is in the same process memory. [HYPOTHESIS]
  • llama-cpp-python is the most widely-used Python binding for GGUF models; it receives regular updates for new quantisation formats and model architectures. [HYPOTHESIS]
  • The GGUF format is the de-facto standard for quantised open-weight models; Phi-3-mini and TinyLlama are both available in GGUF from Hugging Face under Apache-2.0. [HYPOTHESIS]
  • [EVIDENCE src-003] TinyLlama-1.1B at 4-bit quantisation is directly validated as feasible for speech-to-action intent classification on edge hardware in the survey literature.

Negative

  • The daemon's RAM footprint increases by 700 MB (TinyLlama variant) or 2.2 GB (Phi-3-mini variant) at startup. On machines with < 6 GB available RAM, this may cause memory pressure alongside faster-whisper's own footprint. [HYPOTHESIS]
  • llama-cpp-python wheels are platform-specific (C++ extension) and have had version compatibility issues on some Linux distributions (glibc version, AVX2 availability). The CI matrix (Linux x86_64, macOS arm64, Windows x86_64) must validate wheel availability for the pinned version. [HYPOTHESIS]
  • Loading the SLM model adds 2–5 seconds to daemon startup time. Users with slm_model_path configured will notice a longer startup. A loading indicator in yazses doctor output is required. [HYPOTHESIS]

Neutral

  • The model file must be downloaded separately and is not bundled in the YazSes package. This is standard practice for ML model deployments; yazses doctor will direct users to the download command.
  • The llama-cpp-python library exposes a broader API than we use; we consume only Llama.__init__ and Llama.create_completion. This surface is stable and unlikely to change incompatibly.

Alternatives Considered

Alternative Reason Rejected
Ollama local inference server Requires a separate long-running process (ollama serve). Changes the daemon's startup contract from "single process" to "requires Ollama running". Adds 5–15 ms socket IPC overhead per call. Ollama is a great user-facing tool but the wrong abstraction for an embedded daemon component. [HYPOTHESIS]
ctransformers library Functionally equivalent to llama-cpp-python for GGUF models, but has lower maintenance activity (last release > 6 months old as of 2026-05). llama-cpp-python has more active development and broader model support. [HYPOTHESIS]
HuggingFace Transformers (float16 or bfloat16) Requires GPU or very slow on CPU at full precision; 4-bit GGUF quantisation via llama-cpp-python is the purpose-built path for CPU inference. [HYPOTHESIS]
ONNX Runtime with quantised model Valid alternative; ONNX Runtime is well-maintained and has CPU int8 support. However, Phi-3-mini and TinyLlama ONNX exports are less commonly available and less tested than GGUF exports. Revisit if llama-cpp-python wheel issues prove blocking. [HYPOTHESIS]

References

  • [EVIDENCE src-003]: edge-cloud adaptive inference paper demonstrating TinyLlama-1.1B 4-bit for offline speech-to-action.

Output Quality Note

  • All placeholders replaced.
  • First sentence of Decision is "We will use..."
  • Every claim tagged.
  • deciders populated.
  • Alternatives table has ≥ 2 rows.
  • Negative consequences section is honest.

Study: [[yazses-future-voice-hci/input/research_scope|yazses-future-voice-hci]]