Skip to content

ADR-003: LLM Backend — llama.cpp Default + MLX Apple Silicon Fast Path

Status: Accepted
Date: 2026-05-18
Deciders: Mohsen Seyedkazemi Ardebili

Context

The on-device LLM is the engine for the agent loop, constrained tool-call decoding, and future reasoning capabilities. YazSes must cover Linux, macOS, and Windows. Among open runtimes, llama.cpp is cross-platform, supports every open-weight model via GGUF format, provides native GBNF grammar-constrained decoding, and has prompt caching — matching the architectural pattern of the whisper.cpp/llama.cpp ecosystem. MLX-LM is faster than llama.cpp on Apple Silicon due to unified memory and Metal kernels, but is macOS-Apple-Silicon-only. Ollama wraps llama.cpp with an HTTP API but adds a separate daemon. Apple Foundation Models are closed-weights and not adoptable as a default.

The default model must cover both English and Persian/multilingual use. Qwen3-7B-Instruct Q4_K_M satisfies this: its 151,936-token vocabulary provides substantially better Arabic-script coverage than Llama-3's 128K BPE vocabulary, which under-represents Farsi.

Decision

llama.cpp is the default LLM runtime on all platforms. MLX-LM is selected automatically on macOS Apple Silicon as a performance fast path. Platform mapping at startup:

  • Linux, macOS Intel, Windows: LlamaCppBackend
  • macOS Apple Silicon: MlxBackend (falls back to LlamaCppBackend if the model is unsupported)

The default model is Qwen3-7B-Instruct Q4_K_M. English-only alternatives: Llama-3.1-8B-Instruct Q4_K_M (broader community support), Phi-4-mini-instruct Q4_K_M (~2.5 GB, for 8 GB RAM devices). Prompt caching (cache_prompt=true) is mandatory when the EditorBridge LSP context is active.

Optional backends: OllamaBackend (auto-detected via OLLAMA_HOST or ~/.ollama), OpenAICompatibleBackend (explicit opt-in via ~/.config/yazses/cloud.toml, off by default).

Consequences

Positive: - Single GGUF model format across the entire cross-platform fleet. - Native GBNF grammar-constrained decoding supports ADR-004 with microsecond per-token overhead. - Prompt caching keeps the system prompt and editor context warm across turns, supporting the agent-loop latency budget. - Apple Silicon users gain approximately 50% throughput improvement over llama.cpp via the MLX fast path. - Both runtimes are MIT-licensed — no licence friction with the YazSes core.

Negative / trade-offs: - Two backends to maintain (~200–400 LOC of Protocol adapter code each, plus integration tests). - MLX model availability lags llama.cpp by days to weeks for new releases; macOS users fall back to llama.cpp in that window. - llama.cpp chat-template and tool-call schema handling occasionally lags upstream model releases; requires quarterly pinned release bumps.

Implementation

LlamaCppBackend is in yazses-llm-llamacpp/src/. MlxBackend is in yazses-llm-mlx/src/ and uses cxx-rs bindings to the MLX C++ API. The shared LLMBackend trait is defined in yazses-llm/src/backend.rs. Platform detection and backend selection occur at daemon startup in yazses-core/src/daemon.rs.