Skip to content

SoA research — on-device LLM agents & agentic OS (2024–2026)

Date: 2026-07-02 · Tier: design/ — public engineering research · Author: Mohsen Seyedkazemi Ardebili Companion: domain 2 of the 5-domain v2 vision sweep. Fed adr-v2-004 (Context-Primed Dictation), adr-v2-005 (Spoken Recall), adr-v2-006 (Spoken MCP), adr-v2-007 (AT-SPI Voice Pilot), adr-v2-009 (Personal Adapter), and adr-v2-010's needs_confirm policy. See the waves index.

A snapshot of the field on the date above, kept as the research record behind the ADRs it fed. Where a "candidate feature" below never shipped, treat it as a research note, not a promise — check yazses features and the linked ADRs for what is actually built.

Key SoA findings

  • Apple shipped a ~3B on-device model with a public API — Foundation Models framework (iOS/macOS 26, Sept 2025): 2-bit QAT, guided generation (constrained decoding into Swift structs), tool calling, offline. [Apple Newsroom; Apple ML Research tech report 2025]
  • Apple's LoRA hot-swap is the reference design for local multi-task personalization: one frozen ~3B base + many task LoRA adapters (≤~1% weights) hot-swapped without recompilation. [Predibase; Orion/ANE paper]
  • Microsoft Phi Silica — NPU-resident SLM with a public Windows App SDK API (Click-to-Do, summarization, parts of Recall); Windows now has native OS-level MCP support. [Windows Dev Blog Build/Ignite 2025]
  • Google Gemini Nano to third-party apps via ML Kit GenAI on AICore (request-isolated, no retention); Prompt API alpha Oct 2025. [Android Dev Blog]
  • Open small models are agent-capable on CPU/consumer GPU: Phi-4-mini (3.8B, ~3GB Q4), Gemma 3 4B (vision + 140 langs), Qwen3 designed for agents/tool calling. Caveat: Gemma 3 tool-calling weaker than Llama/Mistral/Qwen. [secondary]
  • llama.cpp GBNF grammars give fully-local deterministic tool-calling (schema cannot be violated); json-schema-to-grammar + LlamaGrammar. [llama.cpp README]
  • MCP became the industry-standard agent tool interface in ~12 months (Anthropic Nov 2024 → OpenAI, Google DeepMind → Linux Foundation; Nov 2025 spec). [MCP blog]
  • Computer-use/GUI agents work but are below humans on real OS tasks — OSWorld human baseline ~72–84%; frontier ~low-70s%. Long-horizon degrades sharply past ~50 steps. [OSWorld; OSWorld-Human arXiv 2506.16042] (highest 2026 figures unverified)
  • Accessibility-tree control beats pixel/vision for local agents — AT-SPI/UIA/ macOS AX give exact element IDs, ms actions, nothing leaves machine; historically underused. [UFO2 arXiv 2504.14603]
  • Fully-local RAG "second brain" is commodity — sqlite-vec (SIMD, binary quantization + Hamming) handles 100k+ docs, zero infra; Khoj reference app. [sqlite-vec]
  • On-device fine-tuning moved phone-native — MobileFineTuner/MobileRAG show local LoRA + adapter storage. [arXiv 2512.08211, 2507.01079]
  • Context management, not model size, is the on-device bottleneck for multi-step. [arXiv 2511.03728]

Gaps / opportunities (privacy-first, local, voice-driven agent)

  • Voice is the missing input modality in the agentic-OS wave — everyone shipped on-device text SLM APIs + MCP hosts, but flagship demos are text/screenshot. Hands-free hold-to-talk into local tool calls was underserved; offline STT + hotkey + injection is the hard part most tools don't own.
  • AT-SPI grounding is cited as superior yet barely shipped, especially on Linux — a defensible, privacy-clean approach (structured perception, ms latency, no screenshots).
  • Local structured tool-calling is trivial now but under-exploited for voice.
  • "Recall-style" screen memory is a privacy lightning rod; a voice-annotated, opt-in, encrypted alternative was open space — this project already had the encrypted, consent-first corpus (adr-011/adr-012) that Recall lacked.
  • On-device LoRA personalization proven by Apple, absent from open voice tools.
  • Long-horizon reliability unsolved (<50-step cliff) — a reason to stay narrow: short, verifiable, human-in-the-loop voice actions sidestep the failure mode.

Candidate features considered in this domain

  1. Voice-to-Tool ("Spoken MCP") — speak intent → local SLM emits GBNF-constrained MCP tool call executed against local MCP servers; fully offline. Risk: misfire → mandatory spoken confirm + per-tool allowlist.
  2. AT-SPI Voice Pilot — "click Save", "focus terminal" resolved against the live accessibility tree; no screenshots, ms latency, Linux-first. Risk: broken trees in Electron/terminals → fall back to keystrokes.
  3. Encrypted Spoken Recall — every dictation/command optionally a searchable, timestamped, app-tagged memory in the existing encrypted corpus (sqlite-vec). Risk: privacy perception → strictly opt-in, honor redaction/retention.
  4. Context-Grounded Commanding — resolve deictic commands ("rename this function") via active editor LSP symbols already fed to initial_prompt. Risk: Neovim-first → degrade to dictation.
  5. Personal Command Adapter — opt-in nightly local LoRA on the corpus so the intent classifier learns your phrasings; gated on held-out win. Risk: overfit → hard promotion gate + base fallback.
  6. Verify-Before-Act Voice Loop — for 2–5 step actions the SLM speaks/overlays the plan and executes on "go", to work around the long-horizon cliff. Reuses TTS + overlay. Risk: friction → configurable step threshold.

Verification notes, from the original sweep: the Apple/MS/Google/MCP/sqlite-vec/ llama.cpp/Khoj facts came from primary sources and were treated as solid; specific SLM benchmark/VRAM numbers came from secondary roundups and were treated as directional only; the highest 2026 OSWorld figures (82–85%) were flagged as unverified, trusting only the ~low-70s% frontier and the official leaderboard. Citations here have not been re-verified against research/verify_refs.py.