Skip to content

ADR-004: Grammar-Constrained Tool Calls

Status: Accepted
Date: 2026-05-18
Deciders: Mohsen Seyedkazemi Ardebili

Context

The agent loop produces tool calls that drive OS-level actions (type_text, key_sequence, commit_to_memory, recall, clarify, etc.). A malformed JSON output, an invented tool name, or a hallucinated parameter from the LLM causes a failed dispatch. Grammar-constrained decoding — compiling a target schema into a token-level finite-state automaton that masks logits at every decode step — eliminates this class of failure by construction, with microsecond per-token overhead. Both llama.cpp (GBNF / JSON-Schema) and Outlines (FSA over any PyTorch-class model) provide this primitive. The v1.0 tool space is small (≤ 20 tools) and stable, making a compile-once-at-startup approach practical.

Decision

Every tool is declared once as a typed schema: a Pydantic model on the Python plugin path, or a serde-derived struct on the Rust core path. A ToolSchema registry compiles these to per-backend constraint artifacts at daemon startup:

  • LlamaCppBackend: GBNF grammar, via llama.cpp's native JSON-Schema converter.
  • MlxBackend and other Python-class backends: Outlines FSA, via outlines.generate.json(model, PydanticModel).
  • Rust-defined schemas: schemars translates serde structs to JSON Schema, then to GBNF or Outlines per the same paths.

The dispatcher receives a typed, validated ToolCall instance. Structural validity is guaranteed; semantic correctness remains the LLM's responsibility. The clarify tool is a member of the registered space, giving the model an explicit affordance to request user disambiguation rather than guess.

Consequences

Positive: - Structural validity of LLM tool-call output is guaranteed — an entire class of agent failures is eliminated by construction. - Single source of truth for the tool space: one registry, two compiler paths. - Small models (3B–8B parameters) benefit disproportionately from constrained decoding relative to larger models. - Adding a new tool requires one schema definition and one dispatcher function; grammar regeneration is automatic.

Negative / trade-offs: - Pydantic v2 → JSON Schema → GBNF round-trip fidelity is imperfect for complex types (recursive definitions, advanced unions). v1.0 tool schemas are constrained to a simple subset: primitives, nested simple records, and finite unions. - Outlines is under active development; semver pinning is required. - llama.cpp GBNF syntax evolves; the integration tests must pin a llama.cpp release and gate upgrades on passing tests. - Complex grammar compilation can take seconds on first load; grammars are compiled once at daemon startup and cached in RAM.

Implementation

The ToolSchema registry and compiler are in yazses-core/src/tools/. The GBNF compiler path is in yazses-llm-llamacpp/src/grammar.rs. The Outlines path runs inside the embedded Python plugin layer (see ADR-009) in yazses_plugins/tools/outlines_compiler.py. Dispatcher logic is in yazses-core/src/dispatch.rs.