When to Call the LLM: Adaptive Speech Interfaces for NL Input in XR (ACM VRST 2025)
Summary¶
This ACM VRST 2025 paper evaluates four pipeline architectures for voice → 3D object selection in virtual reality: (1) classic NLP, (2) LLM-assisted parsing, (3) end-to-end LLM, and (4) adaptive hybrid routing. [EVIDENCE src-010] The best result is achieved by the hybrid approach, which routes simple or unambiguous commands to a fast local NLP parser (<50 ms) and complex or ambiguous commands to an LLM (~800 ms). [EVIDENCE src-010] The key insight for YazSes is that an LLM is only needed for ambiguous or complex commands — simple dictation and command dispatch can remain local and fast, with LLM involvement triggered selectively.
Key Findings¶
- Four pipeline architectures tested: (1) classic NLP rule parser, (2) LLM-assisted NLP (LLM provides entity extraction to a rule parser), (3) end-to-end LLM processes raw transcript, (4) adaptive hybrid router. [EVIDENCE src-010]
- Adaptive hybrid wins: The hybrid approach achieves the best accuracy-latency trade-off by routing simple/unambiguous commands locally and escalating to LLM only for ambiguous cases. [EVIDENCE src-010]
- Latency figures: Fast NLP parser path: <50 ms; LLM path: ~800 ms. [EVIDENCE src-010] The hybrid keeps the median latency low while handling edge cases correctly.
- Accuracy-latency sweet spot: Neither always-fast (low accuracy on ambiguous commands) nor always-LLM (slow) is optimal. [EVIDENCE src-010] The routing decision itself requires a lightweight classifier that adds minimal overhead.
- VR object selection context: The study uses 3D object selection as the task — "pick up the red cube", "move it left", "put it next to the table". [EVIDENCE src-010] Command ambiguity arises from spatial context and pronoun resolution, not from dictation.
- Generalisability: The hybrid routing principle generalises beyond VR. [HYPOTHESIS] For YazSes: simple commands ("new line", "delete that", "comma") route to local rule dispatch; complex commands ("rewrite this paragraph more formally") route to LLM.
Relevance to YazSes Innovation Domains¶
| Domain | Relevance | Notes |
|---|---|---|
| Streaming ASR | MEDIUM | Routing happens post-ASR; partial streaming can feed the router in real time |
| SSH/remote | LOW | Hybrid routing is a local concern; remote context does not change the routing logic |
| Code awareness | HIGH | Hybrid routing is the optimal architecture for code commands: simple operations stay fast, complex refactors go to LLM |
| Accessibility | HIGH | Fast local path keeps latency low for high-frequency accessibility commands; LLM handles novel/complex requests |
| AR/VR/XR | HIGH | Direct experimental context; validates hybrid routing for XR at production quality |
| Gaming | HIGH | Fast NLP path (<50 ms) meets real-time gaming requirements; LLM path handles complex quest/strategy commands |
Gaps Surfaced¶
- gap-hybrid-1: The routing classifier itself must be lightweight enough to add negligible latency. The paper does not specify the classifier architecture or its own inference cost. [TODO: find source]
- gap-hybrid-2: The study evaluates spatial object selection in VR, not text dictation or code editing. The ambiguity signals that trigger LLM escalation may differ substantially in a dictation context.
- gap-hybrid-3: The ~800 ms LLM path latency is measured under study conditions; local LLM inference on a developer laptop (e.g. via Ollama) may differ significantly depending on model size and hardware. [TODO: find source]
- gap-hybrid-4: The study does not address how the hybrid router handles the case where the fast NLP parser returns low-confidence results — a threshold-based escalation mechanism is implied but not specified.
- gap-hybrid-5: No evaluation of the hybrid approach under streaming (partial) transcription input — the paper assumes complete utterances are available before routing.