## Source Summary This ACM ICMI 2025 systematic review examines 50 multimodal interface systems that combine speech with spatially expressive modalities (gesture, touch, pen, gaze) from a User-Centered Design perspective. It is the most authoritative recent synthesis of fusion methods for voice+X interfaces. The review classifies systems by fusion timing (early, late, hybrid), fusion mechanism (rule-based, learned, LLM-mediated), input modality combination, and task domain. Key findings include: LLM-mediated late fusion is the emerging trend, replacing rule-based fusion; user correction of fusion errors is under-studied; and the combination of speech+gaze is the most studied pairing for desktop and XR contexts. --- ## Key Claims - [EVIDENCE src-010] Systematic review of 50 speech + gesture/touch/pen/gaze systems from a User-Centered Design perspective. - [EVIDENCE src-010] LLM-mediated fusion is the emerging trend, replacing earlier rule-based and learned-weight fusion approaches. - [EVIDENCE src-010] Speech+gaze is the most studied combination for desktop and XR contexts. - [EVIDENCE src-010] User correction of multimodal fusion errors is identified as an under-studied area — current systems have poor graceful degradation when fusion fails. - [HYPOTHESIS] The dominance of late fusion (modalities processed independently, combined at decision level) over early fusion (raw signal concatenation) in voice+gaze systems is because gaze and voice have very different temporal scales and signal characteristics that resist joint encoding. - [EVIDENCE src-010] The review covers systems from instruction-based interaction contexts — aligned with the YazSes use case of issuing commands. --- ## Technical Mechanisms ### Architecture / System Design [EVIDENCE src-010] Three fusion architectures identified: (1) early fusion — raw features from both modalities concatenated before model; (2) late fusion — independent model per modality, outputs combined by voting or weighting rule; (3) hybrid fusion — late fusion with learned weights or LLM-mediated resolution. The LLM-mediated approach (see also src-004) is classified as a form of late fusion where the LLM receives symbolic descriptions of each modality's interpretation and resolves them semantically. ### Core Algorithms or Methods [EVIDENCE src-010] Rule-based late fusion: if voice command contains a deictic ("this", "that", "here"), resolve the referent from the gaze or gesture channel; otherwise use voice command alone. This is the most common pattern in the 50 reviewed systems. [HYPOTHESIS] For YazSes, a simpler version of this rule: if transcript contains a deictic pronoun, append the OS context (active window + cursor position) to the intent classifier prompt; otherwise skip OS context injection. This avoids LLM overhead for non-deictic commands. ### Key Data Structures or APIs [EVIDENCE src-010] Common interface pattern: voice command is the primary channel; other modalities are secondary/context channels that refine the voice interpretation. This asymmetric design is important: it means voice always has a valid interpretation (inject text or execute command), and the secondary modalities only improve it — they never block it if unavailable. --- ## Identified Gaps ### Stated Limitations - [EVIDENCE src-010] The review covers instruction-based interaction primarily; conversational (dialogue) multimodal interaction is out of scope. - [EVIDENCE src-010] User error correction in multimodal fusion is identified as a gap — when fusion produces the wrong command, how does the user recover? ### Inferred Gaps - [HYPOTHESIS] No reviewed system addresses the case where only one modality is available (e.g., voice-only for motor-impaired users) and graceful degradation to voice-only is the goal — the accessibility case is not the focus of this review. - [HYPOTHESIS] Privacy implications of gaze tracking as a continuous context signal are not addressed in any reviewed system. ### Evaluation / Coverage Gaps - [EVIDENCE src-010] Most reviewed systems report task completion rate; end-to-end latency and cognitive load are rarely measured. --- ## Applicability to Project [EVIDENCE src-010] The asymmetric design pattern (voice = primary, gaze/gesture = context refinement) directly informs YazSes' multimodal architecture. [HYPOTHESIS] YazSes should implement the OS context layer as a secondary, optional refinement channel: if OS context is available (active window, selection), it enriches the intent classifier; if not available, the pipeline falls back to voice-only interpretation. This preserves backward compatibility and doesn't break the existing user experience. [EVIDENCE src-010] The identification of user error correction as an under-studied gap maps to a concrete YazSes feature: a correction protocol where the user can say "undo that" or "that was wrong" after a misinterpreted command, and the daemon selects the second-best intent from its classification n-best list. [HYPOTHESIS] The existing self-correction rollback in the disfluency filter provides the infrastructure for this — extending it to intent-level corrections would close a gap the review identifies across 50 systems. --- ## Notes for SoA Matrix **Covers:** - `multimodal_fusion` — authoritative: systematic review of 50 speech+X fusion systems - `ambient_context_awareness` — gaze and gesture as context signals for voice disambiguation **Partially covers (~):** - `xr_spatial_voice` — some reviewed systems are XR; not the exclusive focus - `code_voice_coding` — instruction-based tasks are adjacent to code commands **Does not cover (✗):** - `silent_speech_biosignal` — not in scope (acoustic speech only) - `emotional_voice_gaming` — not in scope (task-oriented, not expressive) - `llm_intent_routing_offline` — LLM-mediated fusion discussed but offline not addressed - `accessibility_aac` — under-studied, identified as gap - `offline_edge_deployment` — not addressed