Skip to content

A Systematic Review of Fusion Methods for the User-Centered Design of Multimodal Interfaces

Source Summary

This ACM ICMI 2025 systematic review examines 50 multimodal interface systems that combine speech with spatially expressive modalities (gesture, touch, pen, gaze) from a User-Centered Design perspective. It is the most authoritative recent synthesis of fusion methods for voice+X interfaces. The review classifies systems by fusion timing (early, late, hybrid), fusion mechanism (rule-based, learned, LLM-mediated), input modality combination, and task domain. Key findings include: LLM-mediated late fusion is the emerging trend, replacing rule-based fusion; user correction of fusion errors is under-studied; and the combination of speech+gaze is the most studied pairing for desktop and XR contexts.

Key Claims

  • [EVIDENCE src-010] Systematic review of 50 speech + gesture/touch/pen/gaze systems from a User-Centered Design perspective.
  • [EVIDENCE src-010] LLM-mediated fusion is the emerging trend, replacing earlier rule-based and learned-weight fusion approaches.
  • [EVIDENCE src-010] Speech+gaze is the most studied combination for desktop and XR contexts.
  • [EVIDENCE src-010] User correction of multimodal fusion errors is identified as an under-studied area — current systems have poor graceful degradation when fusion fails.
  • [HYPOTHESIS] The dominance of late fusion (modalities processed independently, combined at decision level) over early fusion (raw signal concatenation) in voice+gaze systems is because gaze and voice have very different temporal scales and signal characteristics that resist joint encoding.
  • [EVIDENCE src-010] The review covers systems from instruction-based interaction contexts — aligned with the YazSes use case of issuing commands.

Technical Mechanisms

Architecture / System Design

[EVIDENCE src-010] Three fusion architectures identified: (1) early fusion — raw features from both modalities concatenated before model; (2) late fusion — independent model per modality, outputs combined by voting or weighting rule; (3) hybrid fusion — late fusion with learned weights or LLM-mediated resolution. The LLM-mediated approach (see also src-004) is classified as a form of late fusion where the LLM receives symbolic descriptions of each modality's interpretation and resolves them semantically.

Core Algorithms or Methods

[EVIDENCE src-010] Rule-based late fusion: if voice command contains a deictic ("this", "that", "here"), resolve the referent from the gaze or gesture channel; otherwise use voice command alone. This is the most common pattern in the 50 reviewed systems. [HYPOTHESIS] For YazSes, a simpler version of this rule: if transcript contains a deictic pronoun, append the OS context (active window + cursor position) to the intent classifier prompt; otherwise skip OS context injection. This avoids LLM overhead for non-deictic commands.

Key Data Structures or APIs

[EVIDENCE src-010] Common interface pattern: voice command is the primary channel; other modalities are secondary/context channels that refine the voice interpretation. This asymmetric design is important: it means voice always has a valid interpretation (inject text or execute command), and the secondary modalities only improve it — they never block it if unavailable.

Identified Gaps

Stated Limitations

  • [EVIDENCE src-010] The review covers instruction-based interaction primarily; conversational (dialogue) multimodal interaction is out of scope.
  • [EVIDENCE src-010] User error correction in multimodal fusion is identified as a gap — when fusion produces the wrong command, how does the user recover?

Inferred Gaps

  • [HYPOTHESIS] No reviewed system addresses the case where only one modality is available (e.g., voice-only for motor-impaired users) and graceful degradation to voice-only is the goal — the accessibility case is not the focus of this review.
  • [HYPOTHESIS] Privacy implications of gaze tracking as a continuous context signal are not addressed in any reviewed system.

Evaluation / Coverage Gaps

  • [EVIDENCE src-010] Most reviewed systems report task completion rate; end-to-end latency and cognitive load are rarely measured.

Applicability to Project

[EVIDENCE src-010] The asymmetric design pattern (voice = primary, gaze/gesture = context refinement) directly informs YazSes' multimodal architecture. [HYPOTHESIS] YazSes should implement the OS context layer as a secondary, optional refinement channel: if OS context is available (active window, selection), it enriches the intent classifier; if not available, the pipeline falls back to voice-only interpretation. This preserves backward compatibility and doesn't break the existing user experience.

[EVIDENCE src-010] The identification of user error correction as an under-studied gap maps to a concrete YazSes feature: a correction protocol where the user can say "undo that" or "that was wrong" after a misinterpreted command, and the daemon selects the second-best intent from its classification n-best list. [HYPOTHESIS] The existing self-correction rollback in the disfluency filter provides the infrastructure for this — extending it to intent-level corrections would close a gap the review identifies across 50 systems.

Notes for SoA Matrix

Covers: - multimodal_fusion — authoritative: systematic review of 50 speech+X fusion systems - ambient_context_awareness — gaze and gesture as context signals for voice disambiguation

Partially covers (~): - xr_spatial_voice — some reviewed systems are XR; not the exclusive focus - code_voice_coding — instruction-based tasks are adjacent to code commands

Does not cover (✗): - silent_speech_biosignal — not in scope (acoustic speech only) - emotional_voice_gaming — not in scope (task-oriented, not expressive) - llm_intent_routing_offline — LLM-mediated fusion discussed but offline not addressed - accessibility_aac — under-studied, identified as gap - offline_edge_deployment — not addressed