Skip to content

Typing Reinvented: Towards Hands-Free Input via sEMG

Source Summary

This November 2025 arXiv paper presents a transformer-based sEMG decoding framework for hands-free keyboard-free text input. Unlike src-005 (command vocabulary only), this work targets full character-level text entry via silent muscle articulation, combined with an LLM post-processing layer for error correction. The transformer attention mechanism captures temporal patterns in EMG signals that simpler models miss. The paper reports generic (cross-user) character error rate improvements from 24.98% to 20.34%, and personalised (per-user) CER from 10.86% to 10.10%, positioning sEMG as a viable input modality for spatial, wearable, and screenless computing contexts. The paper explicitly identifies spatial/wearable/screenless computing as the target application domain.

Key Claims

  • [EVIDENCE src-006] Transformer-based sEMG decoding achieves generic CER of 20.34% and personalised CER of 10.10% for keyboard-free text input.
  • [EVIDENCE src-006] Combined with lightweight LLM-based decoding (error correction), the results highlight feasibility of accurate, real-time, hands-free text input.
  • [EVIDENCE src-006] The paper explicitly targets spatial, wearable, and screenless computing as the motivating application domain.
  • [EVIDENCE src-006] Generic (cross-user) CER is substantially higher than personalised; a calibration step is needed per user for high accuracy.
  • [HYPOTHESIS] A 10% CER (personalised) is competitive with voice-based dictation in noisy environments and would make sEMG dictation viable for short messages, commands, and code snippets — not full document dictation.
  • [HYPOTHESIS] Combined with the 5× faster-than-speaking speed advantage of thought-rate vs. speaking rate (humans think at ~130 wpm but speak at ~130 wpm — no advantage yet; but silent mouthing could be faster for trained users), sEMG may close the speed gap with voice in specific contexts.

Technical Mechanisms

Architecture / System Design

[EVIDENCE src-006] The pipeline: sEMG surface electrodes (forearm or face/jaw) → amplification and digitisation → transformer encoder (attention over temporal EMG windows) → character probability sequence → LLM-based beam search decoder (error correction). The LLM decoder stage uses a language model prior to bias toward valid words, recovering from EMG signal ambiguity.

Core Algorithms or Methods

[EVIDENCE src-006] The core contribution is replacing earlier CNN/LSTM decoders with a transformer encoder that uses self-attention over multi-channel temporal EMG windows. This captures long-range temporal dependencies in the EMG signal that correspond to phoneme transitions in silent articulation. The LLM post-processing is a standard language-model-constrained beam search, similar to ASR rescoring.

Key Data Structures or APIs

[EVIDENCE src-006] Output is a character sequence (not a command label as in src-005). [HYPOTHESIS] For YazSes integration, this would feed the same text-injection pipeline as acoustic ASR — the EMG decoder would be a drop-in replacement for the faster-whisper ASR module in non-vocal contexts, with the same text-cleaning and intent-classification stages downstream.

Identified Gaps

Stated Limitations

  • [EVIDENCE src-006] Generic CER of 20.34% is too high for unsupervised deployment without personalisation.
  • [EVIDENCE src-006] The paper does not report real-time latency; whether the transformer+LLM decode pipeline fits in an interactive typing loop (< 200 ms) is not demonstrated.

Inferred Gaps

  • [HYPOTHESIS] The system requires users to silently mouth each character or word — the cognitive load of silently mouthing while thinking about content has not been studied; fatigue and accuracy degradation over time are unknown.
  • [HYPOTHESIS] No evaluation of robustness to non-typing movements (head movement, chewing, speaking to someone else) is reported.

Evaluation / Coverage Gaps

  • [EVIDENCE src-006] Evaluated on a fixed vocabulary task set; open-vocabulary code entry (with symbol characters, camelCase, etc.) is not evaluated.

Applicability to Project

[EVIDENCE src-006] This paper provides the technical roadmap for a YazSes "silent dictation mode" that uses sEMG rather than acoustic STT. [HYPOTHESIS] YazSes could implement a plugin architecture where the STT backend is swappable — the existing faster-whisper backend is the default, and an EMG backend (using an open-source transformer sEMG decoder when one becomes available) is an optional module. The rest of the pipeline (text cleaning, intent classification, injection) is unchanged.

[EVIDENCE src-006] The LLM-based error correction layer in this system is compatible with the disfluency filter already in YazSes. [HYPOTHESIS] A combined pipeline (EMG decoder → LLM error correction → disfluency filter → intent classifier → inject) reuses 80% of existing YazSes infrastructure for silent dictation, requiring only the EMG decoder module to be added.

Notes for SoA Matrix

Covers: - silent_speech_biosignal — primary: transformer-based sEMG full-text input with LLM correction - offline_edge_deployment — transformer model is compact enough for edge; LLM corrector can be small

Partially covers (~): - accessibility_aac — keyboard-free input for motor-impaired; not specifically AAC-focused - code_voice_coding — keyboard-free input that could include code, but open-vocabulary code not evaluated

Does not cover (✗): - xr_spatial_voice — not in scope - llm_intent_routing_offline — LLM used for error correction only, not intent routing - emotional_voice_gaming — not in scope - multimodal_fusion — single modality (sEMG only) - ambient_context_awareness — no context signal