Skip to content

Wave L — non-speech & prosodic voice interaction (SoA research)

Date: 2026-07-02 · Tier: design/ — public engineering research · Author: Mohsen Seyedkazemi Ardebili Companion: fed adr-v2-095 through adr-v2-104. See the waves index.

A snapshot of the field on the date above, kept as the research record behind the ADRs it fed. Feature status should be checked against yazses features and the linked ADRs, not this note.

Method: background SoA scout (2024-2026 CHI/UIST/ASSETS/TOCHI, Interspeech/ICASSP, arXiv cs.HC/cs.CL/eess.AS; Apple/MS/Google accessibility) against the full feature-exclusion list at the time. Every candidate is on-device/offline, off by default, with a pure dependency-light testable core (heavy DSP/models are optional lazy extras). All ten evidence anchors were checked against real research/products at the time.

This wave opened an area the prior net-new features hadn't touched: non-speech vocal signals (formant/pitch/loudness/breath) and acoustic prosody as first-class interaction channels — for users who can phonate but not articulate discrete words, and for word-free eyes-free control.

The ten features (ranked most→least compelling)

  1. Vocal Joystick — sustain vowels/pitch/loudness for continuous analog cursor/scroll control (no words). Core: vowel_to_direction, vocal_control_vector, VocalJoystick state machine. Anchor: Bilmes et al., The Vocal Joystick (ASSETS 2006; D&RAT 2008, PMID 18416516). → adr-v2-095.
  2. Mouth-Sound Switch Access — non-verbal mouth sounds (pop/click/cluck) drive a timed scan-and-select. Core: classify_mouth_sound, ScanSelector. Anchor: Apple Sound Actions for Switch Control (iOS 15, 2021). → adr-v2-097.
  3. Beam-Steered Spatial VAD — 2-mic direction-of-arrival gate (enrollment-free, geometric, complements Cocktail Filter). Core: gcc_phat, tdoa_to_angle, spatial_gate. Anchor: Knapp & Carter GCC-PHAT; real-time DoA (PMC8136617). → adr-v2-098.
  4. Breath-Paced Dictation — inhalation-onset segmentation into breath groups. Core: breath_envelope, detect_breath_onsets, segment_by_breath. Anchor: BreathPrint (MobiSys 2017); in-ear breathing-phase (Sensors 2024, 24(20):6679). → adr-v2-099.
  5. Whisper-Aware Mode — detect whispered phonation, adapt gain/VAD/prompt. Core: voicing_ratio, spectral_tilt, is_whispered, whisper_adaptation. Anchor: Ito et al. (ICASSP 2005); wTIMIT; arXiv 2408.13746 (2024). → adr-v2-100.
  6. Earcon Feedback Language — structured non-speech tones for daemon state (eyes-free, faster than read-back). Core: earcon_for, render_earcon. Anchor: Brewster earcons (TOCHI 1993); Gaver auditory icons (1986); Walker spearcons (Human Factors 2013). → adr-v2-096.
  7. Hesitation-Hold Endpointing — hold the turn open on filled pauses ("uhh…") instead of cutting off. Core: is_filled_pause, endpoint_decision. Anchor: Chatziagapi et al. filled-pause detection (ACII 2022). → adr-v2-101.
  8. Involuntary-Vocalization Auto-Excision — delete cough/throat-clear/sneeze from the stream. Core: is_involuntary_vocalization, excise_nonspeech_spans. Anchor: Hyfe cough tracker (F1000Research 11:730; PMC11809693, 91%/98%). → adr-v2-102.
  9. Pitch-Contour Vocal Gestures — hum a melody shape (rise=confirm, fall=cancel) as a word-free command grammar. Core: normalize_contour, classify_contour, gesture_to_command. Anchor: Vocal Joystick line + Meyer, Whistled Languages (Springer 2015). → adr-v2-103.
  10. Prosodic Auto-Punctuation — insert . , ? from prosody alone (no spoken punctuation words). Core: punctuate_from_prosody. Anchor: Cho et al., Leveraging Prosody for Punctuation Prediction (Interspeech 2022, UW). → adr-v2-104.

Diversity

2 non-speech motor/AAC (#1, #2), 3 robustness/DSP (#3, #4, #5), 1 eyes-free feedback (#6), 2 timing/turn-taking (#7, #9 — distinct: analog control vs discrete symbolic), 2 health/editing (#8,

10). Every "heart" is a deterministic function/state-machine over already-extracted features;

formant/F0/DoA extraction is always the optional lazy extra.

Citations here have not been re-verified against research/verify_refs.py.