The HCI canon behind YazSes¶

Most writing about voice input is about models — word error rate, parameters, benchmarks. YazSes is a bet that the harder problems are interaction problems, and that most of them were studied and answered before the current generation of speech models existed.

This page is the reading list behind that bet: the papers that decide how a dictation tool should behave, what each one establishes, and where it shows up in the software. It is deliberately opinionated — a reference earns a place only if it changes a design decision.

The argument in eight claims¶

Underneath all eight is the field's founding move: treat an input channel as something with a measurable capacity, not a feature list. That is Fitts, and the tradition of costing an interaction in time and errors that followed it (Card, Moran & Newell), together with the principle that the interface should be a directly manipulable representation rather than a language you have to learn (Shneiderman).

1. Speech is fast, but correction is what you actually pay¶

Measured head-to-head, speech entry is roughly three times faster than touchscreen typing (Ruan et al. — 153 vs 52 WPM in English). That number is also the most over-quoted in the field, because dictation is not transcription: it is entry interleaved with correction, and correction consumes a large share of the session (Karat et al.). Repairing an error by switching modality beats repeating yourself (Suhm et al.) — which is why YazSes gives you a command grammar and a keyboard, not just a retry. Trying to do the repair by voice is where dictation gets genuinely awkward, and that difficulty has been documented for two decades (Sears et al.). None of this is newly noticed: the open problems in intelligent text entry were catalogued in 2009 (Kristensson & Jameson) and most are still open.

The honest limit: speech competes with composition for the same mental resources in a way typing does not (Shneiderman). Dictation is not a universal replacement for a keyboard and anyone selling it as one is overselling.

2. Hold-to-talk is a mode design, and the mode literature already picked the winner¶

Recording only while a key is held is the single most consequential decision in YazSes. It is not a UX preference — it is the finding that kinaesthetic feedback prevents mode errors where visual feedback fails (Sellen, Kurtenbach & Buxton). A mode you are physically holding cannot be silently forgotten; a toggle can, and then it records your meeting. Raskin named this a quasimode (Raskin), and Buxton's account of input phrasing treats sustained tension as the natural boundary of one utterance (Buxton). Where a mode must exist, making it perceptible measurably reduces the errors it causes (Monk) — which is what the tray's colour states are for.

3. The remaining error is a slip, so the system should absorb it¶

Dictating into a window that accepts no text is a slip in Norman's sense (Norman) — the error of a competent user, not a confused one. The design response is not a warning; it is to make the error recoverable. YazSes detects the absence of a text target and diverts the transcript to the clipboard rather than typing it into the void. That is the gulf-of-evaluation problem (Hutchins, Hollan & Norman) solved with a tray colour.

4. Gaze picks a window; it will never pick a character¶

Webcam gaze lands around 2–4° — centimetres on a screen. That is enough to know which window you mean and never which character (Sugano et al. measured 2.9° with implicit calibration from mouse clicks; Zhang et al., Krafka et al. set the wider accuracy regime). Two older results fix the design. The Midas touch problem — your eyes are always somewhere, so gaze alone cannot express intent (Jacob) — is why YazSes commits on the held hotkey and never on dwell time. MAGIC pointing (Zhai, Morimoto & Ihde) is the division of labour: gaze selects coarsely, a manual action confirms.

Gaze as text entry has a measured ceiling that settles the division of labour: adjustable-dwell eye typing reached 19.9 WPM after ten sessions of practice (Majaranta et al.) — far below speech. So gaze routes, and speech types.

The ancestor of all of it is Bolt's Put-That-There: speech carries the verb, pointing carries the referent. YazSes's "close this" is that idea on a $0 sensor, 45 years later.

5. Multimodal input is not simultaneous, and it is not always on¶

Users go multimodal under cognitive load, not uniformly (Oviatt et al.), and the modalities rarely arrive at the same instant (Oviatt) — the timing tolerance matters more than the fusion algorithm (Kaur et al. measured exactly this for gaze+speech). Any feature that assumes "look and talk at once" is designing for a user who does not exist.

6. Muscle and brain interfaces are triggers, not typewriters¶

The best non-invasive result handwrites at 20.9 WPM (Kaifosh et al., Nature 2025) — a seventh of speech. Open-vocabulary silent speech sits near 68% WER (Gaddy & Klein). The fast, accurate paths are all invasive (Willett et al., Card et al.). The honest split is therefore silent commands plus spoken prose — the muscle carries the intent to speak, the voice carries the words. The field has understood itself this way for fifteen years (Denby et al.); what changed recently is the invasive ceiling, not the wearable one.

7. Adaptation that the user cannot see gets rejected¶

YazSes's personalization loop proposes configuration changes for approval instead of applying them silently. That follows mixed-initiative principles (Horvitz) and a specific finding: users of adaptive interfaces value predictability alongside accuracy, so an unannounced improvement can still make the interaction worse (Gajos et al.). Explanations measurably help (Lim, Dey & Avrahami), and setting expectations changes acceptance of an imperfect system more than raw accuracy does (Kocielnik et al.). The consolidated human-AI interaction guidelines (Amershi et al.) are the checklist we audit the optional features against.

8. Privacy is a property of the interaction, not a checkbox¶

The useful frame is contextual integrity (Nissenbaum): privacy is violated when information flows outside the context that produced it, which is why an architecture that never transmits is a stronger guarantee than any consent dialog. This is not only principle — privacy concern is a documented reason people decline voice assistants, and users are frequently unaware that recordings are retained (Lau, Zimmerman & Schaub). And because voice interfaces are used in shared, social settings rather than by isolated individuals (Porcheron et al.), the people most exposed are often bystanders who were never asked — the exact situation meeting capture creates.

What this literature says we have not done¶

The most useful thing about assembling a canon is what it exposes. Three gaps, stated plainly:

1. No user study, ever. YazSes publishes word error rate and latency. It has never measured task-level throughput or correction cost with a human participant. In Wobbrock & Kientz's taxonomy (here) it is an artifact contribution missing its empirical counterpart. The method is not an open question — MacKenzie & Soukoreff and Soukoreff & MacKenzie define how to run and report it, and Vertanen & Kristensson is the closest comparable system.
2. The accessibility features are unevaluated with the people they name. The design philosophy YazSes follows here is ability-based design — adapt the system to the abilities a person has, rather than requiring them to conform (Wobbrock et al.). YazSes ships a dysfluency-tolerant mode. Lea et al. show how recognition fails people who stutter and what fixes it; SEP-28k is a public dataset that exists today. Green et al. show personalized models can beat human listeners on short phrases of disordered speech. We have the loop and have not run it with affected speakers — and the disability-studies critique of assistive technology (Mankoff, Hayes & Kasnitz) is the reason that has to mean with, not for.
3. False-activation rate per working day is unpublished by anyone. Every trigger modality reports accuracy on a benchmark; none reports how many times it fires wrongly across a real workday. It is the number that decides whether you can leave a thing switched on.

Whichever of these runs first should be preregistered — it is a single-shot claim about our own software, which is exactly the situation where analysis flexibility quietly becomes a result (Cockburn, Gutwin & Dix).

References¶

Evidence grade: measured (peer-reviewed measurement), secondary (survey, review, or position piece), vendor (claimed by the maker). Where a specific number appears above, it is one we have checked at source or relayed from a page that did.

1. Fitts, P. M. "The information capacity of the human motor system in controlling the amplitude of movement." Journal of Experimental Psychology 47(6), 1954. doi:10.1037/h0055392 — measured
2. Card, S. K., Moran, T. P., Newell, A. "The keystroke-level model for user performance time with interactive systems." Communications of the ACM 23(7), 1980. doi:10.1145/358886.358895 — measured
3. Shneiderman, B. "Direct manipulation: a step beyond programming languages." IEEE Computer 16(8), 1983. doi:10.1109/MC.1983.1654471 — secondary
4. Hutchins, E. L., Hollan, J. D., Norman, D. A. "Direct manipulation interfaces." Human-Computer Interaction 1(4), 1985. doi:10.1207/s15327051hci0104_2 — secondary
5. Norman, D. A. "Categorization of action slips." Psychological Review 88(1), 1981. doi:10.1037/0033-295X.88.1.1 — secondary
6. Sellen, A. J., Kurtenbach, G. P., Buxton, W. A. S. "The prevention of mode errors through sensory feedback." Human-Computer Interaction 7(2), 1992. doi:10.1207/s15327051hci0702_1 — measured
7. Raskin, J. The Humane Interface: New Directions for Designing Interactive Systems. Addison-Wesley, 2000. Book excerpt: doi:10.1145/341836.342022 — secondary
8. Buxton, W. "Chunking and phrasing and the design of human-computer dialogues." Proc. IFIP World Computer Congress, 1986. Reprinted in Readings in Human-Computer Interaction, 1995. doi:10.1016/b978-0-08-051574-8.50051-0 — secondary
9. Monk, A. "Mode errors: a user-centred analysis and some preventative measures using keying-contingent sound." International Journal of Man-Machine Studies 24(4), 1986. doi:10.1016/S0020-7373(86)80049-9 — measured
10. Ruan, S., Wobbrock, J. O., Liou, K., Ng, A., Landay, J. A. "Comparing speech and keyboard text entry for short messages in two languages on touchscreen phones." arXiv:1608.07323, 2016. arXiv — measured (153 vs 52 WPM, English)
11. Karat, C.-M., Halverson, C., Horn, D., Karat, J. "Patterns of entry and correction in large vocabulary continuous speech recognition systems." CHI '99, 1999. doi:10.1145/302979.303160 — measured
12. Sears, A., Feng, J., Oseitutu, K., Karat, C.-M. "Hands-free, speech-based navigation during dictation: difficulties, consequences, and solutions." Human-Computer Interaction 18(3), 2003. doi:10.1207/S15327051HCI1803_2 — measured
13. Suhm, B., Myers, B., Waibel, A. "Multimodal error correction for speech user interfaces." ACM TOCHI 8(1), 2001. doi:10.1145/371127.371166 — measured
14. Shneiderman, B. "The limits of speech recognition." Communications of the ACM 43(9), 2000. doi:10.1145/348941.348990 — secondary
15. Vertanen, K., Kristensson, P. O. "Parakeet: a continuous speech recognition system for mobile touch-screen devices." IUI '09, 2009. doi:10.1145/1502650.1502685 — measured
16. Kristensson, P. O., Jameson, A. "Five challenges for intelligent text entry methods." AI Magazine 30(4), 2009. doi:10.1609/aimag.v30i4.2269 — secondary
17. Oviatt, S. "Ten myths of multimodal interaction." Communications of the ACM 42(11), 1999. doi:10.1145/319382.319398 — secondary
18. Oviatt, S., Coulston, R., Lunsford, R. "When do we interact multimodally? Cognitive load and multimodal communication patterns." ICMI '04, 2004. doi:10.1145/1027933.1027957 — measured
19. Bolt, R. A. "Put-that-there: voice and gesture at the graphics interface." SIGGRAPH '80, 1980. doi:10.1145/800250.807503 — secondary
20. Kaur, M. et al. "Where is 'it'? Event synchronization in gaze-speech input systems." ICMI '03, 2003. doi:10.1145/958432.958463 — measured
21. Jacob, R. J. K. "What you look at is what you get: eye movement-based interaction techniques." CHI '90, 1990. doi:10.1145/97243.97246 — measured
22. Zhai, S., Morimoto, C., Ihde, S. "Manual and gaze input cascaded (MAGIC) pointing." CHI '99, 1999. doi:10.1145/302979.303053 — measured
23. Majaranta, P., Ahola, U.-K., Špakov, O. "Fast gaze typing with an adjustable dwell time." CHI '09, 2009. doi:10.1145/1518701.1518758 — measured (6.9 → 19.9 WPM over ten sessions)
24. Sugano, Y., Matsushita, Y., Sato, Y., Koike, H. "Appearance-based gaze estimation with online calibration from mouse operations." IEEE Transactions on Human-Machine Systems 45(6), 2015. doi:10.1109/THMS.2015.2400434 — measured (2.9° with no explicit calibration)
25. Zhang, X., Sugano, Y., Fritz, M., Bulling, A. "Appearance-based gaze estimation in the wild." arXiv:1504.02863, 2015. arXiv — measured
26. Krafka, K. et al. "Eye tracking for everyone." arXiv:1606.05814, 2016. arXiv — measured
27. Kaifosh, P., Reardon, T. R. et al. "A generic non-invasive neuromotor interface for human-computer interaction." Nature, 2025. doi:10.1038/s41586-025-09255-w — measured (20.9 WPM handwriting)
28. Gaddy, D., Klein, D. "Digital voicing of silent speech." arXiv:2010.02960, 2020. arXiv — measured (open-vocabulary silent speech ≈68% WER)
29. Willett, F. R. et al. "A high-performance speech neuroprosthesis." Nature 620, 2023. doi:10.1038/s41586-023-06377-x — measured (invasive; 62 WPM, 23.8% WER at 125k words)
30. Card, N. S. et al. "An accurate and rapidly calibrating speech neuroprosthesis." New England Journal of Medicine, 2024. doi:10.1056/NEJMoa2314132 — measured (invasive; 2.5% WER)
31. Denby, B. et al. "Silent speech interfaces." Speech Communication 52(4), 2010. doi:10.1016/j.specom.2009.08.002 — secondary
32. Wobbrock, J. O., Kane, S. K., Gajos, K. Z., Harada, S., Froehlich, J. "Ability-based design: concept, principles and examples." ACM TACCESS 3(3), 2011. doi:10.1145/1952383.1952384 — secondary
33. Green, J. R. et al. "Automatic speech recognition of disordered speech: personalized models outperforming human listeners on short phrases." Interspeech 2021. doi:10.21437/Interspeech.2021-1384 — measured
34. Lea, C. et al. "From user perceptions to technical improvement: enabling people who stutter to better use speech recognition." CHI '23, 2023. doi:10.1145/3544548.3581224 — measured
35. Lea, C., Mitra, V., Joshi, A., Kajarekar, S., Bigham, J. P. "SEP-28k: a dataset for stuttering event detection from podcasts with people who stutter." arXiv:2102.12394, 2021. arXiv — measured
36. Mankoff, J., Hayes, G. R., Kasnitz, D. "Disability studies as a source of critical inquiry for the field of assistive technology." ASSETS '10, 2010. doi:10.1145/1878803.1878807 — secondary
37. Horvitz, E. "Principles of mixed-initiative user interfaces." CHI '99, 1999. doi:10.1145/302979.303030 — secondary
38. Gajos, K. Z., Everitt, K., Tan, D. S., Czerwinski, M., Weld, D. S. "Predictability and accuracy in adaptive user interfaces." CHI '08, 2008. doi:10.1145/1357054.1357252 — measured
39. Lim, B. Y., Dey, A. K., Avrahami, D. "Why and why not explanations improve the intelligibility of context-aware intelligent systems." CHI '09, 2009. doi:10.1145/1518701.1519023 — measured
40. Kocielnik, R., Amershi, S., Bennett, P. N. "Will you accept an imperfect AI? Exploring designs for adjusting end-user expectations of AI systems." CHI '19, 2019. doi:10.1145/3290605.3300641 — measured
41. Amershi, S. et al. "Guidelines for human-AI interaction." CHI '19, 2019. doi:10.1145/3290605.3300233 — secondary
42. Nissenbaum, H. Privacy in Context: Technology, Policy, and the Integrity of Social Life. Stanford University Press, 2010. doi:10.1515/9780804772891 — secondary
43. Lau, J., Zimmerman, B., Schaub, F. "Alexa, are you listening? Privacy perceptions, concerns and privacy-seeking behaviors with smart speakers." Proc. ACM Human-Computer Interaction 2(CSCW), 2018. doi:10.1145/3274371 — measured
44. Porcheron, M., Fischer, J. E., Reeves, S., Sharples, S. "Voice interfaces in everyday life." CHI '18, 2018. doi:10.1145/3173574.3174214 — measured
45. MacKenzie, I. S., Soukoreff, R. W. "Text entry for mobile computing: models and methods, theory and practice." Human-Computer Interaction 17(2–3), 2002. doi:10.1207/S15327051HCI172&3_2 — secondary
46. Soukoreff, R. W., MacKenzie, I. S. "Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric." CHI '03, 2003. doi:10.1145/642611.642632 — secondary
47. Wobbrock, J. O., Kientz, J. A. "Research contributions in human-computer interaction." Interactions 23(3), 2016. doi:10.1145/2907069 — secondary
48. Cockburn, A., Gutwin, C., Dix, A. "HARK no more: on the preregistration of CHI experiments." CHI '18, 2018. doi:10.1145/3173574.3173715 — secondary