Where YazSes goes next¶

There is no shortage of things voice interfaces could become. There is a shortage of honest sorting between the ones that would work on the laptop you already own and the ones that need a lab, a data centre, or a research result nobody has yet.

This page is that sorting. Every entry names the problem it answers, says which of three buckets it is in, and states what would move it.

How an idea gets here¶

The first gate does most of the work. YazSes ships 144 capabilities and 64 of them are not reachable from any entry point — the project's constraint has never been ideas, it has been finishing them. So an idea that cannot name the problem it answers does not get built, however good it sounds.

The second gate is ADR-011: it has to run offline, on a CPU, on a machine someone already owns. That is not modesty about ambition — it is the one property that distinguishes this project from every well-funded alternative, and a direction that needs a data centre would trade away the only durable advantage.

Buildable now — on an ordinary laptop¶

These need no new research and no new hardware. They are engineering.

1. Error cost, carried through the pipeline¶

Problem: A1. A their/there and an rm -rf /home/work are treated identically today — both are simply typed. Word error rate, the metric the whole field optimises, weights every word the same. The user never has.

What it looks like. The destination decides the confirmation. Dictating into a terminal, a payment field or a git command is not the same act as dictating into a chat box, and the pipeline already knows the difference: inject/target.py identifies the focused element, and the command safety gate already holds a destructive command pending a spoken confirm.

Why it is the strongest candidate. It generalises three features that were each built as point fixes — the safety gate, staged dictation, and the no-text-target guard — into one idea, and it produces a metric nobody publishes: cost-weighted error rate, errors scored by consequence rather than counted. That is a research contribution and a product improvement in the same change.

What is already there. cmdsafety/classify.py, staged/buffer.py, inject/target.py. What is missing. A destination-risk model and one confirmation policy instead of three.

2. A hands-free mode that is actually a mode¶

Problem: A3. You can dictate a paragraph but you cannot open the file, fix the fourth word, switch window or answer a dialog. Each of those exists as a designed, tested, unwired module — mousegrid, headpointer, pilot (AT-SPI), vocaljoystick, mouthswitch, hatselect.

Why it is not done. Every piece is individually unglamorous and only valuable in combination, which is exactly the shape a feature-driven roadmap never delivers. The measurement is also different: not word accuracy but task completion — can someone finish a real workflow with no keyboard and no mouse, and how much slower is it.

What would make it real. Wire three of them together and run one person through a scripted workflow. That is an afternoon of integration and an hour of observation, and it would produce the first evidence anyone has about this project's accessibility claims.

3. Composition, not transcription¶

Problem: A2. In the largest published sample of real correction behaviour — 266 correction utterances — 62% were re-dictations rather than edits, and what survives as commanding is almost entirely one-or-two-word deletion (Ghosh et al., 2020). People do not fix words; they restate thoughts. Yet every correction affordance in every dictation product is word-level.

What it looks like. Treat the utterance, not the word, as the unit a user revises. YazSes already has the pieces: timeline/history.py tracks whole-utterance injections, and "scratch that" already works on the last burst.

The honest caveat. This is where the measured streaming result bites. Streaming optimises time-to-first-word, and the benchmark shows it makes time-to-final-text worse on every model but tiny.en. Composition support has to be judged on the second number.

4. Structural targets by voice¶

Problem: A2 and A3. You can say "delete the last word" but not "delete that argument". Corrections are character- and word-shaped while the thing being edited has structure — so a single misrecognition inside a function call costs the whole utterance.

Where the idea comes from. Cursorless (1,334★) — one of several projects surveyed in the adjacent-projects study — gives spoken names to syntax-tree nodes, so you address the code's structure rather than its characters. It is the most interesting idea in the adjacent neighbourhood, and it survives all three gates.

What YazSes already has. hatselect is exactly this idea, designed and unwired; jump and the Neovim LSP bridge are the substrate that gives it a tree to address.

The honest caveat. Cursorless is an editor extension with the syntax tree in hand; YazSes is outside the editor looking in, and only the Neovim bridge supplies structure today. What transfers is the naming scheme, not the implementation — and outside a supported editor this degrades to what already exists.

Needs measurement first — a study, not a feature¶

These are plausible, and building them before measuring would produce a feature nobody can defend.

5. Code-switching as the real multilingual failure¶

Problem: A4. The assumption is that bilingual users struggle because of accent. The competing hypothesis is that they struggle because of code-switching — a recogniser locked to one language transcribes the other as noise, mid-sentence.

Status: open question. It is research issue #258 and polyglot/lid.py already implements the routing layer. What is missing is the WER comparison on code-switched versus monolingual utterances from the same speakers. Building the adapter first would be building on an assumption.

6. Uncertainty the user can see¶

Problem: A1, from the other side. If the recogniser knew it was unsure about a word, showing that is cheaper than any correction mechanism. Whether it helps is unmeasured: per-word confidence could reduce correction time, or it could add visual noise to every sentence and slow reading down.

Status: agenda question 5. confidence/ exists; the study does not.

Needs hardware, a lab, or a result nobody has¶

Real, and not reachable from a laptop. Listed so they are not rediscovered as if new.

7. Silent speech¶

Sub-vocal input — sEMG at the jaw and throat, reading articulation without audible sound. It is the genuinely transformative direction for both privacy and accessibility, and the public numbers are not there: no one has published a false-activation rate over a real working day, which is the number that decides whether it is usable at all (our measurement call).

What YazSes has. A working EMGBackend over the YESP serial protocol, and an activation-source seam that already treats a squeeze as equivalent to a keypress. The software is not the blocker; the evidence is.

8. Gaze that is precise enough to mean anything finer than a window¶

Webcam gaze is honest at 2–4°, which is centimetres on a screen — enough to know which window you mean, never which character. YazSes ships that honest version (gaze/, X11 only). Anything finer needs an eye tracker, not a webcam, and the deeper blocker is not optics: Wayland forbids one application from focusing another's window, so the capability is structurally unavailable on the display server most distributions now default to.

9. Personal adaptation that provably helps¶

A per-user speech adapter is the most-requested idea in this space and the easiest to fake — any fine-tune will look better on the data it was tuned on. The bar is a held-out WER win, which requires held-out data, which requires a corpus, which the user has to consent to build. The consent machinery exists (ADR-012, opt-in and encrypted). The evidence does not.

Deliberately not pursued¶

Recorded so the reasoning survives, rather than being relitigated.

Contributing a direction¶

There is a form for this: 🔭 Research idea or future direction. Its three questions are the three gates, so filling it in is the whole review.

The gates above are the whole review. An idea that names its problem, runs on a laptop and can be measured is welcome regardless of how strange it sounds — open an issue or add to the research agenda. An idea that cannot do those three things is not rejected, it is filed as a question, which is a different and often more useful thing.

References¶

1. Ghosh et al., ACM Transactions on Computer-Human Interaction 27:4 (2020). 266 real correction utterances: 62% re-dictation (mean 6.01 ± 3.88 words changed), ~29% command-based deletion (1.82 ± 1.74 words), ~7% replace, ~2% insert. Commanding survives almost entirely as one-or-two-word deletion.
2. Ruan, S., Wobbrock, J. O., Liou, K., Ng, A., Landay, J. A. "Comparing speech and keyboard text entry for short messages on two mobile devices." Proc. ACM IMWUT 1:4 (2016). — the 3× figure, measured on phones, not desktops.
3. See the full reference corpus for the 105 sources behind the research agenda, and benchmarks for every performance number quoted here.