Hugging Face × YazSes: multimodal HCI lab¶

Proposal dated 2026-10-27. Research roadmap, not shipped product functionality. Epic #692.

YazSes is already a multimodal system: local speech recognition, gaze-routing, facial gestures, shared camera perception and EMG activation are documented and partly implemented. Hugging Face can make its model artifacts, evaluation methods, documentation and demos reproducible. It must not become a dependency of core offline dictation or an excuse to add unsupported claims about brain control.

Work items¶

The programme is tracked under epic #692. Existing tasks cover static synthetic Space #693, model provenance #694, camera evaluation #695 and EEG/EOG feasibility #696. Additional bounded tasks handle publishing governance #698, synthetic fixtures #699, safe multimodal fusion replay #700 and independent HF privacy preflight #701.

See the phased engineering roadmap for dependencies, milestones and acceptance criteria. Prototypes, published HF accounts/Spaces and validated biosignal capabilities remain proposed, not delivered.

Five practical HF roles¶

Hardware pricing/eligibility changes. Start from free static HTML/JS Space rather than assuming free hosted Python inference or GPU. Run any expensive analysis locally in a separate, opt-in researcher process. HF is a model registry and distribution surface, not a mandatory cloud inference provider.

Existing internal seams to reuse¶

• src/yazses/stt/ includes the pluggable STT engine and faster_whisper.py cache-first model loading. Android has a separate checksum/pinned-revision issue #683.
• src/yazses/perception/ defines SharedPerceptionSource, PerceptionSample and derived gaze/head/face signals (ADR-v2-145 and eye programme). Never open a second webcam or expose raw frames to general feature consumers.
• src/yazses/gaze/ already supports MediaPipe FaceLandmarker; improve calibration and test real hardware rather than claiming pixel-accurate eye typing.
• src/yazses/platform/emg/ includes YESP and BrainFlow activation sources and graded squeeze logic; the hardware breadth proposal #103 is closed.
• design/emg-protocol.md, intent-source issue #137, external activation contract #139 and research review define the right boundary for biosignal experiments.
• Interaction Kernel and grounded targets supply explicit uncertainty, authority, abstention, confirmation and outcome verification.

Camera and gaze: evidence before model churn¶

Baseline: existing OpenCV + MediaPipe FaceLandmarker, one camera owner. Candidates for a separately evaluated offline backend include ONNX Runtime for a properly licensed gaze/head-pose model; PyTorch only as an optional research dependency, not the default runtime. A Hugging Face model's presence is not evidence of accuracy or license suitability.

Use a calibration dataset with device, resolution, lighting, screen geometry, distance, glasses, head pose, participant-specific baseline, latency and dropped-frame rate. Metrics: median and 95^th-percentile angular error; window-selection correctness; target abstention rate; false activations/hour; end-to-end action latency; CPU/RAM; camera power; accessibility recovery. Split by participant and device, never by adjacent frames (which leak identities and conditions across train/test). Measure X11/Wayland/macOS/Windows permissions separately; do not claim identical behavior.

Prioritize coarse gaze routing + voice deixis ('this window', 'that field') over character-level gaze typing. Camera frames are processed only in volatile RAM; derived numeric signals are not identity, emotion, fatigue or attention diagnoses. Include clear capture-on indicator, an immediate off control, and fallback to hotkey/voice when tracking fails. Existing validation #419 owns co-design.

EMG, EEG and EOG: honest roles¶

Software candidates: BrainFlow for supported device streams and synthetic boards; MNE-Python for offline EEG filtering, epochs and artifacts; Lab Streaming Layer / pylsl for local timestamped acquisition; MOABB for standardized offline benchmark datasets; scikit-learn for interpretable classifiers; LibEMG for muscle pipelines. Evaluate licenses and board capabilities individually. Keep EEG as an optional research source of high-confidence activation events, not direct actions. Never infer diagnosis, mental state, identity or consent from a biometric stream.

Interaction contract¶

flowchart LR
    A["Voice / gaze / face / EMG / experimental EEG"] --> B["Local acquisition adapter"]
    B --> C["Timestamped derived observation + quality"]
    C --> D{"Confidence and freshness gate"}
    D -->|"insufficient"| X["Abstain + visible feedback"]
    D -->|"sufficient"| E["Intent/target resolver"]
    E --> F{"Authority and action risk"}
    F -->|"confirmation required"| G["Explicit user confirmation"]
    F -->|"safe"| H["Local OS action / dictation"]
    G --> H
    H --> I["Verify outcome + recover"]

A minimal research-only fixture schema is {schema_version, time_ms, modality, sample_type, quality, confidence, intent_candidate, target_id, decision, outcome}. No raw sensor payload, biometric identity, transcript or screen content. Timestamps are relative; file IDs synthetic. Separate observation, hypothesis, proposed action, confirmed action, and verified outcome; never let a model emit an unreviewed OS command.

First reproducible prototype: static Hugging Face Space¶

1. Use an HF account/organization only if the owner chooses. Create a Static HTML Space named, for example, yazses-multimodal-hci-lab. Place index.html, style.css, app.js and synthetic fixtures.json in its Space repository. No backend, token or secrets are needed for synthetic public display.
2. Show a fake desktop with three target windows, a simulated gaze point (slider), a voice-intent dropdown, a squeeze/blink toggle, confidence and freshness, and an action-preview pane. All interactions are local Javascript and never actually inject an OS event.
3. Simulate uncertain gaze, mismatched voice command, stale camera frame, and accidental double blink. Require explicit confirmation for risky action; always expose reset/stop.
4. Show an evaluation table that computes accuracy, abstention, errors/hour and replay latency from documented synthetic traces. Label all numbers 'illustrative; not device benchmarks'.
5. Add README/model and dataset cards with license, provenance, supported hardware, intended use, limitations, prohibited uses, test splits and versioned artifacts. Provide local launch instructions from the YazSes main repo. A GitHub→Space sync, if later added, must use reviewed least-privilege credentials, never user sensor data.
6. Optionally run a local Gradio comparison dashboard on the researcher's machine. Real camera processing stays inside the local YazSes process; a hosted Space cannot be advertised as local merely because it uses Gradio.

A first local Python model catalog experiment (independent of the daemon):

from huggingface_hub import hf_hub_download

# Explicitly replace with an audited repo, exact commit and expected file.
path = hf_hub_download(
    repo_id="AUDITED_ORG/AUDITED_MODEL",
    filename="AUDITED_FILE",
    revision="FULL_VERIFIED_COMMIT_SHA",
    local_files_only=True,  # never makes a network request
)
print(path)

First review actual license, model-format compatibility, hash and source. For an explicit online installation step, a separate installer can use snapshot_download(..., revision=<full commit>) after consent and verification. Do not train or share real human biosignal datasets by default.

Evidence gates and measurements¶

References and related planning¶

• Hugging Face Hub: models, datasets, Spaces, Hub client.
• MediaPipe Face Landmarker.
• Eye/camera roadmap, brain-control review, epic #692.
• Engineering and phased roadmap.