Spec: Shared Camera Perception — One Webcam, Derived Signals, Multiple Consumers¶

Goal¶

Create one lazy, privacy-bounded camera/FaceLandmarker source that can serve gaze, head pose and face switches without multiple consumers opening or processing the webcam independently.

Non-goals¶

• no gesture classification;
• no pointer movement;
• no calibration-policy rewrite;
• no dedicated eye tracker;
• no raw-frame event bus;
• no identity/emotion/attention inference.

Public data contracts¶

Public perception values are immutable Python dataclasses/protocol values with derived numbers only.

Required semantics:

Common¶

• timestamp_s: monotonic timestamp of the source observation;
• confidence or quality: normalized/documented;
• no cv2/MediaPipe object references;
• no raw bytes/images.

GazeSignal¶

Carries raw/calibration-input gaze features, not necessarily final screen coordinates. The existing gaze adapter remains responsible for calibration semantics.

Suggested fields: - timestamp; - normalized iris/eye feature x/y; - confidence; - optional head-motion quality if used for gating.

HeadPoseSignal¶

• timestamp;
• yaw/pitch/roll in documented units (radians preferred internally);
• confidence.

FaceSignal¶

• timestamp;
• normalized Mapping[str, float] blendshape scores needed by switch detection;
• confidence/face presence.

The implementation may combine these in PerceptionSample.

Lifecycle¶

States¶

DORMANT
  -> STARTING (first consumer)
  -> RUNNING
  -> DEGRADED (camera/model alive but one channel unavailable)
  -> FAILED (source unavailable; no actions)
  -> STOPPING
  -> DORMANT (last consumer released)

A simpler implementation may not materialize an enum, but tests must cover equivalent behavior.

Consumer ownership¶

The source tracks consumers by lease/subscription, not by each feature independently owning camera close.

Rules: - consumer count 0 -> camera closed; - first consumer -> open once; - additional consumer -> no additional open/model instance; - last release -> close; - duplicate release/close -> harmless; - process shutdown -> close regardless of lease bookkeeping.

MediaPipe configuration¶

One FaceLandmarker result should request/provide what enabled consumers need: - landmarks for iris/gaze; - facial transformation matrix for head pose where available; - blendshapes for face switches.

Optimization is allowed: if only gaze is enabled, the implementation may disable unused expensive outputs if MediaPipe permits dynamic/source-level configuration without creating a second camera.

Correctness rules: - no second capture loop for another consumer; - channel-specific absence does not kill other channels; - timestamp alignment comes from the same source observation where possible.

Threading¶

The camera/model may run on a worker thread/live callback.

Requirements: - public latest-sample read is thread-safe; - shutdown cannot leave capture/model thread alive; - callback never blocks on desktop action; - consumer exceptions do not crash source loop; - the source does not call feature actions.

Freshness¶

Each consumer can determine whether a sample is too old.

The source records the true source timestamp. It must not refresh a stale sample timestamp merely because a consumer read it.

Lazy dependency contract¶

When no camera feature is enabled: - importing core YazSes does not import cv2 or mediapipe; - no model path is resolved/downloaded; - no camera probe occurs.

Optional dependencies remain behind the existing gaze/camera feature installation path unless a later ADR changes packaging.

Model lifecycle¶

Reuse the existing FaceLandmarker model asset/download policy.

The source: - can reuse an already downloaded model; - reports model-missing/download failure accurately; - does not trigger hidden network activity merely because doctor probes the feature; - keeps runtime inference offline after asset availability.

Failure behavior¶

Privacy¶

Hard requirements: - raw frames are not persisted; - raw frames are not sent over IPC; - raw frames are not logged; - default measurement export does not include landmarks/blendshapes; - no runtime network call carries camera-derived data.

Observability¶

Expose derived state sufficient for doctor/daemon status: - camera feature requested yes/no; - source state; - backend/model name; - last sample age; - active consumer names/count if safe; - failure reason; - no biometric detail.

Acceptance criteria¶

• Given zero consumers, camera/model are not imported/opened.
• Given first consumer, capture opens exactly once.
• Given two consumers, capture/model count remains one.
• Given one consumer release, remaining consumer continues.
• Given final release, camera closes.
• Given no face, no stale sample is replayed as new.
• Given missing head transform, gaze sample still survives.
• Given failed camera, ordinary daemon path remains usable.
• Existing gaze behavior remains unchanged after #396.

Required tests¶

• pure signal dataclasses;
• fake capture/model lifecycle;
• multi-consumer one-open test;
• shutdown/idempotence;
• per-channel degradation;
• lazy-import/no-camera test;
• existing tests/test_gaze_mediapipe.py;
• existing tests/test_gaze_wiring.py.

Definition of done¶

uv run python -m pytest tests/test_gaze_mediapipe.py tests/test_gaze_wiring.py tests/ -q
uv run ruff check .
uv run mypy src/yazses

Live-camera measurement is required before feature-tier promotion, not to merge the pure/source foundation.