Skip to content

XR Voice API — WebSocket JSON-RPC Event Server for AR/VR/Metaverse

Description

A WebSocket-based event server that exposes YazSes's transcription and command pipeline to any local application regardless of programming language, runtime, or platform. XR headsets (Meta Quest, Apple Vision Pro running companion Mac app, HTC Vive PC), Unity and Unreal Engine games, browser-based WebXR applications, and custom tooling all connect to ws://localhost:8765 and receive a real-time stream of structured JSON events. An XR-optimised profile reduces latency by using smaller chunk sizes and a lower-latency model configuration. Companion SDKs for Unity, Unreal, and JavaScript lower integration friction to under one hour for a typical game developer. The WebSocket server is an additional transport layer alongside the existing UNIX socket IPC; the existing CLI and tray continue to use the UNIX socket.

User Pain

XR and game developers building voice-driven applications today must either build their own speech recognition pipeline from scratch (complex, time-consuming) or use cloud APIs (latency, cost, privacy). The latency requirement for XR is strict: voice commands that take >200ms to respond produce visible input lag that causes cybersickness [EVIDENCE src-011] [EVIDENCE src-014]. Existing offline alternatives (Vosk, Coqui) require deep integration into the engine's audio pipeline. YazSes already solves the hard parts (audio capture, VAD, ASR) but exposes no interface that non-Python processes can consume. A WebSocket server bridges this gap with zero additional complexity for the game developer: connect, listen, handle events.

Proposed Solution

Server module: src/yazses/platform/xr/server.py

import asyncio
import json
import websockets

class XRVoiceServer:
    def __init__(self, host="localhost", port=8765):
        self.host = host
        self.port = port
        self._clients: set[websockets.WebSocketServerProtocol] = set()

    async def broadcast(self, event: dict):
        if self._clients:
            message = json.dumps(event)
            await asyncio.gather(
                *[c.send(message) for c in self._clients],
                return_exceptions=True
            )

Event schema:

// Partial transcript (emitted every 300ms during speech)
{
  "type": "partial_transcript",
  "text": "go to the main",
  "confidence": 0.87,
  "timestamp_ms": 1715692800423
}

// Final transcript (emitted on speech end)
{
  "type": "final_transcript",
  "text": "go to the main menu",
  "confidence": 0.94,
  "words": [
    {"word": "go", "start": 0.0, "end": 0.21, "confidence": 0.99},
    {"word": "to", "start": 0.22, "end": 0.35, "confidence": 0.99},
    {"word": "the", "start": 0.36, "end": 0.45, "confidence": 0.98},
    {"word": "main", "start": 0.46, "end": 0.72, "confidence": 0.95},
    {"word": "menu", "start": 0.73, "end": 1.10, "confidence": 0.89}
  ],
  "timestamp_ms": 1715692801533
}

// Command detected (when cap-003 grammar is active)
{
  "type": "command_detected",
  "intent": "navigate",
  "action": "go_to_line",
  "params": {"line": 42},
  "raw_transcript": "go to line 42",
  "confidence": 0.97,
  "timestamp_ms": 1715692801533
}

// Recording state changes
{
  "type": "state_change",
  "state": "RECORDING",   // "IDLE" | "RECORDING" | "TRANSCRIBING"
  "timestamp_ms": 1715692800100
}

XR profile (config yazses config set profile xr): 

[profile.xr]
model = "tiny.en"
chunk_size_ms = 200          # vs 300ms default — lower latency
vad_min_silence_ms = 300     # shorter pause tolerance for command mode
streaming = true             # enable partial transcripts
commands_enabled = true      # enable cap-003 grammar
websocket_port = 8765
websocket_host = "localhost" # never bind to 0.0.0.0

Daemon integration: When [profile.xr] or [server.websocket_enabled = true] is set, the daemon starts the WebSocket server alongside the UNIX IPC server at startup. Both servers share the same event bus via an asyncio Queue.

Authentication: WebSocket connections are local-only (localhost). Optional shared secret in config for future remote scenarios: 

[server.websocket]
auth_token = ""   # empty = no auth required (localhost only)

Companion SDKs:

  1. Unity C# package (sdks/unity/YazSes/):
  2. YazSesManager.cs: MonoBehaviour, connects to ws://localhost:8765, raises C# events OnPartialTranscript, OnFinalTranscript, OnCommandDetected.

  3. Distributed via Unity Package Manager (UPM) with a package.json + GitHub Packages registry.

  4. Unreal Engine plugin (sdks/unreal/YazSesPlugin/):

  5. Blueprint nodes: Connect, Disconnect, OnTranscriptReceived event dispatcher.

  6. Uses Unreal's built-in WebSockets module (available since UE 4.26).

  7. JavaScript/TypeScript npm package (sdks/js/yazses-client/):

  8. YazSesClient class for Node.js and browser WebXR.
  9. Published to npm as @yazses/client.

Feasibility Notes

Complexity: Medium-low. The WebSocket server itself is straightforward using the websockets library (already available or trivially added). The main work is the companion SDK development and the profile configuration system. The Unity SDK requires a developer with C#/Unity knowledge.

Dependencies: websockets>=13.0 (pure Python, no native extensions, ~150KB). All SDKs have zero runtime dependencies beyond their native platform SDKs.

Risks: - Port conflicts: 8765 may be in use. Make the port configurable and document yazses doctor checking for conflicts. - Security: binding only to localhost is non-negotiable; document clearly and add a startup assertion that rejects non-localhost bind addresses. - Apple Vision Pro: the companion Mac app approach (running YazSes on the paired Mac) adds a hardware requirement; document as a supported but non-trivial setup. - websockets library is an additional dependency; keep it in an optional extra (pip install yazses[xr]).

Success Metrics

  • End-to-end latency from speech release to final_transcript event on a localhost WebSocket client: <800ms (matching local injection latency).
  • partial_transcript event delivered within 350ms of speech onset.
  • SDK integration time for a developer with no prior YazSes experience: <1 hour to receive events in Unity (measured in user study).
  • WebSocket server handles ≥10 simultaneous connected clients without degradation.
  • Zero WebSocket connections accepted from non-localhost origins (verified by test suite).
  • Unity example scene ships with the SDK and demonstrates voice-controlled UI navigation in <200 lines of C#.

Evidence

  • [EVIDENCE src-010] WebSocket decoupling recommendation for voice event APIs — validates the transport choice and event schema design.
  • [EVIDENCE src-011] XR voice input latency requirements — establishes the <200ms chunk processing threshold for acceptable XR experience.
  • [EVIDENCE src-014] <200ms XR latency threshold for voice commands — confirms the latency target and the impact of exceeding it on user experience.
  • [EVIDENCE src-005] faster-whisper tiny.en CPU int8 latency profile — confirms the XR profile's model and chunk size choices are achievable within the latency budget.