Research: local accelerator backends (CPU, CUDA, ROCm, OpenVINO, Apple, QNN, DirectML, Vulkan) for offline ASR¶

Status: Research note (not a decision). ADR-030 freeze unchanged; no default is changed. Issue: NEXT-ASR-04 #795; parent #787; siblings #792, #793, #794, #797; synthesis #798. Snapshot: YazSes main at a241a77b6c59d65fb9265a0475d42f1a97fdfe6a; upstream pages read 2026-11-05. Reuses vocabulary from: runtime/provenance snapshot, pinned evaluation protocol, Proposed ADR-033 ("resolved engine", "system comparison" vs "runtime comparison", "negative result"). Binding policy: ADR-030, ADR-019. Not done: no measurement, no GPU/NPU job, no model or wheel download, no src/ change. The only command run on a machine was a read-only capability probe (section 4).

Evidence labels¶

• [Verified-primary-source] the statement appears in the cited vendor/upstream page as returned by the fetch tool. Every fetch was a text summary, not a full read; a page marked "not read in full" may say more or less than recorded here.
• [Upstream-claim] a vendor performance, "best-in-class" or "x3 faster" assertion. Never a YazSes result.
• [Verified-in-code] read in this repository at the snapshot, or observed by running read-only code locally.
• [Assumption] my inference; named so it can be struck. [Unknown] not established by anything read.

1. Executive answer¶

Defer every accelerator backend; adopt a no-hardware "truthful device reporting" hardening as a candidate for the owner; reject any mandatory or default accelerator. This is a recommendation for the owner, not a decision.

• Upstream documentation shows a real but fragmented accelerator landscape: each runtime supports a different subset of {OS x vendor x driver}, and no row of the matrix below is supported by one runtime on all three desktop OSes (section 3). [Verified-primary-source, from the cells cited there]
• The shipped default (CPU, int8, faster-whisper) already works on all three OSes with no driver. Any accelerator adds a driver surface YazSes cannot control. [Verified-in-code, section 4]
• The one thing YazSes can fix and test without hardware is honesty: today yazses status reports engine and checkpoint but not the device or compute type actually running (section 4). [Verified-in-code]
• Every statement about speed, power or thermals is [Unknown] here. #795 asks for "repeatable wins"; nothing in this note shows one.

2. Primary sources (all read 2026-11-05; all "not read in full", fetched as text summaries)¶

Could not read: the OpenVINO Intel-GPU and NPU configuration pages on docs.openvino.ai returned only site navigation, so GPU driver minima and NPU driver install steps from OpenVINO's own setup pages are [Unknown]; S15 (repo source of the NPU page) and S17 were read instead. S14/S15 are master, not a pinned release; the docs-site "2025" version may differ. [Assumption] CUDA/cuDNN redistribution licence terms (EULA) were not read: [Unknown].

3. Support matrix: backend x OS x runtime¶

Cell = what the cited page says; "unknown" = the page(s) read are silent. "Speech" = whether the runtime's docs state speech-model support. All cells are [Verified-primary-source] unless a cell says [Upstream-claim] or [Assumption].

Source disagreements recorded, not resolved: (1) S6 calls the ORT ROCm EP "deprecated" while S12 says "removed since 1.23"; S21 still lists ROCm as supported by onnx-asr, so that README line cannot be read as a promise for current ORT. (2) S1 states CC >= 3.5 but S5 requires CUDA 12 / cuDNN 9 for current ctranslate2; the effective minimum GPU generation is therefore [Unknown] (not read). (3) S2 names Apple Accelerate but not Metal; S18 documents Metal for ggml. The two runtimes do not share an Apple GPU story. [Assumption] a statement of "supports speech models" for a generic runtime (ORT, OpenVINO NPU) does not imply the Whisper/Parakeet graphs run efficiently on it; the NPU and QNN pages describe static-shape, quantized or vision-first constraints that Whisper-style autoregressive decoders may not meet [Unknown].

3.1 Three structural findings from the matrix¶

1. Encoder-only offload is the common speech pattern. whisper.cpp documents Core ML, OpenVINO and the Ryzen AI NPU path as accelerating the Whisper encoder only (S18). The autoregressive decoder, which dominates short hold-to-talk utterances, stays on the CPU unless the runtime says otherwise. Whether the encoder dominates a 1-10 s dictation burst is [Unknown]; it is exactly the kind of claim section 8 must measure rather than assume.
2. NPUs and QNN reward static, quantized graphs. S11 (fixed shapes, quantized-only HTP) and S15 (dynamic shapes bounded and "limited to vision models") describe constraints that conflict with variable-length dictation unless utterances are padded or bucketed. Bucketing would be a new design, not a flag. [Assumption on the consequence; the constraints themselves are Verified-primary-source]
3. Vendor lock follows the OS. The cross-vendor rows are Vulkan (whisper.cpp only, S18) and Windows ML's dynamic EP catalogue (S23); every other row is single-vendor. [Assumption] A hardware-profile-targeted doc ("if you have X, set Y") is therefore more honest than a generic "GPU support" claim.

Hypotheses about who might benefit (not findings): long-form yazses transcribe and meeting capture on a CUDA workstation, where throughput matters more than hold-to-talk latency, and large multilingual checkpoints that are slow on CPU. The CPU int8 default targets short bursts on ordinary laptops. [Assumption, untested]

4. Current YazSes behaviour (read in code, snapshot above)¶

• Default: [stt] engine = faster-whisper, device = "cpu", compute_type = "int8"; "cuda" is described in-config as "supported but unmeasured" and auto is mentioned in a comment only (src/yazses/config.py L56-69). [Verified-in-code]
• Only CTranslate2 reads the device. stt/factory.py passes device/compute_type to FasterWhisperEngine only; the Parakeet adapter maps compute_type to an int8 flag and the Moonshine adapter has no device argument (stt/parakeet.py, stt/moonshine.py). Neither passes ONNX execution providers, so which provider onnx-asr/ORT picks is decided by the installed wheel. [Verified-in-code]
• Heavy dependencies stay optional. onnx-asr[cpu,hub] is the parakeet extra, lazily imported, and a missing extra falls back to faster-whisper with a logged reason (pyproject.toml, stt/factory.py). A GPU wheel (onnxruntime-gpu, onnxruntime-qnn, openvino) would be a different distribution of the same import name as CPU onnxruntime; coexistence in one environment is [Unknown] for the pages read. [Assumption]
• Failure reporting gap 1, device is not validated. [stt] device is absent from the closed-set validation table in configcheck.py (it lists stt.engine only). A misspelled device surfaces as a load exception. [Verified-in-code]
• Gap 2, a device failure is reported as a missing model. In _load_model, only messages containing "compute type" become ComputeTypeUnsupportedError; any other exception from WhisperModel(device="cuda") (for example a missing cuDNN) ends as ModelUnavailableError ("model unavailable"), and there is no CPU fallback (stt/faster_whisper.py L66-84). What the real CUDA error strings are on a broken driver is [Unknown] without hardware. [Verified-in-code for the control flow]
• Gap 3, yazses status cannot say where it runs. The status reply carries engine, engine_model and engine_fallback (#644) but no device, compute type or accelerator (core/daemon.py _active_engine_fields; grep finds no compute_type/stt.device in the status path). The device appears only in a log line "Loading STT model ... on cuda (...)". So configured and resolved device are indistinguishable to the user, the same class of defect #644 fixed for engine fallback. [Verified-in-code]
• Read-only probe on this host (Linux, AVX2 CPU, no NVIDIA tool, ctranslate2 4.8.1, onnxruntime 1.28.0, run with the project venv, no network): get_cuda_device_count() == 0; get_supported_compute_types("cpu") returned float32, int8, int8_float32; get_supported_compute_types("cuda") raised RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version; onnxruntime.get_available_providers() listed AzureExecutionProvider and CPUExecutionProvider. [Verified-in-code, observed once on one host]
• ORT provider-list egress question. The CPU wheel lists an Azure provider (S6 marks it "preview", cloud). YazSes selects none today. Whether any provider contacts a network without explicit session options is [Unknown]; the ADR-019 egress review must cover it before any code passes an ORT provider list through.
• Existing evidence about GPU: docs/how-to/cpu-and-battery.md says "[stt] device = "cuda" exists and is untested for power"; docs/install-cost.md records the lesson that PyPI's PyTorch CUDA stack was 7x the size of YazSes until a CPU-only index was forced on 2026-10-21. [Verified-in-code]

5. The "offline + no new mandatory dependency" constraint¶

• A new accelerator must be an optional extra, lazily imported, with the CPU path unchanged (AGENTS.md rules 2-3). ORT is already resolved at two versions in uv.lock (1.23.2 for Intel macOS, 1.28.0 elsewhere) because Intel macOS wheels stopped at 1.23.x; an accelerated ORT wheel adds a third axis to that fork. [Verified-in-code]
• Install size. Recorded in docs/install-cost.md: installed ctranslate2 135 MB, onnxruntime 53 MB. For the same release PyPI lists a Linux x86-64 wheel of roughly 42 MB (S4, fetch summary; the page also showed implausibly small macOS figures, so wheel sizes are not used further). The size of CUDA/cuDNN runtime libraries (S5 pip route), onnxruntime-gpu, openvino and the NPU driver stack was not measured: [Unknown]. [Assumption] CUDA libraries alone could exceed the current base install, as the PyTorch precedent did.
• Driver surface (what YazSes cannot ship or fix). Each backend needs a vendor driver outside the package: NVIDIA driver matched to the CUDA major (S5, S7, S19), Intel NPU/GPU drivers "not included" with OpenVINO (S14), a Vulkan ICD (S18), Qualcomm NPU driver (S23), Windows 11 24H2 for hardware EPs (S22, S23). Version coupling is explicit upstream: cuDNN 8 vs 9 are mutually incompatible (S7); faster-whisper pins ct2 by CUDA/cuDNN generation (S5). [Verified-primary-source]
• Offline first run. Core ML and OpenVINO compile the model to a device-specific blob on first run (S18); the NPU page warns pre-compiled blobs are "not recommended ... in production" (S15); Windows ML downloads EPs on demand and via Windows Update (S23). The last conflicts with a no-network-after-install promise unless the EP is provisioned deliberately; it would need an ADR-019 classification as a new egress path. [Verified-primary-source for the upstream text; the ADR consequence is an Assumption]
• Crash and security surface. A GPU/NPU runtime puts vendor user-mode driver code inside the dictation process; a hang or reset takes down the hold-to-talk path. Subprocess isolation is a design option, not evidence. [Assumption]
• Hybrid laptops, suspend/resume, driver updates mid-session, a missing GPU after docking: none is addressed in the pages read. [Unknown]

6. What can be verified without hardware, and what cannot¶

7. Options for the owner (all inside ADR-030's freeze unless stated)¶

8. Evidence required under ADR-033 before any backend is promoted¶

1. Resolved-backend attestation. The run asserts the device and provider actually executing (not the requested one); a silent CPU fallback is a recorded failure, never a scored accelerator row (ADR-033 points 1 and 3). Needs gap 3 closed first.
2. Pinned environment manifest. Runtime and wheel versions with hashes, driver and firmware versions, OS build, GPU/NPU model, power profile; add driver fields to the provenance record from the 2026-11-03 snapshot.
3. Same-weights discipline. Separate the effect of device from the effect of precision (int8 vs fp16) and conversion, as the snapshot did for runtime vs model; otherwise label a "system comparison".
4. Paired quality and latency on a named machine class. Matched-corpus WER/CER with the paired non-inferiority framing of ADR-033, plus key-release-to-final p50/p95 and cold load including first-run compile; sampled peak RSS and device memory (the current harness reads process RSS only [Assumption]).
5. Negative cases as data. No driver, wrong driver major, GPU absent, driver reset or suspend/resume mid-hold, hybrid GPU, NPU busy, unsupported shape; each must yield a typed, user-fixable error and a CPU path.
6. Energy/thermal only with instrumentation (wall power meter or vendor counters); otherwise "CPU seconds, a proxy", as cpu-and-battery.md already says.
7. Offline and egress. Install and first run with egress blocked; any download (EP, blob, compile cache) registered in ADR-019 first.
8. Footprint. Measured install size and licence review of every redistributed library (CUDA/cuDNN, OpenVINO, QNN, Vulkan loader); this note did not read those terms.
9. Hardware testers. At least one named owner per row (Linux NVIDIA, Windows Intel NPU, Apple silicon, Snapdragon X); a row with no tester stays "unknown".

9. Unresolved questions¶

• Effective minimum NVIDIA generation for current ctranslate2 + CUDA 12 / cuDNN 9; ORT CUDA EP OS support list.
• Whether Whisper-style decoders run on OpenVINO NPU or QNN at all (S15, S11 are silent or restrictive).
• Whether OpenVINO or ORT GPU wheels can coexist with CPU onnxruntime required by onnx-asr[cpu], useful-moonshine-onnx and kokoro-onnx.
• Whether the ORT Azure provider (or any provider) can transmit without explicit options.
• The Intel GPU/NPU driver minima from OpenVINO's setup pages (unreadable here).

10. Checklist against #795¶

• Current implementation audited (section 4). [x] Upstream docs consulted, dated, with fetch limits stated (sections 2-3). [ ] Repeatable method: delegated to #793/ADR-033, applied in section 8. [x] Costs/egress/licences recorded where read; licences of redistributed libraries NOT read. [x] Adopt/defer/reject given (section 7). No ADR text proposed: ADR-033 already carries the gate.