OpenVINO
Upstream contributions to OpenVINO, Intel's inference toolkit — keeping Intel-GPU LLM inference from silently falling back to the CPU on modern drivers. Contributed in the open as pjordanandrsn: 3 merged, 1 open (states verified 2026-09-04 15:18 UTC).
The same practice as the NVIDIA work on this site's ML systems pages, aimed at the other accessible-hardware vendor: getting real inference out of the GPU a person already owns. Here that means fixing the toolkit itself. What was measured and what is only projected sit in two separate blocks below, and the receipt behind the measured block is linked at a pinned commit.
The bug class: a silent CPU fallback on modern Intel drivers
Intel's Compute Runtime (NEO) 23.x and newer stopped declaring the __local-pointer overloads of intel_sub_group_block_read — they moved to a separate extension (cl_intel_subgroup_local_block_io) that not every driver advertises. OpenVINO GPU kernels that read shared local memory through those intrinsics then fail to compile (CL_BUILD_PROGRAM_FAILURE) on current drivers. Under the default AUTO/HETERO device policies that failure is caught and the whole model runs on the CPU with no error surfaced — a large, silent regression that looks like "the GPU just isn't helping." With an explicit device="GPU" the same failure aborts model load, which is the visible mode the upstream PRs document. The trigger is driver-determined, not silicon-age: a Gen 9.5 UHD P630 on NEO 23.43 advertises no extension and hits it, which is what disproved the tempting "old GPU = old code path" assumption.
The fix
A two-tier dispatch for the affected block-read family, living in the shared batch header so every kernel inherits it: the hardware intrinsic when the driver advertises the extension, an inline per-lane gather emulation when it does not — mirroring the pattern already used for the __global case. Landed as a shared-header refactor migrating the affected kernels at once, plus a driver-independent regression test that compiles the family through the kernel cache so the symbols can never silently vanish again.
Contributions
| PR | Title (upstream) | State |
|---|---|---|
| #35661 | [GPU] Fix fully_connected_gpu_gemv compilation on modern NEO drivers (Gen 9.5+) | merged 2026-05-07 |
| #35712 | [GPU] Fix lora/moe kernel compilation on modern NEO drivers (Gen 9.5+) | merged 2026-05-20 |
| #36017 | [GPU] Add driver-independent regression test for __local block-read kernel compilation | merged 2026-06-03 |
| #36543 | [Core/Transformations] Internal-op input-validation sweep + matcher cleanup | open |
States and titles fetched from the GitHub API (gh pr view N -R openvinotoolkit/openvino --json state,mergedAt) and verified 2026-09-04 15:18 UTC; the page is regenerated, never hand-edited, so an older timestamp here means the page has not been rebuilt since, not that the states changed. All PRs: author:pjordanandrsn.
Impact
Quantified with a purpose-built harness, ov-impact-bench (MIT), whose whole point is to replace one estimate with one measurement. It has run on one device. The first block is what that device measured; the second is what the README projects from it and labels as projection. They are not the same kind of statement and this page does not blend them.
Measured receipt
- Box
- Intel(R) Xeon(R) W-1250 CPU @ 3.30GHz + Intel(R) UHD Graphics P630 (Gen 9.5, the weakest affected part), home-lab QNAP with the iGPU passed into a Container Station container, kernel
6.6.32-qnap - Driver
- NEO
23.43.027642;cl_intel_subgroup_local_block_ioabsent — the modern-NEO regime the fix is load-bearing in - Runtime
- OpenVINO GenAI; wheels per the README: release
2026.1.0(pre-fix) and nightly2026.3.0.dev20260520(with the fix).results.jsonlrecords hardware, driver and model but not the wheel version; that attribution is the README's. - Model
OpenVINO/TinyLlama-1.1B-Chat-v1.0-int4-ov, 8 samples per device, timings from GenAI'sPerfMetrics- Run
- 2026-05-21 00:32 UTC — record 4 of 4 in
results.jsonl(the only record with both devices completing)
On the release wheel the GPU worker dies in kernel compilation — captured 3 times in the log as a structured record (terminate called after throwing an instance of 'cl::BuildError' / what(): clBuildProgram), which is the bug itself. On the fixed nightly both devices complete:
| Device | Decode | Time-to-first-token | Per-token | CPU busy |
|---|---|---|---|---|
| CPU — the fallback the bug forces | 13.9 tok/s | 355 ms | 72.1 ms | 92.0% |
| GPU — with the fix | 18.4 tok/s | 133 ms | 54.3 ms | 76.4% |
| GPU ÷ CPU | 1.33× (harness-recorded speedup_x) | 2.7× faster (355.4 ÷ 133.1, derived here; the README rounds to 2.7×) | 1.3× | −15.6 pts |
No energy result. Every completed device row carries energy_backend: null and j_per_token_mean: null: the validated box exposes no powercap/hwmon under QTS and no wall-plug backend was attached, so the harness's energy axis — the J/token the projections below need — has not been measured on any device. The build checks this against the file and fails if a later record adds one.
Also in the log: an earlier CPU-only record (00:23 UTC; its GPU worker died in kernel compilation, the pre-fix failure mode) measured 19.8 tok/s decode and 240 ms TTFT on the same CPU. The ratios above are computed inside the single paired record, not across records; the CPU figure moves between runs on this box, so read the log before quoting a ratio.
Receipt: results.jsonl and README.md at 810ec5f49a9d (fetched 2026-09-04 15:18 UTC; results.jsonl sha256 beb92f69b7d045fa…). Any figure in this block that is not in that file at that commit is a defect in this page.
Projected, not measured projection
Nothing in this block has been run. These are the README's own extrapolations from the single P630 measurement, reproduced here with its labels intact so they cannot be read as results.
- Newer Intel GPU classes. The README expects the GPU-vs-fallback gap to widen on Iris Xe / Meteor Lake / Lunar Lake / Arc — "2–3× decode, 5–8× TTFT" at the population median — from a thermal-cap argument (a faster GPU beside a power-capped fanless CPU), not from a run: "the harness has run on exactly one device." A second-hardware run (Iris Xe, Lunar Lake or a desktop Arc) is what would turn that assertion into a demonstration.
- Affected population. A filter chain from an assumed OpenVINO install base yields a midpoint of ~24k devices with an honest compound range of ~4–75k; every step is a labelled dial, and the ~70–80% share hitting the silent
AUTO/HETERO path rather than the visible explicit-GPU abort is an estimate keyed toAUTObeing the stack default, not a count. - Aggregate energy. ~40–70 MWh/year across that population uses an assumed J/token, because — see the measured block — no J/token has been recorded yet. It inherits the full population uncertainty on top.
Source for every figure in this block: the README's "Population impact (honest)" section at 810ec5f49a9d. The build asserts each quoted phrase is still present there; if the README changes, this page fails to build until the text is revisited.