{
  "schema_version": "1.0",
  "url": "https://cerinamroth.com/ml/openvino/",
  "title": "OpenVINO: Intel GPU LLM Inference — Cerin Amroth",
  "description": "OpenVINO contributions for Intel GPU language-model inference: CPU fallback fixes, upstream pull requests, measured results and clearly labeled projections.",
  "date_modified": "2026-09-25",
  "representations": {
    "json": "https://cerinamroth.com/data/pages/ml/openvino.json",
    "markdown": "https://cerinamroth.com/data/pages/ml/openvino.md"
  },
  "headings": [
    {
      "level": 1,
      "id": null,
      "text": "OpenVINO"
    },
    {
      "level": 2,
      "id": "bug",
      "text": "The bug class: a silent CPU fallback on modern Intel drivers"
    },
    {
      "level": 2,
      "id": "fix",
      "text": "The fix"
    },
    {
      "level": 2,
      "id": "contributions",
      "text": "Contributions"
    },
    {
      "level": 2,
      "id": "impact",
      "text": "Impact"
    },
    {
      "level": 2,
      "id": "measured-h",
      "text": "Measured receipt"
    },
    {
      "level": 2,
      "id": "projected-h",
      "text": "Projected, not measured projection"
    },
    {
      "level": 2,
      "id": "contact",
      "text": "Contact"
    }
  ],
  "links": [
    {
      "text": "Home",
      "url": "https://cerinamroth.com/"
    },
    {
      "text": "ML systems",
      "url": "https://cerinamroth.com/ml/"
    },
    {
      "text": "OpenVINO",
      "url": "https://github.com/openvinotoolkit/openvino"
    },
    {
      "text": "pjordanandrsn",
      "url": "https://github.com/pjordanandrsn"
    },
    {
      "text": "ML systems",
      "url": "https://cerinamroth.com/ml/"
    },
    {
      "text": "#35661",
      "url": "https://github.com/openvinotoolkit/openvino/pull/35661"
    },
    {
      "text": "#35712",
      "url": "https://github.com/openvinotoolkit/openvino/pull/35712"
    },
    {
      "text": "#36017",
      "url": "https://github.com/openvinotoolkit/openvino/pull/36017"
    },
    {
      "text": "#36543",
      "url": "https://github.com/openvinotoolkit/openvino/pull/36543"
    },
    {
      "text": "author:pjordanandrsn",
      "url": "https://github.com/openvinotoolkit/openvino/pulls?q=author%3Apjordanandrsn"
    },
    {
      "text": "ov-impact-bench",
      "url": "https://github.com/pjordanandrsn/ov-impact-bench/tree/810ec5f49a9d6e0f80010999dd29fa25dab93104"
    },
    {
      "text": "results.jsonl",
      "url": "https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/results.jsonl"
    },
    {
      "text": "results.jsonl",
      "url": "https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/results.jsonl"
    },
    {
      "text": "README.md",
      "url": "https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/README.md"
    },
    {
      "text": "810ec5f49a9d",
      "url": "https://github.com/pjordanandrsn/ov-impact-bench/tree/810ec5f49a9d6e0f80010999dd29fa25dab93104"
    },
    {
      "text": "810ec5f49a9d",
      "url": "https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/README.md#population-impact-honest"
    },
    {
      "text": "jordan@cerinamroth.com",
      "url": "mailto:jordan@cerinamroth.com"
    },
    {
      "text": "pjordanandrsn/ov-impact-bench@810ec5f49a9d",
      "url": "https://github.com/pjordanandrsn/ov-impact-bench/tree/810ec5f49a9d6e0f80010999dd29fa25dab93104"
    },
    {
      "text": "ML systems",
      "url": "https://cerinamroth.com/ml/"
    }
  ],
  "structured_data": [
    {
      "@context": "https://schema.org",
      "@type": "BreadcrumbList",
      "itemListElement": [
        {
          "@type": "ListItem",
          "position": 1,
          "name": "Home",
          "item": "https://cerinamroth.com/"
        },
        {
          "@type": "ListItem",
          "position": 2,
          "name": "ML systems",
          "item": "https://cerinamroth.com/ml/"
        },
        {
          "@type": "ListItem",
          "position": 3,
          "name": "OpenVINO"
        }
      ]
    },
    {
      "@context": "https://schema.org",
      "@type": "WebPage",
      "name": "OpenVINO upstream work",
      "url": "https://cerinamroth.com/ml/openvino/",
      "description": "OpenVINO contributions for Intel GPU language-model inference: CPU fallback fixes, upstream pull requests, measured results and clearly labeled projections.",
      "publisher": {
        "@type": "Organization",
        "name": "Cerin Amroth Research",
        "url": "https://cerinamroth.com/"
      },
      "author": {
        "@type": "Person",
        "name": "Jordan Anderson",
        "url": "https://github.com/pjordanandrsn"
      },
      "isBasedOn": [
        "https://github.com/pjordanandrsn/ov-impact-bench/tree/810ec5f49a9d6e0f80010999dd29fa25dab93104",
        "https://github.com/openvinotoolkit/openvino/pull/35661",
        "https://github.com/openvinotoolkit/openvino/pull/35712",
        "https://github.com/openvinotoolkit/openvino/pull/36017",
        "https://github.com/openvinotoolkit/openvino/pull/36543"
      ],
      "dateModified": "2026-09-04"
    }
  ],
  "content_markdown": "[Home](https://cerinamroth.com/) › [ML systems](https://cerinamroth.com/ml/) › OpenVINO \n\n# OpenVINO\n\n \n\nUpstream contributions to [OpenVINO](https://github.com/openvinotoolkit/openvino), Intel's inference toolkit — keeping Intel-GPU LLM inference from silently falling back to the CPU on modern drivers. Contributed in the open as [pjordanandrsn](https://github.com/pjordanandrsn): 3 merged, 1 open (states verified 2026-09-04 15:18 UTC).\n\n \n\nThe same practice as the NVIDIA work on this site's [ML systems](https://cerinamroth.com/ml/) pages, aimed at the other accessible-hardware vendor: getting real inference out of the GPU a person already owns. Here that means fixing the toolkit itself. What was measured and what is only projected sit in two separate blocks below, and the receipt behind the measured block is linked at a pinned commit.\n\n \n\n## The bug class: a silent CPU fallback on modern Intel drivers\n\n \n\nIntel's Compute Runtime (NEO) 23.x and newer stopped declaring the `__local`-pointer overloads of `intel_sub_group_block_read` — they moved to a separate extension (`cl_intel_subgroup_local_block_io`) that not every driver advertises. OpenVINO GPU kernels that read shared local memory through those intrinsics then fail to compile (`CL_BUILD_PROGRAM_FAILURE`) on current drivers. Under the default `AUTO`/`HETERO` device policies that failure is caught and the whole model runs on the CPU with no error surfaced — a large, silent regression that looks like \"the GPU just isn't helping.\" With an explicit `device=\"GPU\"` the same failure aborts model load, which is the visible mode the upstream PRs document. The trigger is driver-determined, not silicon-age: a Gen 9.5 UHD P630 on NEO 23.43 advertises no extension and hits it, which is what disproved the tempting \"old GPU = old code path\" assumption.\n\n \n\n## The fix\n\n \n\nA two-tier dispatch for the affected block-read family, living in the shared batch header so every kernel inherits it: the hardware intrinsic when the driver advertises the extension, an inline per-lane gather emulation when it does not — mirroring the pattern already used for the `__global` case. Landed as a shared-header refactor migrating the affected kernels at once, plus a driver-independent regression test that compiles the family through the kernel cache so the symbols can never silently vanish again.\n\n \n\n## Contributions\n\n \n\n \n | PR | Title (upstream) | State \n | [#35661](https://github.com/openvinotoolkit/openvino/pull/35661) | [GPU] Fix fully_connected_gpu_gemv compilation on modern NEO drivers (Gen 9.5+) | merged 2026-05-07\n | [#35712](https://github.com/openvinotoolkit/openvino/pull/35712) | [GPU] Fix lora/moe kernel compilation on modern NEO drivers (Gen 9.5+) | merged 2026-05-20\n | [#36017](https://github.com/openvinotoolkit/openvino/pull/36017) | [GPU] Add driver-independent regression test for __local block-read kernel compilation | merged 2026-06-03\n | [#36543](https://github.com/openvinotoolkit/openvino/pull/36543) | [Core/Transformations] Internal-op input-validation sweep + matcher cleanup | open\n\n \n\nStates and titles fetched from the GitHub API (`gh pr view N -R openvinotoolkit/openvino --json state,mergedAt`) and verified 2026-09-04 15:18 UTC; the page is regenerated, never hand-edited, so an older timestamp here means the page has not been rebuilt since, not that the states changed. All PRs: [author:pjordanandrsn](https://github.com/openvinotoolkit/openvino/pulls?q=author%3Apjordanandrsn).\n\n \n\n## Impact\n\n \n\nQuantified with a purpose-built harness, [ov-impact-bench](https://github.com/pjordanandrsn/ov-impact-bench/tree/810ec5f49a9d6e0f80010999dd29fa25dab93104) (MIT), whose whole point is to replace one estimate with one measurement. It has run on one device. The first block is what that device measured; the second is what the README projects from it and labels as projection. They are not the same kind of statement and this page does not blend them.\n\n \n\n \n\n## Measured receipt\n\n \n\n BoxIntel(R) Xeon(R) W-1250 CPU @ 3.30GHz + Intel(R) UHD Graphics P630 (Gen 9.5, the weakest affected part), home-lab QNAP with the iGPU passed into a Container Station container, kernel `6.6.32-qnap` DriverNEO `23.43.027642`; `cl_intel_subgroup_local_block_io` absent — the modern-NEO regime the fix is load-bearing in RuntimeOpenVINO GenAI; wheels per the README: release `2026.1.0` (pre-fix) and nightly `2026.3.0.dev20260520` (with the fix). `results.jsonl` records hardware, driver and model but not the wheel version; that attribution is the README's. Model`OpenVINO/TinyLlama-1.1B-Chat-v1.0-int4-ov`, 8 samples per device, timings from GenAI's `PerfMetrics` Run2026-05-21 00:32 UTC — record 4 of 4 in [`results.jsonl`](https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/results.jsonl) (the only record with both devices completing) \n\n \n\nOn the release wheel the GPU worker dies in kernel compilation — captured 3 times in the log as a structured record (`terminate called after throwing an instance of 'cl::BuildError' / what(): clBuildProgram`), which is the bug itself. On the fixed nightly both devices complete:\n\n \n\n \n | Device | Decode | Time-to-first-token | Per-token | CPU busy  \n | CPU — the fallback the bug forces | 13.9 tok/s | 355 ms | 72.1 ms | 92.0% \n | GPU — with the fix | 18.4 tok/s | 133 ms | 54.3 ms | 76.4% \n | GPU ÷ CPU | 1.33× (harness-recorded `speedup_x`) | 2.67× faster (355.4 ÷ 133.1, derived here; the README rounds to 2.7×) | 1.3× | −15.6 pts \n\n \n\nNo energy result. Every completed device row carries `energy_backend: null` and `j_per_token_mean: null`: the validated box exposes no `powercap`/hwmon under QTS and no wall-plug backend was attached, so the harness's energy axis — the J/token the projections below need — has not been measured on any device. The build checks this against the file and fails if a later record adds one.\n\n \n\nAlso in the log: an earlier CPU-only record (00:23 UTC; its GPU worker died in kernel compilation, the pre-fix failure mode) measured 19.8 tok/s decode and 240 ms TTFT on the same CPU. The ratios above are computed inside the single paired record, not across records; the CPU figure moves between runs on this box, so read the log before quoting a ratio.\n\nReceipt: [`results.jsonl`](https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/results.jsonl) and [`README.md`](https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/README.md) at [`810ec5f49a9d`](https://github.com/pjordanandrsn/ov-impact-bench/tree/810ec5f49a9d6e0f80010999dd29fa25dab93104) (fetched 2026-09-04 15:18 UTC; `results.jsonl` sha256 `beb92f69b7d045fa…`). Any figure in this block that is not in that file at that commit is a defect in this page.\n\n \n\n \n\n \n\n## Projected, not measured projection\n\n \n\nNothing in this block has been run. These are the README's own extrapolations from the single P630 measurement, reproduced here with its labels intact so they cannot be read as results.\n\n  \n- Newer Intel GPU classes. The README expects the GPU-vs-fallback gap to widen on Iris Xe / Meteor Lake / Lunar Lake / Arc — \"2–3× decode, 5–8× TTFT\" at the population median — from a thermal-cap argument (a faster GPU beside a power-capped fanless CPU), not from a run: \"the harness has run on exactly one device.\" A second-hardware run (Iris Xe, Lunar Lake or a desktop Arc) is what would turn that assertion into a demonstration. \n- Affected population. A filter chain from an assumed OpenVINO install base yields a midpoint of ~24k devices with an honest compound range of ~4–75k; every step is a labelled dial, and the ~70–80% share hitting the silent `AUTO`/HETERO path rather than the visible explicit-GPU abort is an estimate keyed to `AUTO` being the stack default, not a count. \n- Aggregate energy. ~40–70 MWh/year across that population uses an assumed J/token, because — see the measured block — no J/token has been recorded yet. It inherits the full population uncertainty on top. \n\n \n\nSource for every figure in this block: the README's \"Population impact (honest)\" section at [`810ec5f49a9d`](https://github.com/pjordanandrsn/ov-impact-bench/blob/810ec5f49a9d6e0f80010999dd29fa25dab93104/README.md#population-impact-honest). The build asserts each quoted phrase is still present there; if the README changes, this page fails to build until the text is revisited.\n\n \n\n \n\n## Contact\n\n \n\n[jordan@cerinamroth.com](mailto:jordan@cerinamroth.com)\n\n \n\nGenerated by `scripts/build_openvino.py` from [pjordanandrsn/ov-impact-bench@810ec5f49a9d](https://github.com/pjordanandrsn/ov-impact-bench/tree/810ec5f49a9d6e0f80010999dd29fa25dab93104) (commit 2026-05-24, fetched 2026-09-04 15:18 UTC; `results.jsonl` sha256 `beb92f69b7d045fad3be14923a775a427f876417401e7f67d6102bfff5ffc8c5`) · upstream PR states verified 2026-09-04 15:18 UTC via the GitHub API · measured figures come only from `results.jsonl`; projections come only from the README and are labelled as such · [ML systems](https://cerinamroth.com/ml/)\n",
  "content_sha256": "63c420c34b7bff1e9d943327f424694fea70d762f76749cd8282a1fa651764f5"
}