# MoE Software Status & Known Limits — Cerin Amroth Source: https://cerinamroth.com/ml/status/ [Home](https://cerinamroth.com/) › [ML systems](https://cerinamroth.com/ml/) › Status # Current status The repositories' position pages, rendered verbatim from the pinned commits. Nothing is summarised or reworded here, and the tier counts beside each heading are taken from that repository's claim register at the same commit. ## Evidence vocabulary The words the position pages and the claim registers use, defined once in [`docs/system-manifest.json`](https://cerinamroth.com/ml/system-manifest.json) (identical in both repositories) and rendered verbatim: confirmeda public receipt plus an independent confirmation cell (kernel register)verifieda public receipt plus an independent check (runtime register)measuredthe run happened and the receipt is public in the repositorymeasured-privatethe run happened and the receipt lives outside the repository; NOT publicly reproducible from the repository aloneprojectedarithmetic from measured inputs, not a runopena statement the documents make that has no evidence either way yet; it backs no capabilitysupersededreplaced by a later claim that names it; kept as the record, never currentretiredwithdrawn; kept as the record, never current ## `experts4bit-qlora` v0.58.1 — [docs/STATUS.md](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/STATUS.md) at [605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) (claims tiered in the public register: 465 measured · 8 measured-private · 3 open · 19 superseded · 9 retired) ### Status — what this package does, what is current, what is open As of 2026-10-24, version 0.58.1 (the version of record is `pyproject.toml`'s). This page states the current position in each area, with the claim id behind each number; [`docs/claims.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/claims.json) holds every claim's full text and evidence. The dated narrative behind these positions, with the readings they replaced and why, is [`STATUS-RECORD.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/STATUS-RECORD.md). Evidence words: measured = a run with a receipt in this repository. measured-private = a real run whose receipt is in a private audit tree, so you cannot check it from here; such numbers say so. retired = published, now known wrong, kept so the retraction is findable. superseded = true as measured, but a later row is the one to quote. Speeds are within-box readings on the named card: absolute times do not travel between hosts, and a wall-clock ratio can move with the host's CPU. --- #### What you get today ##### Fit and memory It fits, and it trains, on a 12 GB card. Full bf16 OLMoE-1B-7B runs out of memory on a 12 GB card; in 4-bit it loads at 4.70 GB and trains under 8 GB, and the streaming loader never materialises the bf16 model in CPU or GPU RAM. QLoRA on the frozen NF4 experts moves a held-out Alpaca eval from 1.4813 to 1.0290. `e4b.train.olmoe-fits`, `e4b.train.olmoe-converges` ([METHODOLOGY](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/METHODOLOGY.md)). Expert offload takes 30B-class MoEs past VRAM. Qwen3-30B-A3B trains TC1's matched set (fp32 adapters) on a 12 GB RTX A2000 at 10.463 GB peak allocated (micro-batch 1 × accum 8, the registered secondary after micro-batch 2 ran out of memory on that shared card; one draw; `e4b.train.frontier.qwen3.a2000-12gb.2026-10-02`). An older 5-step reading put it at 7.16 GB and Gemma-4-26B-A4B at 8.47 GB, both out of memory without offload; that reading was not reproduced (a later attempt, at a different configuration on a 5090, read 9.13 GB). Offload costs about +11 % s/step at OLMoE scale. The on-disk arena needs 2.56× to 6.40× less host RAM than the pinned-RAM path: at an 8.59 GB cap Qwen3-30B is OOM-killed on host RAM and completes on the arena. `e4b.offload.fits-30b-class`, `e4b.offload.arena-vs-host-ram` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/host-ram-ceiling/RESULTS-scaling.md)). On a 24 GB card, Qwen3-30B-A3B trains the field recipe under expert offload: TC1's matched set (642,514,944 fp32 adapters) at 11.88 GB peak and 10.52 s/step, its trajectory equivalent to resident training on a 5090 (median per-step |Δ| 0.0025). Resident, e4b runs out of memory (24.45 GB). Unsloth 2026.9.14 trains the same set resident there at 24.22 GB and 10.03 s/step; HF neither fits nor offloads, and axolotl's rows are a harness failure, not readings. A fit table, one draw per row, not a speed position. `e4b.train.frontier.qwen3.4090-24gb.2026-10-02` with its `.unsloth-resident`, `.e4b-resident-oom`, `.hf-axolotl` and `.offload-keeps-trajectory` rows ([TC3](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc3/README.md)). Under expert offload, layer-major accumulation is 2.365× faster per step than the stock loop (median step times, the smaller of two ABBA pairs). `layer_major_step` (`experts4bit_qlora.engines.layer_major`) runs a step's micro-batches layer by layer. Each offloaded layer is staged once per step instead of once per micro-batch: 95 stagings against 380. With `enable_fast_train(dgrad=True)` on Qwen3-30B-A3B on one RTX 4090, the stock loop steps 7.641 / 7.514 s and layer-major 3.231 / 3.173 s, at 7.98 GB peak against 8.41. The predicted band [1.25, 2.0] was missed high, and the held-out term (0.015 against 0.01) sits inside the 0.031 same-loop spread at 34 steps. From the CLI trainer, `OFFLOAD_EXPERTS=1 LAYER_MAJOR=1` (which implies `FAST_TRAIN=1`) makes the same calls, checked bit for bit against the library on a fixture. At the CLI's own default recipe (SEQ 192, token-budget batches of 2048, r 8 with attention LoRA, torch AdamW) on one RTX 4090 it steps 1.455× faster than `OFFLOAD_EXPERTS=1 FAST_TRAIN=1` (median step times, the smaller of two ABBA pairs; 11.596 / 11.595 s against 7.956 / 7.968 s): its larger micro-batches carry more compute, so the copy is a smaller share of the step, and the a-priori band [1.5, 2.6] was missed low. It replicates on one RTX 5090: 2.311× (the smaller of two ABBA pairs; 6.017 / 6.014 s against 2.597 / 2.602 s), inside the band [1.8, 3.0] predicted a priori from the host link. It holds on Mixtral-8x7B on one RTX 4090: 2.118× (the smaller of two ABBA pairs; 11.386 / 11.386 s against 5.372 / 5.375 s), inside the band [1.5, 2.6] predicted a priori, at 5.64 GiB peak against 5.72. `e4b.train.p137.layer-major-offload.qwen3.4090.2026-10-18` ([P137](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p137/RESULTS-p137.md)), `e4b.train.p140.layer-major-offload.qwen3.5090.2026-10-18` ([P140](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p140/RESULTS-p140.md)), `e4b.train.p144.layer-major-offload.mixtral.4090.2026-10-18` ([P144](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p144/RESULTS-p144.md)), `e4b.train.p146.layer-major-cli.qwen3.4090.2026-10-18` ([P146](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p146/RESULTS-p146.md)). On the 12 GB RTX A2000 the same set trains under offload at micro-batch 1, at 10.46 GB peak, held-out 0.8483 beside the 5090's 0.8516 / 0.8487. Unsloth refuses on that card, axolotl's wheels need a newer driver, and HF runs out of memory at load. The A2000 is a correctness-only testbed, so no step time is quoted. `e4b.train.frontier.qwen3.a2000-12gb.2026-10-02`. Kimi-K3 runs at full depth on the 12 GB A2000: all 93 layers at 4.07 GB peak VRAM once Triton's cache is warm, the MXFP4 experts streamed from a 1.446 TB SSD arena and the 108.76 GB dense side read from the checkpoint's byte offsets, with 0 bytes pinned in host RAM. Five processes complete "The capital city of France is" as " Paris. It is"; the per-token NLL correlates with Fireworks' kimi-k3 at r = 0.9965; on grouped-nf4-gemm 0.33.6 the forward is bit-identical run to run. Feasibility and fidelity only, with no decode speed quoted, through a custom driver rather than `load_moe_4bit_streaming`. `e4b.offload.kimi-k3.full-depth.a2000.five-runs.2026-09-28`, `e4b.quality.kimi-k3.vs-fireworks.per-token.a2000.2026-09-28`, `e4b.parity.kimi-k3.reproducible-on-gnf4-0.33.6.a2000.2026-09-29` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/kimi-k3-a2000/RESULTS-kimi-k3-a2000.md)). Dense execution remains in development. DQ7's out-of-sample reading is VOID; no reserve calibration or 24 GB boundary has passed. DQ9 completed sixteen known-subject diagnostics with the corrected full-logit workspace: allocator estimates were above measured peaks, but all ten streamed full-device estimates remained below sampled driver use, by up to 1.14 GB. One setup clear released 32 MiB on one arm and changed no training peak, so it provides no training-memory remedy. DQ9 licenses neither capacity nor calibration; the opt-in and enforced streamed margin remain. `e4b.train.dense-executor.dq9.5090.2026-10-08` ([diagnostic record](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/dq9/receipts/dq9-5090-2/README.md)); `e4b.train.dense-executor.dq7.5090.2026-10-08` ([original VOID reading](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/dq7/RESULTS-dq7.md)). DQ11 is a fusion build candidate, measured-private. Unsloth (U) was faster than e4b's dense path (L) in this matched SDPA, eager, full-logits configuration, not Unsloth's default. The registered quality gates pass and the independently rederived decision is `BUILD_CANDIDATE`; this licenses a separate fusion investigation and carries no capacity or shipping claim. `e4b.train.dense-adapter.dq11.matched-sdpa.5090.2026-10-11` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/dq11/RESULTS-dq11.md)). Its MLP-adapter fusion build closed at NO_GAIN (Phase 2, 2026-10-20; [results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/dq11/RESULTS-dq11-mlp-build.md)). ##### Training against other frameworks Qwen3-30B-A3B against Unsloth 2026.9.14: Unsloth spends 1.92× e4b's GPU time per step (medians of two draws); wall-clock 2.80× on an AMD EPYC 7713. TC1's field recipe (alpaca, seq 2048, micro-batch 2 × accum 4, r 16, AdamW-8bit) on one RTX 5090, the work matched, both frameworks on one stack, e4b at every current default. The wall ratio depends on the host: Unsloth runs 14× e4b's CPU ops per step and keeps its GPU busy for 0.33 of its step against e4b's 0.49, so a slower host stretches Unsloth's step more. Held-out loss agrees within 0.0012; e4b peaks 1.89 GB higher (26.16 against 24.27 GB). HF + PEFT runs out of memory on the 32 GB card. `e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.field-device-ratio`, `e4b.train.h2h.unsloth.qwen3.5090.2026-10-02.hf-oom` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos69.md)). Where those 1.89 GB sit: the 1.883 GB peak gap is transient. 2.044 GB of e4b's transient at the peak is grouped-nf4-gemm's padded LoRA delta, and every resident tensor class is byte-identical on both sides (TC1 amendment 73, a memory census at the claim's commits; descriptive, one draw a side). `e4b.train.memory-census.field.5090.2026-10-22` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos69mc.md)). At today's pins the padded delta is not the last limit: grouped-nf4-gemm's compact delta or bucketed padding removes all 2.348 GB of it at the peak, but the peak falls only 0.639 GB, to 25.861 GB, because the LM loss backward then sets it (TC1 amendment 74, descriptive, one draw a side; by its registered rule no switch is a default candidate). `e4b.train.padded-delta-switches.field.5090.2026-10-22` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos74pd.md)). Amendment 75's arms do not fit a 4090: with grouped-nf4-gemm's compact delta and the chunked LM loss both on, the field recipe's nvidia-smi peak is 25,727 MiB on an RTX 5090, 1,163 MiB over a 4090's total (TC1 amendment 75, descriptive, one draw a side; its rule credits both savings, and no arm fits). The compact delta's own backward is the next transient. `e4b.train.4090-fit-read.field.5090.2026-10-22` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos75cl.md)). TC1's field recipe fits a 24 GiB RTX 4090 with the opt-in pb1cl256 levers (`NF4_QLORA_COMPACT_DELTA=1`, `NF4_QLORA_PAD_BUCKETS=1`, `E4B_CHUNKED_LM_LOSS=256`). Two RTX 4090s on two machines trained all 60 steps with no out-of-memory error, with 586 MiB of the card's total left at the fullest step (TC1 amendment 78, FIT for this fixture only; the levers stay opt-in until a default-change PR). `e4b.train.4090-fit.field.4090.2026-10-24` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos78fit.md)). On packed 4,096-token rows, Unsloth 2026.9.14 spends 1.160× e4b's GPU time per step (medians of two draws). The wall-clock ratio follows the host: 1.472 on the profiled host, 1.287 and 1.773 on two others. At e4b's defaults its training phase peaks 1.04 GB above Unsloth's 24.86 GB (medians of two draws), and 0.30 GB above with the opt-in `E4B_CKPT_OFFLOAD=1`. `e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.packed-4k-device-ratio`, `e4b.train.memory.packed-4k-position-defaults.5090.2026-10-07` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos68.md)). Mixtral-8x7B against Unsloth 2026.9.14: e4b is faster, Unsloth/e4b 1.144 (medians of two draws), both resident on one stack (torch 2.12.1 / transformers 5.5.0), on an AMD EPYC 7B13: 3.24 against 3.71 s/step, at 2.07 GB more peak on e4b (its fp32 expert absmax). The register's other Mixtral ratios answer other questions: 0.836 put e4b on the field image's torch 2.8 stack (medians of two draws) on a Core Ultra 9 285K, the host most favourable to Unsloth seen (the same-stack box read that stack at 1.013), and 0.361 is tp2's footprint trade (a median of steps 11..60; 2026-09-06, Unsloth 2026.9.2), e4b under CPU expert offload at 3.22 GB against Unsloth resident at 29.16 GB, not a resident position. `e4b.train.h2h.unsloth.mixtral.5090.2026-10-05.same-stack`, `e4b.train.h2h.unsloth.mixtral.5090.2026-10-04.dense-default`, `e4b.train.h2h.unsloth.mixtral.5090.2026-09-06` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc2/RESULTS-tc2-mixtral-samestack.md)). Granite and OLMoE: e4b is faster than each comparator (TC2, 2026-10-04, one RTX 5090; transformers 5.18.0 + PEFT 0.21.2, axolotl 0.20.0, Unsloth 2026.9.14): HF/e4b 1.299 (medians of steps 11..60 over two draws each) and axolotl/e4b 1.150 (medians of steps 11..60; one axolotl draw, two e4b draws) on Granite-3.1-3B-A800M, Unsloth/e4b 1.821 (medians of steps 11..60; one Unsloth draw, two e4b draws) on OLMoE-1B-7B, each pair comparable or equivalent on held-out loss. e4b's peak is about half HF's on Granite and 1.70 GB above Unsloth's on OLMoE. Unsloth adapts no expert parameter of Granite, so there is no Unsloth position there, and gpt-oss-20b has no adapter set common to the frameworks. `e4b.train.h2h.hf.granite.5090.2026-10-04`, `e4b.train.h2h.axolotl.granite.5090.2026-10-04`, `e4b.train.h2h.unsloth.olmoe.5090.2026-10-04`, `e4b.train.h2h.unsloth.granite.5090.2026-10-02.coverage`, `e4b.train.h2h.unsloth.gptoss.5090.2026-10-02.no-common-set` ([TC2](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc2/README.md)). Qwen3.6-35B-A3B: at its defaults e4b does not fit the matched set resident on 32 GB; Unsloth 2026.9.14 does, at 30.47 GB. Under expert offload e4b trains the set at 19.35 GB. With the comparator's bytes at micro-batch 1 (a measurement hook, not a training option) e4b trains it resident and Unsloth/e4b reads 2.049 (medians of two draws), a labelled row. `e4b.train.h2h.unsloth.qwen3_5.5090.2026-10-04`, `e4b.train.h2h.unsloth.qwen3_5.5090.2026-10-02.e4b-offload`, `e4b.train.h2h.unsloth.qwen3_5.5090.2026-10-04.mb1-dq-frozen4`. axolotl 0.20.0 on Qwen3-30B-A3B: e4b is faster, axolotl/e4b 2.775 (medians of two draws) at matched work (1.979 on a second host). At steady state (steps 101–200) axolotl's scattermoe native-best / e4b as shipped reads 1.238 and 1.146 on two hosts (medians of two draws), while at step 200 axolotl's held-out is 0.7705 against e4b-as-shipped's 0.7948. Both were read on 2026-10-03, before the later training defaults. `e4b.train.h2h.axolotl.qwen3.5090.2026-10-03`, `e4b.train.h2h.axolotl.qwen3.5090.2026-10-03.native-steady-state` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-matched19.md)). On an H100 NVL at e4b 0.45.0's default settings, e4b is faster per step than Unsloth 2026.9.14: Unsloth/e4b 1.061, at 2.93 GB more peak on e4b and ×1.23 Unsloth's energy per step. `e4b.train.h2h.unsloth.qwen3.h100.release-0.45.0` ([TC1c](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1c/README.md)). ##### Training support per family Reach for the fused path: `enable_fast_train(model, dgrad=True)`. Across two 30B-class MoEs, five datasets each, 200 steps per cell, it runs 1.52–1.81× faster per step at 0.75–0.81× peak VRAM and 0.80–0.92× energy, with loss parity and the frozen 4-bit stack bit-identical. `e4b.train.flagship-matrix` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/flagship-matrix/RESULTS-flagship-matrix.md)). Support is stated per family and per path. Each family's real weights go through `load_moe_4bit_streaming` + `verify_moe_4bit(strict=True)`, then each path trains 60 steps against the family's own reference loop (|Δ final loss| and median step-wise |Δ| ≤ 0.05). A PASS is a PASS on one text, not a convergence claim. The first six rows are lane tp1's ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/train-parity-20260905/tp1/RESULTS-tp1.md)), the last five lane MG1's ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/moegen/mg1/mg1-5090-2/RESULTS-mg1.md)). | model_type | quantize | reference_train | fast_train | batched_train | nvme_train | native_mxfp4_train | `olmoe` | supported | supported | supported · `e4b.train.parity.tp1.olmoe.fused.2026-09-05`, `e4b.train.parity.mg1.olmoe.fused.2026-10-04` | void · `e4b.train.parity.tp1.olmoe.batched.2026-09-05` | not_tested | n/a | `qwen3_moe` | supported | supported | supported · `e4b.train.parity.tp1.qwen3.fused.2026-09-05` | void · `e4b.train.parity.tp1.qwen3.batched.2026-09-05` | not_tested | n/a | `gemma4_text` | supported | supported | supported · `e4b.train.parity.tp1.gemma4.fused.2026-09-05`, `e4b.train.parity.mg1.gemma4.fused.2026-10-04` | void · `e4b.train.parity.tp1.gemma4.batched.2026-09-05` | not_tested | n/a | `granitemoe` | supported | supported | supported · `e4b.train.parity.tp1.granite.fused.2026-09-05` | supported · `e4b.train.parity.tp1.granite.batched.2026-09-05` | not_tested | n/a | `mixtral` | supported (offload) | supported (offload) | supported under offload · `e4b.train.parity.tp1.mixtral.fused.2026-09-05` | supported · `e4b.train.parity.tp1.mixtral.batched.2026-09-05` | not_tested | n/a | `gpt_oss` | supported | refused; attention-only QLoRA trains · `e4b.train.parity.tp1.gptoss.attn_only.2026-09-05` | refused · `e4b.train.parity.tp1.gptoss.fused.2026-09-05` | refused · `e4b.train.parity.tp1.gptoss.batched.2026-09-05` | refused (#402) | experimental · `e4b.train.parity.tp1.gptoss.mxfp4.2026-09-05` | `lfm2_moe` | supported | supported | supported · `e4b.train.parity.mg1.lfm2.fused.2026-10-04` | not_tested | not_tested | n/a | `granitemoehybrid` | supported | supported | supported · `e4b.train.parity.mg1.graniteh.fused.2026-10-04` | not_tested | not_tested | n/a | `ernie4_5_moe` | supported | supported | supported · `e4b.train.parity.mg1.ernie.fused.2026-10-04` | not_tested | not_tested | n/a | `nemotron_h` | supported | supported | supported · `e4b.train.parity.mg1.nemotron.fused.2026-10-04` | not_tested | not_tested | n/a | `qwen3_5_moe` | supported | supported | supported · `e4b.train.parity.mg1.qwen3_5.fused.2026-10-04`, `e4b.train.parity.mg1.qwen3_5.p2.2026-10-05` | not_tested | not_tested | n/a `void` means the arm ran but the kernel was not reached on every layer, so it carries no parity number. On `gemma4_text` the attention-4-bit configuration is supported too, judged against the family's measured training-parity floor (0.121 final / 0.1275 median). `e4b.train.p67.gemma4.attn4-reorder-floor.5090.2026-09-24`. The machine-readable form is `training_support` in [`capabilities.json`](https://cerinamroth.com/ml/compatibility/#experts4bit-qlora), whose `model_families` is exactly the families whose `fast_train` is supported. ##### Training defaults and opt-ins `enable_fast_train`'s defaults each come from a registered A/B on Qwen3-30B-A3B on one RTX 5090 (table below). Five earlier step-time changes (one host sync per MoE layer pass, the lean LoRA delta, the tile rule, fused RMSNorm, host reuse) each stepped the field recipe at 0.847–0.968 of the code before, held-out within 0.0021. `e4b.train.host-syncs.qwen3.5090.2026-10-03`, `e4b.train.lora-delta-lean.qwen3.5090.2026-10-03`, `e4b.train.prefill-tile-rule.qwen3.5090.2026-10-03`, `e4b.train.fused-rmsnorm.qwen3.5090.2026-10-03`, `e4b.train.host-reuse.qwen3.5090.2026-10-04`. Opt-ins, each read and left off: - `E4B_CKPT_OFFLOAD=1`: 0.74 GB off the packed-row training peak for 1.023 of the step. `e4b.train.ckpt-flavour.default.5090.2026-10-07` - `E4B_MOE_KEEP_LAYERS` (with grouped-nf4-gemm's compact delta): 0.835 of the step with 32 of 48 layers' MoE activations kept (medians of two draws), at +4.5 GB; use it where that headroom exists ([CHOOSING](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/CHOOSING.md)). `e4b.train.moe-keep.qwen3.5090.2026-10-04` - `GNF4_TRAIN_GEMM=decoded`: 1.066 of the fused kernels' step on Qwen3 (medians of two draws), so `auto` does not take it. `e4b.train.decoded-route.qwen3.5090.2026-10-06` - `NF4_QLORA_SINGLE_LADDER` (grouped-nf4-gemm): its `auto`, the ladder only with fp32 adapters, becomes the default. With fp32 adapters it stepped 0.797 of the time on a host-bound box and 1.031 on a GPU-bound one (medians of two draws; registered bound 1.05). With bf16 adapters, the shipped arm, it does not engage (1.002). `e4b.train.single-ladder.field.5090.2026-10-08`, `e4b.train.single-ladder-auto.gpu-bound.5090.2026-10-08` Energy: on a card that already fits the model, 4-bit costs energy. `bnb.matmul_4bit` costs 1.748× native bf16's J/op at decode, 1.601× at prefill and 1.965× in training. The fused 4-bit MoE forward's J/token at batch 4096 is 0.063 of batch 64's. `e4b.train.energy-honest.5090.2026-10-05` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p114/RESULTS-p114.md)). ##### Serving `serve_paged` decodes under bucketed CUDA graphs by default: 5.597× the eager server with 16 requests (the smaller of two pairs) and 9.020× with one (Qwen3-30B-A3B NF4, one RTX 5090), at no measurable quality cost (+0.00036 nats against eager, P110 AT_PARITY). `e4b.serve.p109.decode-graphs-vs-eager-default.qwen3.5090.2026-10-03`, `e4b.serve.p110.graph-arithmetic-quality.qwen3.5090.2026-10-03` ([SERVING](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/SERVING.md)). Capacity: 64 slots serve 12 req/s on one RTX 5090, against 4 at the old 16 slots (Qwen3-30B-A3B int4, 512-token prompts, 2,048 tokens a slot, TTFT ≤ 1.0 s and TPOT ≤ 100 ms). Since 0.50.0 `serve_paged` sizes its slots to the serve estimate and decodes the widest step as one graph replay. One 64-row piece costs no measurable quality against four 16-row pieces (−0.0032 nats, P117 AT_PARITY). Under load, output text depends on which requests a step batches together; serial output is unchanged. `e4b.serve.sc2e.64-slots-buckets-auto.qwen3.5090.2026-10-07` (12 req/s), `e4b.serve.sc2e.64-slots-default-buckets.qwen3.5090.2026-10-07` (8 req/s on the 1–16 buckets), `e4b.serve.p117.wide-bucket-quality.qwen3.5090.2026-10-08` ([SC2e](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/sc2e/README.md)). Prefill: K19's lean dispatch by default. On the int4 K19 prefill route, gate_up reads the token rows itself and down stores in the caller's order, so the 4,096-row gather and the unsort are gone. It is bit-identical on every window and ×0.968 [0.966, 0.970] per 512-token first-chunk prefill replay (median replay time, mean over 8 processes; Qwen3-30B-A3B int4, one RTX 5090, P130). `e4b.serve.p130.prefill-lean-dispatch.qwen3.5090.2026-10-10` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p130/RESULTS-p130.md)). Prefill: the decode folds take prompt rows on Qwen3-MoE. On `qwen3_moe`, a prefill's norms, residual-and-norm and q/k norm with the rotary run decode's fused kernels: ×1.118 [1.116, 1.119] per 512-token first-chunk prefill replay (mean over 4 process pairs), at parity on a teacher-forced read (bias −0.0035 nats; Qwen3-30B-A3B int4, one RTX 5090, P130). Other families keep the eager chain until a read of their own. `e4b.serve.p130.prefill-glue-speed.qwen3.5090.2026-10-10`, `e4b.serve.p130.prefill-glue-quality.qwen3.5090.2026-10-10` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p130/RESULTS-p130.md)). Single-request decode. grouped-nf4-gemm's bandwidth-targeted NF4 decode GEMV decodes one request 1.2417× as fast (the smaller of two ABBA pairs), 16 unchanged, within P110's quality bar; it is grouped-nf4-gemm's default from 0.43.0 at Qwen3's NF4 expert shapes on GPUs with 160 or more SMs. `e4b.serve.p116.gemv-bw.qwen3.5090.2026-10-07`. On Mixtral-8x7B, at plans chosen on an RTX 5090, the same GEMV decodes one request 2.41× as fast (the smaller of two ABBA pairs; 19.5 → 8.1 ms a step), 16 unchanged, within the same bar. `e4b.serve.p141.gemv-bw.mixtral.5090.2026-10-18` ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p141/RESULTS-p141.md)). Two more reads set grouped-nf4-gemm 0.47.0's `auto` scope. On an RTX 4090 (128 SMs), Qwen3-30B-A3B decodes one request 1.5145× as fast (the smaller of two ABBA pairs; `e4b.serve.p133.gemv-bw.qwen3.4090.2026-10-18`, [results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p133/RESULTS-p133.md)). On an RTX 5090 at a 400 W limit, Granite-3.1-3b-a800m decodes it 1.3979× as fast and OLMoE-1B-7B 1.5011× (each the smaller of two ABBA pairs; `e4b.serve.p134.gemv-bw.granite.5090.2026-10-18`, `e4b.serve.p134.gemv-bw.olmoe.5090.2026-10-18`, [results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p134/RESULTS-p134.md)). Sixteen requests are unchanged and quality is within the same bar in each. - Scope since grouped-nf4-gemm 0.47.0: its default also takes Mixtral-8x7B's, Granite-3.1-3B-A800M's and OLMoE-1B-7B's expert shapes on GPUs with 160 or more SMs, and the RTX 4090 (128 SMs, sm_89) at Qwen3-30B-A3B's two shapes only. The shapes are licensed, not the families: any model with those expert shapes takes them. Every other shape and smaller GPU keeps dot-pad; `GNF4_GEMV_BW=0` restores it. Against bitsandbytes, op level and eager (lane P236, 2026-10-24; measured, descriptive: [`bench/p236/RESULTS-p236.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p236/RESULTS-p236.md), `e4b.kernel.p236.nf4-gemv-vs-bnb.moe-experts.5090.2026-10-24`): on the same NF4 bytes and one RTX 5090, bitsandbytes 0.50.2's `gemm_4bit` is faster than grouped-nf4-gemm 0.51.0's decode GEMV at all 30 cells (six MoE expert projections of Mixtral-8x7B, Qwen3-30B-A3B and OLMoE-1B-7B, at M = 1 to 16): the ratio (bitsandbytes' time over gnf4's) runs 0.30 to 0.92, and 0.49 to 0.87 at M = 1. These are eager calls that include each library's Python dispatch, not the served decode path, which runs under captured graphs, and not K33's device time; the served reads above stand. No default moves. The B=1 fused stack is the default on Qwen3-MoE, Qwen3.5/3.6-MoE and Mixtral. On top of grouped-nf4-gemm 0.43.0's bandwidth GEMV, fused q/k/v plus three glue folds decode one request 1.5902× and 16 requests 1.1644× as fast on Qwen3-30B-A3B NF4, and P115 Phase D's SANE read at T == 1 passes (bias +0.00541 nats, argmax agreement 0.9674). The four knobs resolve to `auto` on the families with such reads and stay off elsewhere. Lane FAM read the others at T == 1, each against its own neutral floor (#1362). Granite-MoE fails on one gated entry of 12, by 2 positions in 1536. gpt-oss-20b fails on every knob, because its own floor sits below the 0.90 agreement backstop; that floor also contains Phase C's 0.924, so Phase C's miss was the gate, not the knob. Qwen3.6-MoE, where only the router epilogue engages, passes, and the epilogue decodes one request there in 0.973 of the step (median step, the larger of two interleaved blocks; FAM Amendment 4), so the family joined the allowlist. Mixtral-8x7B passes on every knob, resolved at a ×0.90 softmax-scale change, and its three folds (no fused q/k/v engages there) decode one request in 0.9393 and 0.9277 of the step (median step times, one per interleaved block), 20.2–20.5 → 19.0 ms (P135, FAM Amendment 5's speed read), so the family joined the allowlist too. Gemma-4-26B-A4B's knobs stay off: the family's own neutral floor sits below the agreement backstop, so the instrument cannot license any knob on its NF4 stack (closed at its proof; the knobs pass the relative gate). `0` on each knob is the way back. `e4b.serve.p115.fused-stack-combined.qwen3.5090.2026-10-08`, `e4b.serve.p115.fused-stack-speed.qwen3.5090.2026-10-07`, `e4b.serve.p115.fused-stack-quality.qwen3.5090.2026-10-07`, `e4b.serve.fam.fused-stack-t1.granite.5090.2026-10-09`, `e4b.serve.fam.fused-stack-t1.gptoss.5090.2026-10-09`, `e4b.serve.fam.fused-stack-t1.qw36.5090.2026-10-09`, `e4b.serve.fam.router-epilogue-speed.qw36.5090.2026-10-09`, `e4b.serve.fam.fused-stack-t1.mixtral.5090.2026-10-09`, `e4b.serve.fam.fused-stack-t1.gemma4.5090.2026-10-10`, `e4b.serve.p135.fold-speed.mixtral.5090.2026-10-18` ([P115](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p115/RESULTS-p115.md), [FAM](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/fam/RESULTS-fam.md), [P135](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p135/RESULTS-p135.md)). Where the default single-request step goes. A kernel-class census of the shipped default on Qwen3-30B-A3B NF4, run on a 5090 with the fused stack and the bandwidth GEMV both on, puts the one-request step at 4.75 ms. The dense bf16 GEMVs (attention projections, router logits, lm_head) lead at 38 %, then the NF4 experts at 26 %, attention at 15 % and launch-bound routing glue at 11 %; the GPU is busy 92 % of the step. At 16 requests the small-M grouped NF4 GEMM is 69 % of an 18 ms step. The census moves no default: it prices the next single-request lever, the dense GEMVs, which needs a quality bar of its own. Its router census of 48 is the 5090 check of the fp32 router probe on the shipped default. `e4b.serve.p123.default-decode-census.qwen3.5090.2026-10-09` ([P123](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p123/RESULTS-p123.md)). The paged FP8 attention decode is capped at 16 splits on Qwen3-MoE. The library plans its split count from the block table's capacity: 32 at one request on 4096-token slots, most of them empty at served contexts. A cap of 16 splits moves buckets 1, 2 and 4 only. - Speed: one request decodes 1.044× as fast on a 5090 (255.6 → 267.0 tok/s; the smallest of four ABBA pairs, binding). Sixteen requests are unchanged (0.992). - Quality: SANE exact (bias 0.0, argmax 1.0). The cap is bitwise-identical to the plan at contexts ≤ 1024, and within A2's fp64 contract (error 0.9976–1.0000× the plan's) at 2048 and above. - Default: on for `qwen3_moe` (16); `E4B_ATTN_SPLIT_CAP=0` restores the plan. - Measured basis: Qwen3-30B-A3B on an RTX 5090 (170 SMs). Other `qwen3_moe` checkpoints (e.g. Qwen3-235B-A22B) and other cards take the default by family without a read of their own. The plan is SM-scaled, so the cap moves B 1–6 on a PRO 6000 and B 1–4 on a 4090. `e4b.serve.p143.attn-split-cap.qwen3.5090.2026-10-19` ([P143](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p143/RESULTS-p143.md)). Qwen3-30B-A3B's lm_head serves one-row decode from an int8 copy (on by default). `E4B_INT8_HEAD_BW=1` rounds the 151936 x 2048 head in place to its per-row int8 grid, P150's I8HEAD. One-row calls go to grouped-nf4-gemm K36's int8 GEMV, which reads half the head's bytes. - Speed: one request decodes 1.052× as fast on a 5090 (263.7 → 277.5 tok/s; the smallest of four ABBA pairs). The step saving is 0.189 ms, 0.998 of P148's microbench projection. - Sixteen requests: 0.992, inside the 0.99 line. S1 is below S0 in every W16 pair, though the W16 rate counts only 16-row steps, where the head is passed to `F.linear`. The cost is observed, not yet attributed. Token divergence, named at review, is bounded out by a $0 router replay (distinct experts per step 0.9990, S1/S0; bf16, teacher-forced on the recorded streams). The resident int8 copy is the remaining named candidate. - Quality: SANE inside the bars (bias +0.0022, argmax 0.989), in line with P150's GO at both of P125's gates. The knob rounds the head at every row count, so every row count's logits change. - Default: on for `qwen3_moe` at this head shape (151936 x 2048). It needs K36's GEMV, first released in grouped-nf4-gemm 0.48.0. With an older release the default resolves off, recorded in `/health`'s `levers.int8_head_bw.source`. `E4B_INT8_HEAD_BW=0` opts out. - Quality basis: read on Qwen3-30B-A3B @ad44e777. Same-shape checkpoints inherit the default unread. - Measured basis: this checkpoint's head on an RTX 5090 (170 SMs), at K36's tuned plan. Other heads take K36's untuned default plan, and other cards are not read. `e4b.serve.p153.int8-head-bw.qwen3.5090.2026-10-19` ([P153](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p153/RESULTS-p153.md)). The int8 head at 2 to 8 rows (`E4B_INT8_HEAD_ROWS`, 8 by default on `qwen3_moe`). In a captured microbench of the head call (one RTX 5090, lane P229), grouped-nf4-gemm K37's `linear_i8` runs Qwen3-30B-A3B's 151936 x 2048 int8 head at 2, 4 and 8 rows in a median 0.51 of `F.linear`'s time on the grid's bf16 rounding (upper bounds 0.5109, 0.5148 and 0.5085 against 0.70), and within a median 1.01-1.03 of the one-row int8 GEMV. The verdict is GO_ROWS_8, so the knob is licensed to its maximum of 8 rows. `e4b.kernel.p229.int8-head-rows.qwen3.5090.2026-10-23` ([P229](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p229/RESULTS-p229.md)). - Served (lane P232, one RTX 5090 at 575 W): on Qwen3-30B-A3B's default server, `E4B_INT8_HEAD_ROWS=8` decodes 2, 4 and 8 requests 1.0311×, 1.0243× and 1.0165× as fast as `1` (the smallest of four ABBA pairs at each), inside P153's quality bars at every group size (NLL gap at most 0.0001 nats, argmax 3064-3066 of 3072), TRIPWIRE none. The verdict is FLIP_ROWS_8. `e4b.serve.p232.int8-head-rows.qwen3.5090.2026-10-24` ([P232](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p232/RESULTS-p232.md)). - Outputs change at 2 to 8 requests: K37 and `F.linear` round differently within bf16's output rounding of the same grid, so near-ties can flip (76 of 112 greedy rows matched in P232). - Default: 8 for `qwen3_moe` at this head shape (151936 x 2048), where the int8 head is on (`E4B_INT8_HEAD_BW` resolved 1). It needs K37, first released in grouped-nf4-gemm 0.49.0. With an older release, with the head off, and with speculative decode (`E4B_PAGED_SPEC`) the default resolves to 1, recorded in `/health`'s `levers.int8_head_bw.rows.source`. Every other family and head shape stays at 1. `E4B_INT8_HEAD_ROWS=1` opts out. - Quality basis: read on Qwen3-30B-A3B. Same-shape checkpoints inherit the default unread. Pinned cuBLASLt GEMMs for o_proj and the router at 2 to 8 rows (`E4B_DENSE_LT_PIN`, opt-in). In a captured microbench of the two dense sites (one RTX 5090, lane P228), the cuBLASLt configuration selected on the card (algo 21, split_k 16) runs Qwen3-30B-A3B's `o_proj` in a median 0.60 of `F.linear`'s time at 2 and 4 rows (upper bounds 0.5981 and 0.6009 against 0.85), and the router gate in a median 0.28 (upper bounds 0.2832 and 0.2843). Both sites PASS. At 8 rows, which never rules, the router reads 0.88: there cuBLAS's own choice goes split-K and closes most of the gap. The selected pins fill K1's table next, with no speed claim. No served speed or quality is claimed: a served read with B3's quality bar comes first, and the knob stays off by default. `e4b.kernel.p228.dense-lt-pin.o-proj.qwen3.5090.2026-10-23`, `e4b.kernel.p228.dense-lt-pin.router.qwen3.5090.2026-10-23` ([P228](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p228/RESULTS-p228.md)). Qwen3-30B-A3B's batched decode runs K25 with K39's byte-permute decode (on by default). `E4B_K25_LUT=prmt` swaps K25's select-tree codebook decode for grouped-nf4-gemm K39's (`lut="prmt"`), which yields the same values bit for bit. - Speed: sixteen requests decode 1.0925× as fast on a 5090 (about 976 → 1067 tok/s; the smallest of four ABBA pairs, 1.0925-1.0934). One request is unchanged (0.9999): K25 is not on the T == 1 path. - Identity: every token the prmt arms decoded is the tree's, at 16 and 1 requests, on the same prompts, lengths and arrival order; the tree's own null held across five processes. - Clocks: the median SM clocks differ by 0.25 %. The tree met the software power-cap event (0x4) in more samples (0.40-0.44 of its samples, against 0.07-0.18), under an unchanged 575 W limit: part of the served difference rather than a confound. - Default: on for `qwen3_moe`. It needs K39's decode, first released in grouped-nf4-gemm 0.50.0, and an NVIDIA device. With an older release or on ROCm the default resolves to the tree (the same values), recorded in `info["k25_lut"]["source"]`. `E4B_K25_LUT=tree` opts out; an explicit `E4B_K25_LUT=prmt` still refuses a package or device that cannot serve it. - Scope: the default covers every `qwen3_moe` checkpoint, Qwen3-235B-A22B included. The decode is bit-identical to the tree, so the values are the same everywhere; the speed was read only at Qwen3-30B-A3B's expert shapes on an RTX 5090 (170 SMs) at today's K25 plan, and no speed is claimed for other shapes (a 235B scale-up lane is where it gets read). Other families stay on the tree. `e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20` ([P158](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p158/RESULTS-p158.md)). qwen3_moe's default from 0.56.1: K25 at plan `c1` under K39's decode (`E4B_K25_PLAN`, e4b#1877; the default from lane P161). `c1` runs K25 at P156's first carried plan pair, gate_up `{block_n 64, kc 256, warps 4, stages 2}` and down `{block_n 128, kc 128, warps 8, stages 3}`, in place of today's single plan. - The default: an unset `E4B_K25_PLAN` resolves at engine build to `c1` on `qwen3_moe` where K25's decode resolved to `prmt`, and to `today` everywhere else, with the reason in `info["k25_plan"]["source"]`. It never refuses: where the decode is not `prmt`, an unset knob serves `today`. `E4B_K25_PLAN=today` opts out. Other families stay on today's plan until a read of their own. - Only under `prmt`. The select tree at `c1`'s shapes is slower than at today's (P155's selection: gate_up 64/256/4/2 at 14.23 ms against 8.32, at 255 registers; [RESULTS-p156](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p156/RESULTS-p156.md), "The cause is the decode"), so an explicit `c1` under any other decode is refused at engine build and in the forward, never run. It is also refused with K25 off or absent. - Measured: a kernel microbench, then a served read. At Qwen3-30B-A3B's W16 shapes on an RTX 5090, under `prmt`, both K25 GEMMs ran 1.6598× faster at `c1` than at today's plan (kernel time today / `c1`, above 1 is faster; the median of 126 paired reps, lower bound 1.6568, clock-adjusted 1.6525; P160 `p160-5090-2`, [RESULTS](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p160/RESULTS-p160.md); UNVERIFIED: no register row). In the served step, P161 (`p161-5090-1`, one RTX 5090 at 575 W, e4b `da3eb5c5`) read sixteen concurrent requests decoding 1.465433× as fast at `c1` as at today's plan, both under `prmt` (rate, the smallest of four ABBA pairs), one request unchanged (0.9966), and every token identical on the same schedule: `e4b.serve.p161.k25-plan-c1.qwen3.5090.2026-10-20`, [RESULTS](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p161/RESULTS-p161.md). One read on one card and one family; qwen3_moe takes `c1` by default on it. - Same bits. The tree, `prmt`, and `prmt` with `c1` give bit-identical outputs through the served call (`tests/test_k25_lut_gpu.py`, including a K that neither of `c1`'s KCs divides). - Records: `info["k25_plan"]` (the build's resolution) and `prefill_routes()["seen"]["moe_k25_plan"]` (each K25 call's plan). Int8 attention projections on Mixtral are opt-in (e4b v0.56.0). The served read is a FLIP, held from the default by its routing qualifier. - What it does: `E4B_SERVE_ATTN_INT8=1` (#1848) holds Mixtral-8x7B's q, k, v and o projections in int8 with a per-row fp32 scale. Decode calls of 1-16 rows run grouped-nf4-gemm 0.49.0's K37 kernels (`dense_i8.linear_i8`), and larger calls dequantize transiently. - Where it refuses: on other families, without K37, and alongside int4 attention or `E4B_DENSE_GEMV_BW=1`. - Its premises have read: - the int8 round trip keeps P147's quality gate: I8ATTN GO, `p147-a2000-1`; - the kernels beat cuBLAS bf16 at Mixtral's shapes: P149 box K, K_GO, `p149k-5090-8`, a microbench premise. - The served read (2026-10-20): P149 box Q+S, `p149-5090-2`, read FLIP on one RTX 5090 with quality at decode GO at both gates. One request decodes 1.16× faster (the smallest of four ABBA pairs) and sixteen requests 1.018× (`e4b.serve.p149.int8-attn.mixtral.5090.2026-10-20`, one host). - Why it stays off by default: the token-pinned routing qualifier read QUALIFIER_VOID. S0, fed its own stream, differed from it at 5 forced tokens over 1,100 decode steps in every block. So the default-flip PR is held for a routing-matched read, which needs a later registration. [P147](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p147/RESULTS-p147.md), [P149](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p149/RESULTS-p149.md). Calibrated int4 attention is not licensed on the default. On Qwen3-30B-A3B, `E4B_SERVE_ATTN_INT4_CALIB=1` decodes one request 1.154× as fast (1.238× with the int4 lm_head too). The mean NLL bias stays inside K8's budget, but only 92.9 % of next-token argmaxes agree with the default at one window per pass (91.8 % with the head), against the 0.95 gate. It stays opt-in. A K8-style ppl gate alone would have passed it. Calibration is deterministic and takes 95–225 s a build, depending on the host. `e4b.serve.p125.int4-attn-calib-default.qwen3.5090.2026-10-09` ([P125](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p125/RESULTS-p125.md)). The launch-bound glue is cut, bitwise. - What changed. P127 fused or removed the small launches around the MoE routing and rotary on the default decode path: three host casts (#1448), and five grouped-nf4-gemm options (#526–#530) wired in by #1472, #1477 and #1482. - What it buys. On Qwen3-30B-A3B, with grouped-nf4-gemm 0.45.0 or later, the default now decodes one request 1.132× as fast as the stack before #1448 (226 → 256 tok/s, 0.52 ms saved per step), and 16 requests 1.016× as fast. With 0.44.0 or older, only Phase 1 runs. - What it costs. Tokens and per-step logits are bitwise identical on one schedule, and memory does not change. - In context. The saving is about the size of the launch-bound routing glue that P123 priced at 11 % of a 4.75 ms step. - Scope. The claim is the P127 delta at `7f044dd9`. `e4b.serve.p127.launch-bound-glue.qwen3.5090.2026-10-09` ([P127](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p127/RESULTS-p127.md)). Other opt-ins. `E4B_PAGED_DECODE_LOOKAHEAD=1` recovers the whole host gap between decode steps, but the gap is small: 1.0198× at one request (the smaller of two ABBA pairs), SLOWER against its 1.02 bar. `E4B_PAGED_LAST_LOGITS=1` projects only the final prompt position through the LM head (#1337). `e4b.serve.p118.decode-lookahead.qwen3.5090.2026-10-08`. Serving past VRAM: `E4B_PAGED_PLACEMENT=solver` keeps quality inside P153's bars (opt-in, quality only). On Qwen3-30B-A3B at 8.0 GiB VRAM / 3.0 GiB DRAM (one RTX 5090, lane P233), teacher-forced quality against the same server at `all-vram` reads SOLVER_SANE at group 1 and 8: NLL gap +0.0021 and -0.00005 nats, argmax agreement 0.974, KL 0.0039 and 0.0037. The TRIPWIRE fired, since the agreement is below its 0.98 look-bar. The read's written look finds the disagreement smaller in every window than the all-vram server's own chunked-prefill floor and unbiased in NLL. It is an opt-in placement, never a default, and no speed is read (decode at `solver` is eager). `e4b.serve.p233.solver-placement-quality.qwen3.5090.2026-10-24` ([P233](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p233/RESULTS-p233.md)). The per-row expert GEMV at 2-8 decode tokens (`E4B_NF4_SMALLM_BW`: the default on Mixtral at 2-4 tokens; `1` on any family at 2-8). In a captured microbench of the expert call (one RTX 5090, lane P227), the per-row route runs Mixtral-8x7B's experts in a median 0.36 and 0.46 of K25's time at two and four tokens (upper bounds 0.3601 and 0.4655, PASS against 0.70), and Qwen3-30B-A3B's in a median 0.74 and 0.90 (upper bounds 0.7478 and 0.9047, MISS). `e4b.kernel.p227.per-row-gemv.mixtral.5090.2026-10-23`, `e4b.kernel.p227.per-row-gemv.qwen3.5090.2026-10-23` ([P227](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p227/RESULTS-p227.md)). Served on Mixtral-8x7B's default NF4 server (lane P234, 2026-10-24; measured: [`bench/p234/RESULTS-p234.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p234/RESULTS-p234.md), `e4b.serve.p234.nf4-smallm-bw.mixtral-nf4.5090.2026-10-24`): the switch decodes two requests 2.12452× and four requests 1.76481× as fast as unset (73.2 → 155.7 and 107.3 → 189.4 tok/s). One request is unchanged, with identical tokens; sixteen requests decode at the same speed (1.00013), with tokens that differ in some rows (reported). Quality at group 4 is within P110's bars on both texts. LICENSED for buckets 2 and 4, and the default there (see the defaults table): unset, Mixtral's NF4 store sends calls of 2 to 4 tokens through the per-row GEMV on the RTX 5090 only (sm_120 at 170 SMs, the part read) and at Mixtral-8x7B's expert shapes and 8-expert top-2 routing (same-shape, same-routing checkpoints inherit it unread), so served tokens at two and four requests change there; `E4B_NF4_SMALLM_BW=0` is the way back. Calls of 5 to 8 tokens stay opt-in (`=1`). P231's earlier read of the same question was FUNCTION_FAIL, on its sixteen-request token-identity check, whose admission ramp runs the switched buckets. The tile table over 4 programs. `E4B_INT4_TILE_PROGRAMS=4` splits the one-launch cumsum tile table over 4 programs (#1433, grouped-nf4-gemm #524). On Qwen3-30B-A3B int4 it makes the captured 64-row step about 15.5 % faster with identical tokens: P126 read DEFAULT_ON_4 on attempt 2, under its Amendment 1. It is the default (`auto`) since #1476; see the defaults table. `e4b.serve.p126.tile-programs.qwen3-int4.5090.2026-10-09` ([P126](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p126/RESULTS-p126.md)). Qwen3-30B-A3B's licensed int4 stack (calibrated int4 experts and attention, folds, router epilogue) passes the K8 gate on both texts loaded by fingerprint (expert pack `sha256:0c9955a9…`; wikitext −0.05275, c4val1 −0.06622 ppl). On the bo7 census (the bo6c pack) it decodes ×2.067 at B=1 (238.1 tok/s) and ×2.602 at B=16 (1327.5 tok/s) against e4b's own NF4 control on the same box. `e4b.serve.p55x.qwen3.all-calibexp-streamed-64k.k8.2026-09-22`, `e4b.serve.census.bo7.qwen3.b1.5090.2026-09-05`, `e4b.serve.census.bo7.qwen3.b16.5090.2026-09-05` ([bo7](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/hybrid-g9/throughput-20260904/bo7/RESULTS.md)). Against vLLM, vLLM is ahead: 1.087 at B=1 and 1.396 at B=16 (P58: one box, identical prompt ids; vLLM 0.30.0 on Qwen's GPTQ-Int4 checkpoint against e4b's int4 stack). The B=16 gap is mostly the experts (P86); K19 and K23's lean glue have since taken the int4 B=16 step to 0.905× and a further 0.950×. At the 0.58.0 release (P235, P58's harness on another box, one RTX 5090 at a 470 W limit), vLLM 0.32.0 / e4b int4 reads 1.038 at B=1 and 1.203 at B=16, vLLM still ahead, with every distribution proven to be its published release; it sits beside P58's figures and is not divided into them, since the box differs. `e4b.serve.h2h.vllm-0.32.0.p235.qwen3.b1.5090.2026-10-24`, `e4b.serve.h2h.vllm-0.32.0.p235.qwen3.b16.5090.2026-10-24` ([P235](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p235/RESULTS-p235.md)). `e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b1.5090.2026-09-22`, `e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b16.5090.2026-09-22`, `e4b.serve.p86.qwen3.b16.kernel-census-vs-vllm-0.30.0.5090.2026-10-01` ([P58](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p58/RESULTS-p58.md)). The other families, each against its own NF4 control on the same box: - Granite: the licensed stack decodes ×1.341 at B=1 and ×1.160 at B=16. `e4b.serve.census.bo7.granite.b1.5090.2026-09-05` - OLMoE and Mixtral: the position is NF4; their int4 recipes fail the second text. `e4b.serve.census.bo7.olmoe.b1.5090.2026-09-05`, `e4b.serve.census.bo7.mixtral.b1.5090.2026-09-05`, `e4b.serve.p44.olmoe.two-text-k8.2026-09-19` - gpt-oss-20b: the native MXFP4 store is licensed by KL from the dequantised reference (0.0019 nats, against 0.0222 for an NF4 requant). `e4b.serve.p44.gptoss.store-r12.kl-vs-bf16.2026-09-19` - Qwen3.6-35B-A3B (hybrid): the paged state matches transformers' own; graphs decode 2.02× (16 requests) and 3.28× (one) eager, and with flash-linear-attention and causal-conv1d installed the Gated DeltaNet kernels add 1.135× / 1.118× at no measurable quality cost. `e4b.serve.p97.qwen36-hybrid-paged-state.5090.2026-10-02`, `e4b.serve.p101.qwen36-hybrid-decode-graphs.5090.2026-10-03`, `e4b.serve.p105.qwen36-gdn-kernels.5090.2026-10-03`, `e4b.serve.p106.qwen36-gdn-kernel-quality.5090.2026-10-03` - Gemma-4: no serving position. The paged path is at parity with transformers' own perturbations, but the served NF4 stack is 1.077 nats/token from the bf16 checkpoint, a cost in the early expert layers that calibration does not rescue (measured-private) and a graded store map does not bring under the bar ([P52](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p52/RESULTS-p52.md)). `e4b.parity.gemma4.p108.paged-vs-floor.5090.2026-10-03`, `e4b.serve.p44.gemma4.nf4-vs-bf16.2026-09-19`, `e4b.quality.gemma4.calibration-refuted` Parity behind the serving defaults. Paged decode is indistinguishable from the model's own attention on Granite, gpt-oss-20b and Qwen3-30B-A3B (measured-private; [SERVING-PARITY](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/SERVING-PARITY.md)), read against each model's routing-flip floor, never zero. Served T > 1 paths are inside the shipped KL bar against T = 1 decode (KL ≤ 0.0169, top-1 ≥ 0.946), and the int8 expert activation step and the bf16 router-weight cast are indistinguishable from their controls. `e4b.parity.granite.paged-vs-own-attention`, `e4b.parity.gptoss.paged-vs-own-attention`, `e4b.parity.qwen3.paged-vs-own-attention`, `e4b.parity.moe-routing-flip-floor`, `e4b.serve.p68.qwen3.t-gt-1-vs-t1.within-bar.5090.2026-09-24`, `e4b.serve.p64.qwen3.b1.expert-int8-step.5090.2026-09-24`, `e4b.serve.p70.qwen3.b1.router-weight-cast.5090.2026-09-25`. --- #### Defaults and the reads behind them | default | where | way back | read | chunked LM loss where fp32 logits would reach 1 GiB | `enable_fast_train` | `E4B_CHUNKED_LM_LOSS=0` | `e4b.train.chunked-lm-loss.auto.default-decision.5090.2026-10-05` | double-quantized expert absmax | `enable_fast_train` | `E4B_ABSMAX_DQ=0` or `absmax_dq=False` (offload, batched, residency and NVMe engines need fp32) | `e4b.train.absmax-dq.qwen3.5090.2026-10-05`, `e4b.train.absmax-dq.packed-4k.5090.2026-10-07` | prebound Triton launches | e4b, grouped-nf4-gemm | `E4B_TRITON_PREBIND=0`, `GNF4_TRITON_PREBIND=0` | `e4b.train.prebind.qwen3.5090.2026-10-05` | PyTorch's reentrant checkpoint | `enable_fast_train` (a `Trainer` with `gradient_checkpointing=True` re-enables Hugging Face's) | `E4B_CKPT_OFFLOAD=0` | `e4b.train.ckpt-flavour.default.5090.2026-10-07` | chunked held-out loss | `enable_fast_train` | `E4B_CHUNKED_EVAL_LOSS=0` | `e4b.train.chunked-eval-loss.packed-4k.5090.2026-10-07` | expert combine over row chunks | `enable_fast_train` | `E4B_COMBINE_CHUNK=0` | `e4b.train.combine-row-chunks.packed-4k.5090.2026-10-07` | fused q/k/v training projection (Qwen3-MoE, NF4 attention with adapters; 0.868–0.906 of the step on two RTX 5090 hosts (medians of two draws), 13.9–14.6 % fewer launches, about 24 MB; `disable_fast_train` restores the three projections) | `enable_fast_train` | `E4B_TRAIN_FUSE_QKV=0` | `e4b.train.p129.fused-qkv.qwen3.5090.2026-10-10`, `e4b.train.p129.fused-qkv.second-host.5090.2026-10-10` | bucketed LoRA-delta padding, compact | grouped-nf4-gemm | `NF4_QLORA_PAD_BUCKETS=0`, `NF4_QLORA_COMPACT_BUCKETS=0` | `e4b.train.pad-buckets.auto.default-decision.5090.2026-10-06`, `e4b.train.compact-buckets.packed-4k.5090.2026-10-07` | bucketed CUDA-graph decode (sm_89+) | `serve_paged` | `E4B_PAGED_GRAPHS=0` | `e4b.serve.p109.decode-graphs-vs-eager-default.qwen3.5090.2026-10-03`, `e4b.serve.p110.graph-arithmetic-quality.qwen3.5090.2026-10-03` | estimate-sized slots | `serve_paged` | `E4B_PAGED_MAX_SEQS=16` | `e4b.serve.sc2e.64-slots-default-buckets.qwen3.5090.2026-10-07` | one graph per decode step | `serve_paged` | `E4B_PAGED_BUCKETS=1,2,4,8,16` | `e4b.serve.sc2e.64-slots-buckets-auto.qwen3.5090.2026-10-07`, `e4b.serve.p117.wide-bucket-quality.qwen3.5090.2026-10-08` | K19's lean dispatch on prefill rows (`auto`: wherever the installed K19 takes `scatter=` / `gather_div=`; bitwise) | serving | `E4B_PREFILL_LEAN_DISPATCH=0` | `e4b.serve.p130.prefill-lean-dispatch.qwen3.5090.2026-10-10` | the decode folds on prefill rows (`qwen3_moe` only; changes prefill arithmetic, read at parity) | `serve_paged` | `E4B_FUSE_PREFILL_GLUE=0` | `e4b.serve.p130.prefill-glue-speed.qwen3.5090.2026-10-10`, `e4b.serve.p130.prefill-glue-quality.qwen3.5090.2026-10-10` | one KV-table selection per step | decode graphs | `E4B_KV_STEP_SELECT=0` | `e4b.serve.p111.kv-step-select.qwen3.5090.2026-10-04` | programmatic dependent launch, capped at 8 rows | grouped-nf4-gemm | `GNF4_PDL=0` | `e4b.serve.p113.gnf4-pdl-capped.qwen3-int4.5090.2026-10-04`, `e4b.serve.p230.gnf4-pdl.qwen3-nf4.5090.2026-10-23` | bandwidth-targeted NF4 decode GEMV | grouped-nf4-gemm | `GNF4_GEMV_BW=0` | `e4b.serve.p116.gemv-bw.qwen3.5090.2026-10-07` | grouped small-M routes above T == 1 (K19, K23, K21, K25; K25 read on Granite, OLMoE and Qwen3's W16 step) | serving | `E4B_INT4_GROUPED_SMALLM=0`, `E4B_INT4_LEAN_GLUE=0`, `E4B_MXFP4_GROUPED_SMALLM=0`, `E4B_NF4_GROUPED_SMALLM=0` | `e4b.serve.p88.qwen3.int4.k19-b16.5090.2026-10-01`, `e4b.serve.p89.qwen3.int4.k23-lean-glue-b16.5090.2026-10-01`, `e4b.serve.p90.gptoss.mxfp4.k21-b16.5090.2026-10-01`, `e4b.serve.p96.nf4-families.k25-windowed-k8.5090.2026-10-02`, `e4b.serve.p121.k25-w16.qwen3.5090.2026-10-08` | the one-launch tile table above 256 routed rows, up to the size read (`next_pow2(E) × next_pow2(R) ≤ 128 × 512`; takes effect with a grouped-nf4-gemm release carrying #519's chunked table; the one-piece table read 1.44× slower (median of 256 captured replays)) | serving | `E4B_INT4_WIDE_TILES=0` | `e4b.serve.p122.wide-tiles-chunked.qwen3-int4.5090.2026-10-08`, `e4b.serve.p120.wide-tiles.qwen3-int4.5090.2026-10-08` | the one-launch tile table over 4 programs, inside the size read (`next_pow2(E) × next_pow2(R) ≤ 128 × 512`; takes effect with a grouped-nf4-gemm release carrying #524's `programs=`; Qwen3-30B-A3B's 64-row step about 15.5 % faster) | serving | `E4B_INT4_TILE_PROGRAMS=1` | `e4b.serve.p126.tile-programs.qwen3-int4.5090.2026-10-09` | attention projections of 17–64 decode rows on the int4 small-M GEMM, from the int4 grid instead of a cached bf16 copy (with the K16 route on; takes effect with a grouped-nf4-gemm release carrying #522's `block_m=`; Qwen3-30B-A3B's 64- and 32-row steps 3.2–3.8 % faster) | `Int4Linear` (serving) | `E4B_ATTN_INT4_WIDE=0` | `e4b.serve.p124.attn-int4-wide.qwen3-int4.5090.2026-10-09` | router weights cast to bf16 at ≤ 64 rows (`softmax_topk`) | fused router epilogue | `E4B_ROUTER_EPI_CAST=0` | `e4b.serve.p70.qwen3.b1.router-weight-cast.5090.2026-09-25` | the paged FP8 attention decode capped at 16 splits (`qwen3_moe` only; buckets 1, 2 and 4 move, 32 / 32 / 23 -> 16; bitwise-identical to the plan at contexts ≤ 1024; one request 1.044× as fast (the smallest of four ABBA pairs), 16 unchanged, measured on Qwen3-30B-A3B on an RTX 5090 (170 SMs); other `qwen3_moe` checkpoints (e.g. Qwen3-235B-A22B) and other cards (the cap moves B 1–6 on a PRO 6000, B 1–4 on a 4090) take it by family without a read of their own) | `serve_paged` | `E4B_ATTN_SPLIT_CAP=0` | `e4b.serve.p143.attn-split-cap.qwen3.5090.2026-10-19` | the int8 lm_head's 2 to 8 rows through grouped-nf4-gemm K37's `linear_i8` instead of `F.linear` (`qwen3_moe` at the 151936 x 2048 head only, where the int8 head is on; needs grouped-nf4-gemm 0.49.0 or later, else it resolves to 1, recorded, as it does with `E4B_PAGED_SPEC`; outputs change at 2 to 8 requests, within bf16's output rounding; 2, 4 and 8 requests 1.0311×, 1.0243× and 1.0165× as fast (the smallest of four ABBA pairs), inside P153's quality bars, measured on Qwen3-30B-A3B on an RTX 5090 (170 SMs); same-shape checkpoints inherit the default unread; other head shapes and families stay at 1; the served read is P232, `bench/p232/RESULTS-p232.md`) | `serve_paged` | `E4B_INT8_HEAD_ROWS=1` | `e4b.serve.p232.int8-head-rows.qwen3.5090.2026-10-24`, `e4b.kernel.p229.int8-head-rows.qwen3.5090.2026-10-23` | the lm_head on its per-row int8 grid, its one-row decode GEMV through grouped-nf4-gemm K36 (`qwen3_moe` at the 151936 x 2048 head only; needs grouped-nf4-gemm 0.48.0 or later, else it resolves off, recorded; one request 1.052× as fast (the smallest of four ABBA pairs), 16 requests 0.992, an observed cost not attributed, measured on Qwen3-30B-A3B on an RTX 5090 (170 SMs); quality read on Qwen3-30B-A3B @ad44e777, same-shape checkpoints inherit the default unread; other head shapes, e.g. Qwen3-235B-A22B's 151936 x 4096, stay off) | `serve_paged` | `E4B_INT8_HEAD_BW=0` | `e4b.serve.p153.int8-head-bw.qwen3.5090.2026-10-19` | K25's codebook decode by grouped-nf4-gemm K39's byte permute (`lut="prmt"`; `qwen3_moe` only; bit-identical to the select tree, every token identical on one schedule; needs grouped-nf4-gemm 0.50.0 or later and an NVIDIA device, else it resolves to the tree, recorded in `info["k25_lut"]`; sixteen requests 1.0925× as fast (the smallest of four ABBA pairs), one request unchanged, measured on Qwen3-30B-A3B on an RTX 5090 (170 SMs) at the `today` plan; other `qwen3_moe` checkpoints (e.g. Qwen3-235B-A22B) take it by family with no speed claimed; other families stay on the tree) | `serve_paged` | `E4B_K25_LUT=tree` | `e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20` | K25's tile plan `c1` under K39's decode (from 0.56.1; `qwen3_moe` only, where K25's decode resolved to `prmt`, else `today`, recorded in `info["k25_plan"]["source"]`; bit-identical to `today`'s plan, every token identical on one schedule; sixteen requests 1.4654× as fast (the smallest of four ABBA pairs), one request unchanged (0.9966), measured on Qwen3-30B-A3B on an RTX 5090 (170 SMs); other `qwen3_moe` checkpoints take it by family with no speed claimed; other families stay on `today`) | `serve_paged` | `E4B_K25_PLAN=today` | `e4b.serve.p161.k25-plan-c1.qwen3.5090.2026-10-20` | B=1 fused stack (fused q/k/v and three glue folds) on Qwen3-MoE; its router epilogue, the only fold that engages, on Qwen3.5/3.6-MoE; its three glue folds on Mixtral | `serve_paged` | `E4B_PAGED_FUSE_QKV=0`, `E4B_FUSE_T1_GLUE=0`, `E4B_FUSE_T1_GLUE_R2=0`, `E4B_FUSE_ROUTER_EPI=0` | `e4b.serve.p115.fused-stack-combined.qwen3.5090.2026-10-08`, `e4b.serve.p115.fused-stack-speed.qwen3.5090.2026-10-07`, `e4b.serve.fam.router-epilogue-speed.qw36.5090.2026-10-09`, `e4b.serve.p135.fold-speed.mixtral.5090.2026-10-18` | Mixtral's NF4 expert calls of 2 to 4 decode tokens (buckets 2 and 4) through K33's per-row bandwidth GEMV (`mixtral` only, on the RTX 5090 only (sm_120 at 170 SMs; any other part resolves off, named), at Mixtral-8x7B's expert shapes (28672, 4096) and (4096, 14336) and 8-expert top-2 routing only (same-shape, same-routing checkpoints inherit it unread; any other resolves off, named), on the all-resident collapse; two requests 2.12452× and four 1.76481× as fast, served tokens at two and four requests change, quality within P110's bars at group 4; 5 to 8 tokens stay opt-in with `E4B_NF4_SMALLM_BW=1`; wherever an explicit `1` would refuse, it resolves off, recorded in `nf4_smallm_bw["source"]`) | `serve_paged` | `E4B_NF4_SMALLM_BW=0` | `e4b.serve.p234.nf4-smallm-bw.mixtral-nf4.5090.2026-10-24` --- #### What changed Superseded and retired readings are not repeated here. The register keeps each with its successor or its reason, and [`STATUS-RECORD.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/STATUS-RECORD.md) keeps the dated account. The retractions most often met elsewhere: - The 13.47× training speedup is retired (`e4b.retired.13.47x-training-speedup`): transformers v5 fused the per-expert loop upstream, so roughly half the multiple is now upstream's work. - "vLLM 6.31× ahead" is retired (`e4b.retired.vllm-6.31x-ahead`): template prompts on a different box. - "The fp8 paged KV cache costs +0.047 ppl on Qwen3" is retired (`e4b.retired.fp8-kv-cost`): +0.0058 nats, below the model's own floor. - The pre-registered KL gate is retired (`e4b.retired.kl-gate-threshold`): its first measurement rejected NF4 experts, which this package ships. - Granite's "302 tok/s, ×1.59 with int4 experts" is retracted: those experts cost +0.063 ppl, over the 0.05-ppl gate. --- #### What is open - #344: Gemma-4 failed to load on 2 of 6 rented hosts; lane P55 did not reproduce it on the current loader, and `E4B_LOAD_SYNC_DEBUG=1` now binds a fault to a load stage. `e4b.open.gemma4-load-fault` - gpt-oss expert LoRA is opt-in and unmeasured: `ExpertsLoRA`'s unfused reference loop (`E4B_GPTOSS_EXPERT_LORA=1`, CPU-graded against a transformers-matching oracle); grouped-nf4-gemm's `ExpertsMxfp4LoRA` route is experimental. - `enable_batched_train`'s envelope (which shapes reach its kernel) is unmeasured; `batched_fallback_stats(model)` counts the fallback. - #728: the int4 attention projections' int8 activation step is unpriced, and so is the int8 step at B = 16. - The B=16 expert GEMV runs at about 77 % of #564's byte roofline at real routing, about 1.4 ms/step of headroom, but no lever is licensed. `e4b.serve.p57.qwen3.b16.distinct-experts.5090.2026-09-22`, `e4b.serve.p61.qwen3.b16.expert-gemv-cost-split.5090.2026-09-23` - The residency cold-cost model under-predicts the pipelined engine's gather by 1.57–2.02× on a PCIe gen 4 link; the follow-up is filed against grouped-nf4-gemm's `kernel/cold_deadline.py`. `e4b.serve.p66.qwen3.pipelined-gather-over-cold-deadline.5090.2026-09-24` - P37's divergent pack: four builds on at least three boxes produce the licensed pack byte for byte; why P37's host built a different one is unexplained, and no open issue carries it. - Open register rows: reproducing the TR2 training receipt from published artifacts (`e4b.open.tr2-repro-gap`); int8-offload's best training eval, confounded by an evaluator offset (`e4b.open.int8-offload-confounded`). - Open campaigns: training #835, serving #846, single-stream decode #1313. P127 read: its launch-bound glue changes (Phase 1, #1448; Phase 2, #1472/#1477/#1482 with grouped-nf4-gemm #526–#530, first released in 0.45.0) decode one request 1.1321× as fast and 16 requests 1.0156× on Qwen3-30B-A3B on an RTX 5090, bitwise identical on one schedule (`e4b.serve.p127.launch-bound-glue.qwen3.5090.2026-10-09`, [results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/p127/RESULTS-p127.md)). #1472's kernel options take effect with grouped-nf4-gemm 0.45.0 or later. - Held-out loss parity against Unsloth and axolotl is unread. HP1 closed at its calibration: at 48 layers the shared evaluator's per-row zero-adapter gap set a tolerance of 0.044, against a 0.01 ceiling ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/hp1/RESULTS-HP1.md)). HP2 stopped at G1, because its instrument could not read the expected margin ([results](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/hp2/RESULTS-HP2.md)). Neither makes a parity, ahead or behind claim. The held-out losses in "Training against other frameworks" above are each framework's own in-loop reading, not a shared evaluator's. HPN ([design](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/hpn/HPN-DESIGN.md), $0 on an RTX A2000) reads where the evaluator-vs-in-loop gap comes from. - Older documents' debts: `POST_AUDIT_WORK_QUEUE.md` (Q1–Q4), `TRAIN_PLACEMENT_CERTIFICATE.md` (a scoped S10), `LAYOUT_FACTS.md` (training determinism UNKNOWN), and `support_matrix.md`'s footer hash, which no longer matches its bytes and is recorded, not fixed, because the file is anchored. --- #### Where the detail is - [`STATUS-RECORD.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/STATUS-RECORD.md): the dated narrative to 0.50.0, every superseded reading with its reason. Frozen; it is not appended to. - [`claims.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/claims.json) ([schema](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/claims-schema.md)): every claim's full text, conditions, status and evidence. - [`CHANGELOG.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/CHANGELOG.md): what each release changed and the reads behind it. - `bench//RESULTS-*.md`: each lane's receipt and verdict, linked from the positions above. - [`capabilities.json`](https://cerinamroth.com/ml/compatibility/#experts4bit-qlora), [`SOLUTIONS.md`](https://cerinamroth.com/ml/solutions/#experts4bit-qlora), [`SERVING.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/SERVING.md), [`SERVING-PARITY.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/SERVING-PARITY.md) and [`METHODOLOGY.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/METHODOLOGY.md): entry points, serving and method. ## `grouped-nf4-gemm` v0.51.0 — [docs/STATUS.md](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/STATUS.md) at [30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) (claims tiered in the public register: 11 confirmed · 36 measured · 3 measured-private · 1 projected · 2 open · 1 superseded · 4 retired) ### Status — what this kernel does, what changed, what is open As of 2026-10-24, `grouped-nf4-gemm` version 0.51.0. This page states the current position in each area, with the claim id behind each number; [`docs/claims.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/claims.json) holds every claim's full text and evidence. The dated narrative behind these positions, with the readings they replaced and why, is [`STATUS-RECORD.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/STATUS-RECORD.md). Evidence tiers: confirmed = pre-registered, stamped, blind confirmatory run; measured = a run with a committed receipt; measured-private = a real run whose receipt is in a private audit tree, so you cannot check it from here; projected = arithmetic, not a run. A retired claim was published and is now known wrong; a superseded one is true as measured, but a later row is the one to quote. To check whether a benchmark claim has been superseded or retired, read its `status` in [`claims.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/claims.json). --- #### What it does today The kernel. One Triton launch runs the grouped expert GEMM directly on 4-bit-packed weights: NF4 on the bitsandbytes layout, and native MXFP4 (OCP e2m1 + e8m0) on a checkpoint's exact released bytes. With fp32 accumulation it has never measured less accurate than dequantise-to-bf16-then-GEMM in any registered confirmatory cell (`gnf4.kernel.fused-more-accurate-than-dequant-bf16`; [confirmatories](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/kernel/RESULTS-v6-confirmatory.md)). | Item | measured | tier | claim ID | Decode, census MoE shapes vs the dequant path (sm_86) | 1.16–2.73× median | confirmed | `gnf4.kernel.decode-speed-census` | Energy, J/token below baseline | 104 of 112 cells | confirmed | `gnf4.kernel.energy-104-of-112` | Real OLMoE QLoRA finetune, fused vs per-expert loop (prose) | 4.50× (4090), 4.75× (H100) | confirmed | `gnf4.kernel.e2e-training-real-prose` | vs Unsloth's own kernel, 4-bit-storage regime, decode | 1.70× (H100), 2.79× (4090) | confirmed | `gnf4.kernel.h2h-unsloth` | vs `torch._grouped_mm` on bf16, Qwen3-30B cell (RTX 5090) | 2.1–6.0×, on 3.56× fewer weight bytes | measured | `gnf4.kernel.sm120-census-vs-grouped-mm` | Training backward, one launch: gradient vs the exact per-expert loop (A2000, a correctness read) | ~0.0029 relative, inside the bf16 budget | measured | `gnf4.kernel.dgrad` | Fused training step with it vs the per-expert loop, Qwen3-30B-A3B, 48 layers (rented RTX A6000; experts4bit-qlora's receipt) | 2.52× (1.72× without it) | measured | `gnf4.kernel.dgrad-step.a6000.2026-08-06` Where it loses. Three limits on every number above: - Against a CUDA-graphed baseline the fused path loses at decode (0.949× on a 4090, 0.858× on an H100); what survives is the memory-traffic win at training shape on bandwidth-limited cards (`gnf4.kernel.graphed-baseline-decode-loses`). - Unsloth wins its own regime: 2.6–5.3× faster at prefill on an H100 with their bf16-resident kernel. The margin above is the 4-bit-storage regime (`gnf4.kernel.h2h-unsloth`). - Known losers: `top_k=1` cells are instance-unstable, and shapes under about 5 M weight elements lose outright; a dispatch floor routes them back to the dequant path (`gnf4.kernel.decode-speed-census`). ##### Decode and serving (the RTX 5090 is the primary target) - Single-row NF4 decode. Since 0.43.0 the bandwidth-targeted GEMV (`GNF4_GEMV_BW=auto`) serves Qwen3-30B-A3B's two expert shapes on GPUs with at least 160 SMs: K33 read its kernel time at 0.341× the dot-pad route's (`gnf4.kernel.k33-nf4-decode-gemv-bw.5090.2026-10-07`; [results](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/kernel/RESULTS-k33-nf4-decode-gemv-bw.md)), and served it decodes one request 1.2417× as fast, and 16 requests unchanged (`e4b.serve.p116.gemv-bw.qwen3.5090.2026-10-07`). Dot-pad covers the same two shapes, so it now runs only with `GNF4_GEMV_BW=0`; every other shape takes the scalar GEMV. The one smaller part it also serves is the RTX 4090 (sm_89, 128 SMs), where experts4bit-qlora's P133 read one request 1.5145× as fast and 16 unchanged, within P110's quality bar (`e4b.serve.p133.gemv-bw.qwen3.4090.2026-10-18`, run `p133-4090-2`); dot-pad stays off there. Since P134 it also serves Granite-3.1-3b-a800m's and OLMoE-1B-7B's expert shapes on >= 160-SM parts: on an RTX 5090 one request decoded 1.3979× and 1.5011× as fast, 16 unchanged, within P110's bar (`e4b.serve.p134.gemv-bw.granite.5090.2026-10-18`, `e4b.serve.p134.gemv-bw.olmoe.5090.2026-10-18`, run `p134-5090-1`, on a 400 W card). Since P141 it serves Mixtral-8x7B's too, at the plans P141's phase 1 selected on the card: one request decoded 2.4075× as fast (19.53 → 8.11 ms a step), 16 unchanged (0.9964), within P110's bar (`e4b.serve.p141.gemv-bw.mixtral.5090.2026-10-18`, experts4bit-qlora#1754, run `p141-5090-2`). - The decode anchor for Qwen3-30B-A3B on the RTX 5090 class is 7.37 ms/step ±4.2% (≈130–142 tok/s); quote the range, not a point (`gnf4.serve.decode-anchor-5090`). - Paged attention takes fp8 compute where fp8 can run (sm_89+) and f32 otherwise, and never downgrades an explicit request; both paths are supported (`gnf4.serve.m3-defaults-on`, `gnf4.serve.f32-arms-ran-fp8`). Windows, sinks, a custom scale and stride overrides serve Granite, Gemma-4 and gpt-oss (`gnf4.serve.fp8-paged-attn-windows-sinks-scale`). - Programmatic dependent launch is on for launches of at most 8 rows (`gnf4.kernel.k28-pdl-decode-chain.5090.2026-10-04`; served 1.0404× at one request, `e4b.serve.p113.gnf4-pdl-capped.qwen3-int4.5090.2026-10-04`). - int4-b32: the decode GEMV (`gnf4.serve.int4-b32-gemv`, measured-private); K16's small-M GEMM, 2.91× the bf16 dequant path on o_proj at M=16 and experts4bit-qlora's default for the attention projections (`gnf4.kernel.k16-smallm-int4-gemm.5090.2026-09-19`); K19's grouped small-M GEMM at 0.736× the served GEMV route at B=16 (`gnf4.kernel.k20-k19-plan-sweep.5090.2026-10-01`). NF4's K25 decodes with an exact select tree (`gnf4.kernel.k26-nf4-decode-ablation.5090.2026-10-01`), or, with `lut="prmt"` (K39, from 0.50.0; opt-in here), through byte-permute table lookups, bit-identical to the tree. experts4bit-qlora serves `prmt` by default on `qwen3_moe` since its P158, which read sixteen requests 1.0925× as fast and one unchanged on Qwen3-30B-A3B on an RTX 5090 (`e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20`). - K16's 32- and 64-row tiles (#522) serve experts4bit-qlora's attention projections of 17–64 rows by default since its P124 read. K34's census of 48 plans kept the shipped plan: it is the fastest at `qkv`. The one faster cell, `o` at 64 rows (0.794× with BLOCK_N 32, a grid-fill effect), is about 1 % of the served step and licenses nothing (`gnf4.kernel.k34-k16-wide-plan-census.5090.2026-10-09`; [results](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/kernel/RESULTS-k34-k16-wide-plan-census.md)). - Row-count invariance. A token decoded alone and inside a 16-, 17- or 160-token call gets the same bits from `gemv_int4_b32`, the NF4 dot-pad GEMV and `combine_rows` (`gnf4.kernel.int4-gemv-row-invariant.5090.2026-09-24`, `gnf4.kernel.nf4-dotpad-gemv-row-invariant.5090.2026-09-24`, `gnf4.kernel.combine-rows-row-invariant.5090.2026-09-24`); the grouped int4 GEMM and the scalar NF4 GEMV differ only in summation order (`gnf4.kernel.int4-grouped-gemm-reorder.5090.2026-09-24`, `gnf4.kernel.nf4-scalar-gemv-splitk-reorder.5090.2026-09-24`). ##### Training - The backward is one launch over the packed bytes, on by default (`gnf4.kernel.dgrad`; the step is in the table). - The grouped-LoRA delta pads by default. Calls of at least 16,384 routed rows are bucketed (0.42.0) and run as one compact autograd node (0.43.0); shorter calls, TC1's field recipe among them, keep the single padded block. Read on Qwen3-30B-A3B and one RTX 5090 under torch 2.12 and 2.8 (`e4b.train.pad-buckets.torch28.qwen3.5090.2026-10-06`). - The single-block ladder is on by default for fp32 adapters. `NF4_QLORA_SINGLE_LADDER` (#513, #514) puts the single block on ladder rungs so cuBLAS stops paying a new-shape host cost on nearly every call. Unset is `auto`: the ladder exactly when the adapters are fp32. With fp32 adapters it stepped 0.797 of the time on a host-bound box and 1.031 on a GPU-bound one, within the registered 1.05 (`e4b.train.single-ladder.field.5090.2026-10-08`, `e4b.train.single-ladder-auto.gpu-bound.5090.2026-10-08`). bf16 adapters, e4b's default, are untouched (1.002). `0` turns it off; `1` takes it for any adapter dtype. - The training GEMM route (`GNF4_TRAIN_GEMM=auto`) is grouped_mm on sm_90, dense off sm_90 for calls with at most 16 present experts, and the fused kernels everywhere else. ##### Correctness contracts - int64 offsets: every carrier of the expert-base promotion is observed past its own 2^31 boundary (#87, #374, #386; `gnf4.kernel.expert-offset-boundary.5090.2026-09-05`, `gnf4.kernel.word-boundary-wide-dotpad.5090.2026-09-23`, `gnf4.kernel.boundary-gathers-appenders.interp.2026-09-23`). - `combine_rows` and `reduce_partials` carry an accuracy contract, not a bitwise one; `reduce_partials` is bitwise the slot-order sum (#393; `gnf4.kernel.combine-rows-accuracy.5090.2026-09-23`, `gnf4.kernel.reduce-partials-slot-order.5090.2026-09-23`; [results](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/kernel/RESULTS-b393-combine-reduce-bitwise.md)). - The MXFP4 prefill combine returns the same bits on every call: 1 distinct output in 50 identical calls (#408; `gnf4.kernel.mxfp4-prefill-combine-ordered.a2000.2026-09-28`); the QLoRA fused combine was fixed the same way (#409). ##### Past VRAM, and provenance - Qwen3-235B-A22B decodes at 4.3–4.4 tok/s on 15.2 GB of VRAM from a 438 GB checkpoint. The law is per box, `t_token ≈ c_box + bytes/link`, with `c_box` 53.5–114.0 ms across seven hosts (`gnf4.flagship.235b-phaseB`; [results](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/bench/phase3/flagship/RESULTS-flagship-phaseB.md)). - The NVMe tier is a batch tier: at `S ≈ 3.45 GB/s` a fully cold 235B is ~2.3 s/token. It buys reachability and provenance, not latency (`gnf4.nvme.tier-batch-only`). - Pinned-tier sizing models PyTorch's power-of-two pinned allocator (#71 closed; `gnf4.kernel.k29-pinned-charge-cgroup-v2.5090.2026-10-04`), and the cold cost model's `link_eff` is measured per host (`gnf4.calib.link-efficiency.5090.2026-09-24`). - Provenance. gpt-oss-120b serves on its exact released MXFP4 bytes at ppl 26.72 against the shipped-precision reference 26.75, and QLoRA-trains at 9.82 GB peak with its frozen expert bytes hash-identical after training (`gnf4.mxfp4.serve-tax-deleted`, `gnf4.mxfp4.train-9.82gb`). The reference MXFP4 decode reproduces Kimi K3's declared reference exactly (`gnf4.k3.oracle-exact`). --- #### Defaults and the reads behind them Each default moved after a registered read; most reads are experts4bit-qlora's, on Qwen3-30B-A3B and one RTX 5090. | default | where | way back | read | fp8 compute where fp8 can run (sm_89+), f32 otherwise | paged decode attention | `GNF4_ATTN_COMPUTE=f32` | `gnf4.serve.m3-defaults-on` | dot-pad NF4 GEMV at its census shapes, ≥ 160 SMs; since 0.43.0 only under `GNF4_GEMV_BW=0` (the BW GEMV takes the same shapes first) | NF4 decode | `GNF4_GEMV_DOTPAD=0` | `gnf4.serve.m3-defaults-on` | bandwidth-targeted NF4 GEMV at Qwen3-30B-A3B's two shapes (≥ 160 SMs, and on 128-SM sm_89 parts: the RTX 4090) and at Granite-3.1-3b-a800m's, OLMoE-1B-7B's and Mixtral-8x7B's (≥ 160 SMs) | NF4 decode | `GNF4_GEMV_BW=0` (`1` forces it everywhere) | `e4b.serve.p116.gemv-bw.qwen3.5090.2026-10-07`; the RTX 4090: experts4bit-qlora P133 (#1695); Granite and OLMoE: experts4bit-qlora P134 (#1697); Mixtral: experts4bit-qlora P141 (#1754) | programmatic dependent launch, at most 8 rows | decode kernels | `GNF4_PDL=0`; `GNF4_PDL_MAX_ROWS=0` lifts the cap | `e4b.serve.p113.gnf4-pdl-capped.qwen3-int4.5090.2026-10-04` | int4-b32 split-K planned from N alone on every part | `int4_b32._plan` | none (a constant, `SPLITK_R_TERM_MAX_SMS`) | `gnf4.kernel.k30-splitk-r-term.l4.2026-10-05`, `gnf4.kernel.k32-splitk-r-term.a4000.2026-10-05` | grouped_mm on sm_90; dense off sm_90 at ≤ 16 present experts | training GEMMs | `GNF4_TRAIN_GEMM=fused` | `e4b.train.h2h.unsloth.qwen3.h100.2026-10-04.route-v2`, `e4b.train.dense-route.mixtral.5090.2026-10-04` | prebound Triton launches | training GEMMs | `GNF4_TRITON_PREBIND=0` | `e4b.train.prebind.qwen3.5090.2026-10-05` | one grouping per MoE layer pass | grouping | `GNF4_HOST_REUSE=0` | `e4b.train.host-reuse.qwen3.5090.2026-10-04` | pinned index ring outside capture | index transfers | `GNF4_PINNED_RING=0` | `e4b.train.host-syncs.qwen3.5090.2026-10-03` | prefill M-tile height from the group sizes | NF4 prefill | `GNF4_PREFILL_TILE_RULE=max` | `e4b.train.prefill-tile-rule.qwen3.5090.2026-10-03` | lean padded LoRA delta | grouped-LoRA delta | `NF4_QLORA_LEAN_DELTA=0` | `e4b.train.lora-delta-lean.qwen3.5090.2026-10-03` | bucketed padding at ≥ 16,384 routed rows (`auto`) | grouped-LoRA delta | `NF4_QLORA_PAD_BUCKETS=0` | `e4b.train.pad-buckets.auto.default-decision.5090.2026-10-06` | bucketed calls as one compact autograd node | grouped-LoRA delta | `NF4_QLORA_COMPACT_BUCKETS=0` | `e4b.train.compact-buckets.packed-4k.5090.2026-10-07` | the single-block ladder for fp32 adapters (`auto`, since 0.44.0) | grouped-LoRA delta | `NF4_QLORA_SINGLE_LADDER=0` (`1` for any dtype) | `e4b.train.single-ladder.field.5090.2026-10-08`, `e4b.train.single-ladder-auto.gpu-bound.5090.2026-10-08` --- #### What changed — retired, superseded, corrected The register keeps every retired and superseded claim with its reason or successor. The dated account is under [What changed](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/STATUS-RECORD.md#what-changed--retired-superseded-corrected) in `STATUS-RECORD.md`; the ones most often met: - sm_120 is not "parked": that roadmap line is retired (`gnf4.retired.sm120-parked`). - "4.67× vs the grouped-bf16 execution class" is superseded (`gnf4.kernel.comparators-v6-execution-class`): that backend never ran Unsloth's own kernel; quote the head-to-head. - #319 was a mislabelled test arm, not an f32 kernel defect. Its open row is retired (`gnf4.open.f32-compute-modes-triton34`); the old advice to gate lanes with `-k "f8dot or pf8"` excluded exactly the mislabelled arms. - Split-K on the NF4 dot-pad decode GEMV is refuted (K7), its claim retired (`gnf4.retired.splitk-gemv`). - The int4-b32 split-K R term is off on every part: its A2000 sweep is retired (`gnf4.serve.int4-b32-splitk-row-term.a2000.2026-09-10`), and K30 (L4) and K32 (RTX A4000) both read OFF. - Refuted designs stay closed: a fixed fraction-of-waterfall as a law, the cold-engine "free floor" premise (`gnf4.cold-engine.phase0-premise-refuted`) and expert prefetch (`gnf4.flagship.prefetch-closed-negative`). - Exact, but not levers: K14 (`gnf4.kernel.k14-smallm-int4-gemm-refuted.5090.2026-09-11`), K17's fused split-K reduce and K18's grouped expert GEMV (`gnf4.kernel.k17-fused-splitk-gemv.5090.2026-09-21`, `gnf4.kernel.k18-grouped-expert-gemv.5090.2026-09-22`) ship opt-in or dormant, never as defaults. --- #### What is open - #60: arena staging blocks ~30% of a training step; the next layer's rows are prefetchable (`gnf4.open.issues`). - The chunked tile table is read at 128 experts × 512 rows only (P122); larger tables stay opt-in in experts4bit-qlora. - The P127 decode options (0.45.0) were read together with experts4bit-qlora's launch-bound glue changes, not alone: one request 1.1321× as fast, sixteen 1.0156×, bitwise-identical tokens (`e4b.serve.p127.launch-bound-glue.qwen3.5090.2026-10-09`). Each option's own share is unread. - The multi-program tile table is read at 4 programs on Qwen3-30B-A3B int4 only (`e4b.serve.p126.tile-programs.qwen3-int4.5090.2026-10-09`); 8 programs was not licensed. - A per-shape plan for the small-M GEMM (BLOCK_N chosen so the grid fills the card) is unread served; K34 put it at 0.794× for one cell, about 1 % of the step. - The bandwidth GEMV is read at four families' expert shapes (Qwen3-30B-A3B, Granite-3.1-3b-a800m, OLMoE-1B-7B, Mixtral-8x7B) on an RTX 5090 and at Qwen3-30B-A3B's on an RTX 4090; other families, shapes and cards are unread. - The cold cost model: why two gen 4 x16 hosts read different `link_eff` is unmeasured, and `cold_dest="deadline"` still omits the hybrid tier's mixed-layer dispatch term. - `docs/context-budgets.md` is rung-one only; full-depth confirmation is pending (`gnf4.open.context-budgets-rung-two`). - `docs/cold-engine/STAGE3-SYNTHESIS.md` has one outstanding correction: quote no read count from it until gate 1 is re-run. - Every non-CUDA row is a port target: `PROJECTIONS-multiarch.md` is arithmetic, stamped before the silicon (`gnf4.projection.multiarch`, projected). --- #### Reading the numbers without getting them wrong - Quote the hardware and the baseline. Ratios move with the card, and almost every ratio here is against this project's own per-expert loop or the dequant path (bitsandbytes `dequantize_4bit` per active expert, then a bf16 matmul), not a third party's kernel, except the head-to-head that says so. - Benchmark on real text. Random token ids understate this kernel (2.75× against 4.50× on prose, on a 4090) (`gnf4.kernel.e2e-training-real-prose`). - Peak VRAM does not improve. The fused arms peak higher; only the transient held from forward to backward shrinks. - Quote the register that owns the number. A model-level figure (tok/s for a named model on a named card, a perplexity-gate verdict) is registered in [experts4bit-qlora](https://github.com/pjordanandrsn/experts4bit-qlora)'s `docs/claims.json`; quote it with that register's claim ID and status. A `measured-private` status there means not publicly reproducible, exactly as here, where three rows are: `gnf4.serve.int4-b32-gemv`, `gnf4.serve.gptq-pack-int4-b32`, `gnf4.serve.decode-glue-kernels`. --- #### Where the detail is - [`STATUS-RECORD.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/STATUS-RECORD.md): the dated narrative to 0.43.0, every superseded reading with its reason. Frozen; it is not appended to. - [`claims.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/claims.json) ([schema](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/claims-schema.md)): every claim's full text, conditions, status and evidence. - [`CHANGELOG.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/CHANGELOG.md): each release and the reads behind it; [`changelog.d/`](https://github.com/pjordanandrsn/grouped-nf4-gemm/tree/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/changelog.d) holds what has merged since 0.43.0. - `kernel/RESULTS-*.md` and the `RESULTS-*.md` under `bench/`: each lane's receipt and verdict. - [`capabilities.json`](https://cerinamroth.com/ml/compatibility/#grouped-nf4-gemm), [`SOLUTIONS.md`](https://cerinamroth.com/ml/solutions/#grouped-nf4-gemm), [`KERNEL_CONTRACT.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/KERNEL_CONTRACT.md) and [`TOLERANCE_CONTRACT.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/TOLERANCE_CONTRACT.md): entry points, layouts and the fidelity bound. rendered from experts4bit-qlora 0.58.1 · source: [pjordanandrsn/experts4bit-qlora@605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) · checked 2026-10-25 rendered from grouped-nf4-gemm 0.51.0 · source: [pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) · checked 2026-10-25 Sources and publication details The generated ML documentation surfaces on this site (`/ml/` and its sub-pages, `/ml/packages.json`, both `llms.txt` files) are derived from the repositories below at pinned commits; each repository's `docs/claims.json` at that commit is the only source for numbers. Pages with their own source are listed separately. - [pjordanandrsn/experts4bit-qlora@605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) (2026-10-24) · package `experts4bit-qlora` 0.58.1 · PyPI 0.58.1 (published) · documents: [`docs/STATUS.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/STATUS.md), [`docs/claims.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/claims.json), [`docs/system-manifest.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/system-manifest.json) · registers: claims, status · synced 2026-10-25T00:58:32Z · [files at this commit](https://github.com/pjordanandrsn/experts4bit-qlora/tree/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) · [latest on main](https://github.com/pjordanandrsn/experts4bit-qlora) (unpinned; not the source of any fact rendered here) · sections: `#experts4bit-qlora` - [pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) (2026-10-24) · package `grouped-nf4-gemm` 0.51.0 · PyPI 0.51.0 (published) · documents: [`docs/STATUS.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/STATUS.md), [`docs/claims.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/claims.json), [`docs/system-manifest.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/system-manifest.json) · registers: claims, status · synced 2026-10-24T14:51:49Z · [files at this commit](https://github.com/pjordanandrsn/grouped-nf4-gemm/tree/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) · [latest on main](https://github.com/pjordanandrsn/grouped-nf4-gemm) (unpinned; not the source of any fact rendered here) · sections: `#grouped-nf4-gemm`