This is a dated research narrative. For the current installable package, supported capabilities, API and active measurements, see grouped-nf4-gemm (2026-09-05). Numbers on this page are quoted as they stood when it was written; the repository's docs/claims.json says which are still current.
Receipts-driven engineering
Performance claims in ML tooling are, as normally published, unfalsifiable. A benchmark is run an unknown number of times, on unknown hardware, against a baseline configured by the person with the incentive; the number that survives is the number that flattered. This project applies the fix medicine already knows — preregistration — to a GPU kernel, so every headline links to a protocol that provably existed before the data did.
grouped-nf4-gemm is a Triton kernel that runs mixture-of-experts matmuls directly on 4-bit-packed weights: one launch for all active experts, the codebook decoded in registers inside the GEMM mainloop, no dequantize-to-bf16 round trip. It serves both 16-entry codebooks — NF4 on the bitsandbytes packed format unchanged, and MXFP4 (OCP e2m1) on a checkpoint’s exact released bytes — and the NF4 path is differentiable, so fine-tuning never leaves the packed format either. MIT, pip install grouped-nf4-gemm, every wheel signed. The method is the part that transfers.
The discipline
Each claim runs the same pipeline, and the pipeline has no discretionary exits:
- Freeze — the code under test is committed; the archive shipped to test hardware is built from that commit.
- Preregister — a protocol names the devices, run counts, exact pass/fail bars, and reduction procedure, plus what will be reported without bars and why.
- Stamp — anchored via OpenTimestamps into Bitcoin. After this, the bars cannot be quietly moved.
- Run blind — fresh cloud hardware provisioned after the stamp; three fresh-process repetitions per device.
- Reduce mechanically — a committed script reads the bars from the stamped file and prints PASS or FAIL. It is smoke-tested on both sides of every bar before the stamp, so the verdict is arithmetic, not judgment.
- Publish either way — failures at the same volume as wins, no re-runs until the code changes. Stamped documents are never edited; corrections go in separate errata.
What the discipline caught
These findings exist because of the protocol, not despite it. Each one would have been invisible to ordinary review.
- Five red verdicts before a green one. The first five confirmatories failed as registered, and each is a published document naming what was wrong — a config table overfitting its own census, an architectural premise that was really measurement noise, a dispatch-floor regression, and a clean reference-device run still published as not-confirmed because one contended cell missed its band. The sixth passed all five criteria. A green verdict means something different when five red ones precede it.
- Gates that cannot fail are the real hazard. Building the differentiable path, three defects hid behind checks that passed vacuously: a bit-exactness comparison hashing empty tensors (under offload the module attribute is a zero-element placeholder, so it compared
sha256(b"")to itself and reported 192/192 exact while hashing nothing); two consecutive fixes that were never in the code under test; and a missingalpha/rfactor that halved every update while staying invisible to memory, hashes, and step-zero parity alike. Every gate now carries a positive control that must fire. The 48-layer verification is also published as FAIL beside its passing re-run — the reducer printed FAIL, our fixture was the thing that was wrong, and both runs ship in that order. The companion training matrix has since completed all 20/20 registered cells across two 30B-class models: loss parity passes everywhere on both metrics the protocol registers (|Δ final train loss| and median step-wise |Δ|, worst cell 0.03653 against a registered 0.05). Its first ten cells carried this very defect and their bit-exactness gate is withdrawn — an earlier version of this page cited that “192/192” as evidence two clauses after describing why it means nothing. The second model uses the corrected gate and hashes 12,846,366,720 bytes per cell, a figure derived from the checkpoint config before any receipt existed and matching to the byte. The registered “best result” question is adjudicated as not separable, and a cross-host replication strengthens that: the same cell re-run on a second RTX 4090 with identical seeds flips the winner, so a 6% margin is noise at this sample size. - We registered the strongest objection to our own headline — and it survived. The flagship demo streams a 235B model over PCIe with the fused kernel as the only expert compute. The obvious counter: maybe the baseline was just a slow implementation. So we registered that as the prediction — that bitsandbytes’ own CUDA dequant kernel would also hide under the copy shadow, collapsing our advantage to zero — stamped it, and ran it blind. Refuted: the standard path reaches 40% of the bus ceiling, the fused kernel 93–94% on the same machine. We would have published the opposite under the same stamp.
- The baseline lottery. A pre-stamp rehearsal showed the baseline swinging ~25% between cloud hosts while our kernel held within 0.2 ms everywhere. Bars written against a baseline ratio would have made the verdict a coin flip on which host we drew. The protocol therefore made its primary criterion a same-instance paired ratio, and disclosed the lottery as a finding. Most benchmarks never discover this about themselves, because nothing forces them to run twice before believing it.
- A negative result that became an equation. Four registered attempts at expert prefetching failed, and the fourth produced the mechanism: speculative fetching at hit rate H moves (2−H)× the bytes, and on a bus already ~78% utilized the measured loss matched that arithmetic to ~1%. Break-even needs a router predictor above 0.95. We then measured that ceiling across five model families — 0.787–0.91, family-dependent — and every plateau, including the 0.93 one-layer-lead ceiling the ladder was built to interrogate, sits below 0.95 — so the prefetch result generalizes past the one model that produced it. The program closed because the failure became a law with its constants filled in.
- A projection that named the physics it was missing. When measured evidence showed our cross-hardware throughput model omitted a per-token serialization cost, the fix was an erratum that predicts its own falsification: stamped rows untouched, a dated addendum registering the corrected model, narrower bands, and which measurements it revises. Both outcomes were registered before the faster hardware was touched.
The numbers
Every row carries its evidence tier. confirmed = a stamped blind protocol. measured = a real benchmark whose protocol predates its data in public git, without an OpenTimestamps anchor. projected = arithmetic from measured constants. The distinction is not decorative — it is the whole apparatus, so the newer lanes are labelled honestly rather than left to inherit the stamped tier’s credibility.
| Claim | Result | Tier |
|---|---|---|
| MoE decode vs the bitsandbytes dequant path (sm_86) | 1.16–2.73× median across the model census; energy below baseline in 104 of 112 cells; numerical error below baseline in every cell | confirmed |
| Prefill after the v6 register-LUT mainloop rewrite | 1.39–1.54× the prior mainloop on every census cell; 1.14–2.78× vs dequant. Bracketed 26→170 SM with zero retune | confirmed |
| 235B on ≤16 GB VRAM, real checkpoint | coherent generation at 4.3–4.4 tok/s on 15.2 GB, replicated across five machines — all on 45–55 GB/s datacenter links. Per-token rate is link- and host-dependent (t ≈ cbox + bytes/link, cbox 53.5–114.0 ms across seven hosts), so the rate does not transfer | confirmed |
| vs bitsandbytes’ own CUDA dequant kernel, identical pipeline | 2.33× throughput, 2.21× energy — the registered contrary prediction refuted | confirmed |
| MXFP4 serve — gpt-oss-120b on the vendor’s shipped bytes (receipts) | exact-chunk ppl 26.72 vs dequant-reference 26.75 — the requantization tax measured at 20b (+9.4%) is deleted at 120b | confirmed |
| MXFP4 train — 120b QLoRA, native bytes (receipts) | peak 9.82 GB VRAM; 144/144 expert tensors sha256-identical to the release before, during and after training | confirmed |
| Fused training vs reference — Qwen3-30B-A3B, all 48 layers, offload + grad checkpointing (receipts) | held-out eval 0.95% apart (0.337073 vs 0.340271) against a registered 0.05 band; the zero-adapter floor is 0.099%, so the trained gap sits within an order of magnitude of pure summation order. Peak VRAM 0.768× reference — it uses less. 192/192 frozen tensors byte-identical over 16.31 GB hashed, under a byte-flip control proving the check can fail | measured |
| NVMe tier — experts read from disk (ceiling · determinism) | per-box S ≈ 3.45 GB/s, which makes this a batch tier by construction: ~2.3 s/token for a fully cold 235B, ~7.5 s/token at K3 scale. Interactive use is not the claim. The arena itself is reproducible — two GPU generations, two hosts, two independent 438 GB downloads, 20,480/20,480 segment hashes identical | measured |
| vs the grouped-GEMM class unsloth’s MoE backend rides | faster on every census cell — decode median 4.67×, prefill 3.02×. Their kernel targets bf16-resident training, a job it is excellent at; this is the 4-bit-storage regime | measured |
| Consumer hardware (RTX 4090, sm_89) | 1.18–3.72× vs the dequant path — best card tested, same code, zero retuning | measured |
| Routing-informed hot sets, the residency dial (receipts) | +19–120% decode over the all-cold floor at equal VRAM on bandwidth-limited hosts. The gain belongs to the link, not the model: ~+40% on a thin-link A2000, ~0% on a fat-PCIe L40S | measured |
| 235B on a desktop (receipts) and on gen5 bare metal (receipts) | 2.60 and 3.92 tok/s real decode, greedy-identical. The gen5 box reached 0.55× of its link ceiling where slower hosts read 0.77× — which falsified the pure-bandwidth model exactly as pre-registered and forced the additive correction above | measured |
Known losers, published at the same volume: top_k=1 models are instance-unstable in both directions (0.47–1.12×); experts under ~5M weights lose outright and are auto-routed back to the standard path; the smallest-expert census shape stays below parity at one regime (~0.6×). Full loss ledger in the repository README.
Why this lives here
Cerin Amroth’s disclosure practice runs on a doctrine: honest severity, no overclaim, evidence a third party can check without trusting us. This is the same doctrine pointed at performance engineering. A benchmark whose protocol was stamped into a blockchain before the data existed is the performance-claim equivalent of a signed advisory — it converts “trust me” into “check me.” Protocols, stamps, raw receipts, reducers, failures and all: github.com/pjordanandrsn/grouped-nf4-gemm.
Current documentation: grouped-nf4-gemm · ML systems · this page was previously served at /ml/grouped-nf4-gemm/.