Receipts-driven engineering
Performance claims in ML tooling are, as normally published, unfalsifiable. A benchmark is run an unknown number of times, on unknown hardware, against a baseline configured by the person with the incentive; the number that survives is the number that flattered. We spent the last week applying the fix the medical world already knows — preregistration — to a GPU kernel, and the result is a public repository where every headline links to a protocol that provably existed before the data did.
The kernel is grouped-nf4-gemm: a Triton kernel that runs mixture-of-experts matmuls directly on 4-bit-packed weights — one launch for all active experts, the codebook decoded in registers inside the GEMM mainloop, no dequantize-to-bf16 round trip. Both 16-entry codebooks ship: NF4 on the bitsandbytes packed format unchanged, and — since July 20 — MXFP4 (OCP e2m1), computing on a checkpoint's exact released bytes. MIT, public since July 13, 2026; pip install grouped-nf4-gemm since July 18, every wheel carrying a signed provenance attestation. This page is about the method as much as the kernel; the method is the part that transfers.
The discipline
Each claim goes through the same pipeline, and the pipeline has no discretionary exits:
- Freeze — the code under test is committed; the archive shipped to test hardware is built from that commit and no other.
- Preregister — a protocol file states the devices, the run counts, the exact pass/fail bars, and the reduction procedure. It also states, in advance, what will be reported without bars and why.
- Stamp — the protocol is anchored with OpenTimestamps into the Bitcoin blockchain. After this moment, the bars cannot be quietly moved.
- Run blind — fresh cloud hardware is provisioned after the stamp; three fresh-process repetitions per device; the property suite runs alongside.
- Reduce mechanically — a committed script reads the bars from the stamped file and prints PASS or FAIL per criterion. The script is smoke-tested on synthetic fixtures on both sides of every bar before the stamp, so the verdict is arithmetic, not judgment.
- Publish either way — a no-tune clause: failures are reported at the same volume as wins, with no re-runs until the code actually changes. Stamped documents are never edited; corrections go in separate errata files.
Five failures, then a pass
Six confirmatories have run against this kernel. The first five failed as registered, and each failure is a published document that names what was wrong:
- v1 caught the per-shape configuration table overfitting the very census it was tuned on.
- v2 validated the replacement — a single universal constant — but only partially, on the terms it registered.
- v3 found that an architectural premise we believed (an SM-count-conditional config) was measurement noise, and named the kernel's two real loss classes:
top_k=1models (instance-unstable in both directions, 0.47–1.12×) and tiny experts (lose outright). It also established a methodology rule we now treat as law: latency-bound cells support paired claims only. - v4 shipped a dispatch floor and caught its own regression at the measurement boundary.
- v5 fixed that and came back clean on the reference device — and still published as not-confirmed because one contended-hardware noise cell missed its band.
- v6 — a rewrite of the prefill mainloop that moves the quantization codebook from per-element memory gathers into register shuffles — passed all five registered criteria: 1.39–1.54× the prior mainloop on every census cell, median 1.50×, with the compute-bound regime moving from a documented weakness to a barred win.
A green verdict means something different when the record shows five red ones published at full volume first.
Registering predictions against ourselves
The strongest protocol in the set was designed to hurt. The kernel's flagship demo streams a 235-billion-parameter mixture-of-experts model from system RAM over PCIe, using the fused kernel as the only expert compute — the model decodes at the physical limit of the bus on under 16 GB of VRAM. The obvious objection: maybe the baseline we beat was just a slow implementation. So we registered the objection as a prediction — that bitsandbytes' own CUDA dequantization kernel would also hide under the PCIe copy shadow, which would have collapsed our speed advantage to zero — stamped it, and ran it blind.
The prediction was refuted. The standard path reaches 40% of the bus ceiling; the fused kernel reaches 93–94% on the same machine. Against the strongest fair comparator, the advantage is 2.33× on throughput and 2.21× on energy per token, and it is architectural: per-expert dequantize-plus-matmul compute outlasts the copy window no matter how fast the dequant kernel is. We would have published the opposite finding under the same stamp. That is the entire point.
The doctrine runs forward as well as backward. When the repository's cross-hardware throughput projections met later measured evidence that a term was missing — a per-token serialization cost the pure-bandwidth model omitted, which grows more binding as the link gets faster — the fix was an erratum that predicts its own falsification. The original stamped rows are never edited; a dated, re-stamped addendum registers the corrected model and its narrower bands, states which measurements it revises and which it leaves alone, and pre-commits the expectation that a future run on the faster hardware lands inside the revised band and below the original. Both outcomes were registered before that hardware was touched. A projection that names the physics it was missing and stakes a falsifiable prediction on the correction is the method working, not the method failing.
The second codebook: computing on the checkpoint’s own bytes
e2m1 — the MXFP4 codebook gpt-oss ships its experts in — is also a sixteen-entry
table, so the same in-register-decode mainloop serves it by table swap. The point is
provenance: the kernel computes on the exact bytes the lab released, no requantization,
which makes two things measurable that normally are not. First, the conversion tax —
requantizing gpt-oss-120b to NF4 costs a measured +9.4% perplexity (KL 0.066 against the
shipped-precision reference); computing natively, the stamped serve run lands at the reference
number and the tax is deleted. Second, the weights themselves become verifiable:
verify_provenance re-hashes the checkpoint’s expert byte ranges against a
served or trained arena from the artifact alone, and the stamped training run — a
120-billion-parameter QLoRA at 9.82 GB peak VRAM on native bytes — closes with
144/144 sha256(file) == sha256(loaded) == sha256(post-train): the frozen base is
byte-identical after fine-tuning. Same pipeline as everything above — preregistered,
stamped, published either way. A gather loader for the DeepSeek-lineage per-expert layout is
included and verified against Kimi K2; every K3-specific number stays unclaimed until the
weights drop and the oracle re-adjudicates against K3’s own dequantization reference.
The v6 mainloop also picked up its smallest-card replication on July 20: an out-of-sample report-only leg on a 26-SM RTX A2000 lands 7 of 8 paired prefill cells inside the confirmed band (the eighth 0.9% below the floor, quantified in the report) — the register-LUT rewrite is bracketed from 26 to 170 SM with zero retune.
What the method catches that review does not
Two findings from this arc exist because of the discipline, not despite it.
The baseline lottery. A pre-stamp dress rehearsal on a second cloud instance showed the baseline — not our kernel — swinging ~25% between hosts, while the fused kernel held within 0.2 ms everywhere. Bars written against a baseline ratio would have made the verdict a coin flip on which host the confirmatory drew. The registered protocol therefore made its primary criterion a same-instance paired ratio that both sides share, and disclosed the lottery as a finding. Most published benchmarks never discover this about themselves, because nothing forces them to run the same protocol twice before believing it.
The wire law. Four registered attempts at expert prefetching all failed — and the fourth failure produced an equation. Speculative fetching at hit rate H moves (2−H)× the bytes; on a bus already ~78% utilized, the measured loss matched that arithmetic to about 1%. The program closed not because we got tired, but because the failure mode became a law with the constants filled in: break-even needs a predictor above 0.95, and this model family tops out at 0.93. Negative results with mechanisms are worth more than positive results without them.
The predictor ceiling, five families
The wire law rests on one number — the routing-predictability ceiling that puts speculative prefetch out of reach. That raised a sharper question: how predictable is a mixture-of-experts router intrinsically, and is the ceiling a property of the model or of the probe that measures it? We built a capacity ladder — a linear probe through six-layer attention probes, extended by a stamped amendment when the first ladder ran short — and, under a committed reducer that emits one of four verdicts and is smoke-tested on both sides of its plateau criterion before any data, measured the ceiling across five model families.
It is family-dependent, and the dependence has a shape. Low-expert-count families pin cleanly at the first data volume — gpt-oss-20b (32 experts) at 0.83, IBM Granite (40) at 0.90, OLMoE (64) at 0.91, each returning the reducer's model-limited verdict on the first pass. Both 128-expert families refuse to converge: three successive data doublings on Qwen3-30B lifted the plateau to 0.845 without ever certifying a ceiling, and gpt-oss-120b tracks the same arc one doubling behind. The honest one-line finding is that expert count, not top-k alone, governs how measurable routing predictability even is — and every value observed or extrapolated sits below the 0.95 the wire law needs, so the negative prefetch result generalizes past the one model that produced it. Exploratory tier; the ladder, the receipts, and the reducer that prints each verdict are in the repository.
The numbers, tiered
Every number below carries its evidence tier — the same scoping the repository enforces. confirmed means a stamped blind protocol; measured means a real benchmark at exploratory tier; projected means arithmetic from measured constants, not yet run on that hardware.
| Claim | Result | Tier |
|---|---|---|
| MoE decode vs the bitsandbytes dequant path (sm_86 reference devices) | 1.16–2.73× at median across the model census; energy below baseline in 104 of 112 cells; numerical error below baseline in every cell measured | confirmed |
| Prefill after the v6 mainloop rewrite | 1.39–1.54× the prior mainloop on every census cell; 1.14–2.78× vs dequant with all three large gate-projections above 1.15× | confirmed |
| 235B model, real checkpoint, ≤16 GB VRAM | coherent generation at 4.3–4.4 tok/s on 15.2 GB, replicated across five machines; synthetic pipeline at 102–103% of the measured link ceiling | confirmed |
| vs bitsandbytes' own CUDA dequant kernel, identical pipeline | 2.33× throughput, 2.21× energy — the registered contrary prediction refuted | confirmed |
vs the grouped-GEMM class unsloth's MoE backend rides (grouped_gemm.ops.gmm, dequant inside the timed path as 4-bit storage requires) | faster on every census cell — decode median 4.67×, prefill median 3.02× | measured |
| Consumer hardware (RTX 4090, sm_89) | 1.18–3.72× vs the dequant path across the census — best card tested; same code, zero retuning | measured |
| Routing-predictability ceiling across five MoE families (capacity ladder, reducer-adjudicated) | family-dependent, 0.75–0.91; low-expert-count families pin cleanly (0.83 / 0.90 / 0.91, model-limited), both 128-expert families data-unpinnable (Qwen3-30B ≥0.845 after three doublings, gpt-oss-120b ≥0.787) — all below the 0.95 prefetch break-even | measured |
| 235B on a desktop (gen4 x16, ~150 GB system RAM) | ≈3 tok/s ceiling by the waterfall arithmetic | projected |
Known losers, published at the same volume: top_k=1 models are instance-unstable in both directions; experts under ~5M weights lose outright and are auto-routed back to the standard path; the smallest-expert census shape remains below parity at one regime (~0.6×). The full loss ledger is in the repository README.
Why this lives here
Cerin Amroth's disclosure practice runs on a doctrine: honest severity, no overclaim, evidence a third party can check without trusting us. This project is the same doctrine pointed at performance engineering. A benchmark whose protocol was stamped into a blockchain before the data existed is the performance-claim equivalent of a signed advisory — it converts "trust me" into "check me." The repository — protocols, stamps, raw receipts, reducers, failures and all — is at github.com/pjordanandrsn/grouped-nf4-gemm.