# experts-4bit, the 2026 narrative — Cerin Amroth Source: https://cerinamroth.com/research/experts-4bit/ [Home](https://cerinamroth.com/)[ML](https://cerinamroth.com/ml/)[JWST](https://cerinamroth.com/jwst/)[Security](https://cerinamroth.com/policy/)[Contact](https://cerinamroth.com/#contact) This is a dated research narrative. For the current installable package, supported capabilities, API and active measurements, see [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/) (generated from its pinned source commit of 2026-10-24). Numbers on this page are quoted as they stood when it was written; the repository's `docs/claims.json` says which are still current. [Cerin Amroth Research](https://cerinamroth.com/) · [ML systems](https://cerinamroth.com/ml/) · experts-4bit [upstream PR](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1965) # experts-4bit Large Mixture-of-Experts LLMs, fine-tuned and served on accessible NVIDIA GPUs. MoE won the open-weights frontier and outgrew the GPUs. Qwen3-, Mixtral- and DeepSeek-class models activate few parameters per token but must hold every expert resident, so a 30B-class MoE wants 20+ GB for weights alone against the 8–16 GB on accessible cards — and fine-tuning stacks optimizer state on top. The losers are local and private deployments: teams whose data cannot leave the building are priced out of exactly the models built for them. experts-4bit (`e4b`) quantizes the fused expert weights that stock [bitsandbytes](https://github.com/bitsandbytes-foundation/bitsandbytes) silently skips, composing with its 4-bit stack rather than forking it — so it inherits the ecosystem. Recompute-in-backward keeps training memory flat as expert count grows. ## What it does - Fine-tunes and serves past VRAM Layer-granular expert offload covers both. Pipelined residency keeps each layer’s hottest experts VRAM-resident and streams the cold tail from pinned host RAM; the cold engine computes that tail on the host CPU instead (bit-exact, correctness-tier until its AVX2 kernel lands). Five architectures including gpt-oss run end-to-end on real weights. The dial that matters: pick hot sets from a routing histogram, not by index — informed top-8 on a 128-expert Gemma-4 layer is 6% of the pool yet covers half of all routed selections. - Trains through the fused kernel — `enable_fast_train` The sibling kernel is differentiable as of its 0.2.4, so fine-tuning no longer leaves the 4-bit format in the backward pass. On Qwen3-30B-A3B at all 48 layers with offload and gradient checkpointing: 0.95% from the reference path on held-out eval loss at 0.768× its peak VRAM — it uses less — with 192/192 frozen expert tensors byte-identical afterward over 16.31 GB hashed, under a byte-flip control proving the check can fail. The [48-layer gate is published as FAIL](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/fused-train-gate/RESULTS-fused-train-gate.md) beside its passing re-run: our first fixture trained one repeated sequence and the reference arm memorised it. - Generalizes to N-bit — `ExpertsNbit` The same fused stack at selectable precision — `nf4`/`fp4`, `int8`/`fp8`, `bf16`/`fp16` passthrough — with a test-pinned fidelity ordering, so the precision knob is a measured trade rather than a vibe. [The support matrix](https://github.com/pjordanandrsn/experts4bit-qlora#storage-modes-the-support-matrix). - Upstream-first, and it meets people where they are Submitted to bitsandbytes as [PR #1965](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1965) — a deliberately clean 4-bit diff whose test file collects 59 tests on a CPU+CUDA box (22 functions, device-parametrized; 34 on a CPU-only machine) including composition probes. The N-bit generalization is staged behind it. An axolotl `expert_offload` plugin is open as [PR #3797](https://github.com/axolotl-ai-cloud/axolotl/pull/3797), and the public QLoRA kit [experts4bit-qlora](https://github.com/pjordanandrsn/experts4bit-qlora) ships on PyPI with a provenance attestation on every release. ## Measured Everything below ran on a single RTX A2000 (12 GB) in a NAS's PCIe 3.0 ×8 slot — not a tuned bench rig — except where noted. Sources are public: the axolotl PR and the kit's [METHODOLOGY.md](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/METHODOLOGY.md). ### Seed-matched A/B — the controlled experiment OLMoE-1B-7B QLoRA, identical seed and data order in both arms. | Config | Loaded GPU | Peak GPU | Held-out eval loss | Experts resident | 4.70 GB | 6.00 GB | 1.6448 → 1.2213 | Experts offloaded | 1.08 GB | 2.60 GB | 1.6448 → 1.2270 Peak VRAM −57%, load footprint −77%, and the loss curves overlay — convergence is preserved. The cost is the per-block H2D copy, ~+11% s/step uncontended. Per-step JSONL, charts and the A/B runner are published in [`ab-telemetry/`](https://github.com/pjordanandrsn/experts4bit-qlora/tree/main/ab-telemetry), including an axolotl-native re-run with DDP arms. ### Fitting and serving on one small card | Fused MoE | QLoRA peak with e4b | Without | Decode (v0 path — superseded, see below) | OLMoE-1B-7B | <8 GB (loads at 4.70) | 13.9 GB bf16 — OOM | 3.08 tok/s @ 4.86 GB | Gemma-4-26B-A4B | 8.47 GB (offload) | OOM | 0.43 tok/s @ 6.2 GB | Qwen3-30B-A3B | 7.16 GB (offload) | OOM | 0.22 tok/s @ 4.4 GB The 26B/30B rows decode on a card where resident decode simply OOMs, and adapters serve over the exact NF4 base they trained against — no GGUF/AWQ re-quantization shifting the error surface. Quality holds: QLoRA on frozen NF4 experts improves OLMoE's held-out Alpaca eval loss 1.4813 → 1.0290, and the streaming loader never materializes the bf16 model in CPU or GPU RAM (verified under a 3 GB container cap). The decode column is the v0 offload path and the engine has moved past it. It is the oldest data on this page, kept because it is the honest floor — what you get with no tuning at all on a 12 GB card in a NAS. The pipelined engine (`enable_pipelined_residency`) supersedes it, and picking hot experts from a routing histogram lifts it further: on the same A2000, informed hot sets take OLMoE from 3.448 tok/s naive to 4.099 tok/s at K=4 (+19%, coverage 7.3% → 29.1%). Bigger cards move it much further — Qwen3-30B-A3B reaches 1.65 tok/s in 4.41 GB with prefetched offload and 5.19 tok/s resident at 20.04 GB — but those are different hosts (A5000 / A100 80GB) and, because offload decode is transfer-bound, none of these rates transfers between boxes. That is why they are reported per host rather than as a headline number. ## What these numbers do not say - The 7.16 GB and 9.13 GB figures are different configurations, not a failed replication Worth being exact, because “unreplicated” would be the wrong word. 7.16 GB was measured on the A2000 at `SEQ=128`; the 9.13 GB figure is a newer card at seq 192, and a five-dataset matrix at seq 512 put it at 9.126–9.153 GB, flat to 0.3%. Activation memory scales with sequence length, so those are answers to different questions — nobody has re-run the original configuration on a comparable card, which is a gap in coverage rather than a contradiction. What all of them agree on: the same model needs 27.30 GB unoffloaded, so offload is the difference between running and not. Quote each figure with its sequence length. - `enable_fast`’s 3.65× is the expert projection, not end-to-end generation We tried to reproduce it end-to-end and could not. An interleaved A/B with both paths warmed and the flag toggled in place read medians of 1.006× and 1.026× at batch 1 over 24-token decode — against a 20–22% noise floor that dwarfs the effect. Two earlier attempts were discarded by their own controls. Per-token overhead outside the expert GEMM dominates at batch 1, so a real kernel win need not survive Amdahl there. The end-to-end decode question is open, not settled in our favour; testing it requires timing the expert GEMM directly. - The energy penalty applies to storage-only 4-bit — the fused path reverses it Stated as a flat “memory technology, not an energy one,” this would now be out of date. It is true of plain NF4 storage: the GEMM still runs in bf16 and you pay dequant, so on a GPU that already fits the model 4-bit costs 1.2–2.3× energy per token. But e4b ships the fused path for both directions — `enable_fast` for inference and `enable_fast_train` for training — and through [grouped-nf4-gemm](https://cerinamroth.com/ml/grouped-nf4-gemm/) that path deletes the dequantize round trip entirely: energy below baseline in 104 of 112 confirmatory cells at 1.16–2.73× decode, and fused training measured 1.65× faster per step at 0.768× the peak memory. So: a capacity technology that becomes an energy and speed win wherever the fused kernel covers the module, and stays a pure capacity trade where it doesn't. The training speedups are single same-process A/B pairs — a direction, not a barred band. - The residency gain belongs to the host, not the model Informed hot sets buy +19% to +120% decode over the all-cold floor at equal VRAM — on bandwidth-limited hosts. A hot expert only pays when the transfer it avoids costs more than the resident path's own overhead, so the same dial measures ~+40% on a thin-link A2000 and roughly nothing on a fat-PCIe L40S. - The training matrix is complete at 20/20 — and one of its gates was withdrawn All twenty registered cells ran: two 30B-class models, five structurally different synthetic datasets, both arms, 200 steps each. Parity holds everywhere, on both metrics the protocol actually registers — |Δ final train loss| ≤ 0.05 and median step-wise |Δ| ≤ 0.05. Worst cell across twenty is 0.03653. Cost: 1.52–1.81× faster per step at 0.75–0.81× peak VRAM and 0.86–0.92× the energy, with the memory ratio flat to three decimals within each model — a residency policy, not a data-dependent effect. An earlier version of this page cited “bit-exactness 192/192 in all ten” and a 0.00723 worst case. Both are withdrawn. That gate hashed `getattr(module, "gate_up_proj")`, which under expert offload is a 0-element placeholder — so it compared `sha256(b"")` with itself, 192 times per cell, and could not have failed. The 192 counted tensors, not bytes. And 0.00723 was the eval delta, which is not the quantity either protocol registers. The second model’s ten cells use the corrected gate and hash 12,846,366,720 bytes per cell — a figure derived from the checkpoint config before any receipt existed, matching to the byte — with zero empty tensors and a byte-flip control that must fire before a cell may start. The registered “best result” question is adjudicated, and the honest answer is neither path wins. Stronger than that: the same cell re-run on a second RTX 4090, same seeds and config, flips the winner — reference by 5.83% on one host, fused by 6.00% on the other. Both clear the registered 0.99% separability threshold by roughly six-fold and the sign still reverses, so that threshold — inherited from the first model’s zero-adapter floor — understates real run-to-run variance here by about an order of magnitude. A per-dataset winner is not a measurable quantity at this sample size. The fused path reproduces the reference; it does not compete with it. [Second-model receipts](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/flagship-matrix-model2/RESULTS-flagship-matrix-model2.md) · [first ten, with the withdrawal](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/bench/flagship-matrix/RESULTS-flagship-matrix.md) · [the stamped protocol](https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/PREREG-flagship-matrix-model2.md). ## In production, and next e4b is not a lab demo. It serves scheduled Qwen3-30B-class jobs today from a single RTX A2000 (12 GB) on shared infrastructure that also carries unrelated production GPU workloads, and the current release is what is running — the live endpoint logged its most recent generations under 0.6.5 on the day it shipped. Three mechanics make that safe on one small card: every run appends tokens, peak VRAM and full version fingerprints to a ledger; each warm-up re-generates a golden canary and records its sha256, so a drifted endpoint is caught before it takes traffic; and a reaper returns the VRAM when idle, so a 30B model shares a GPU with unrelated jobs instead of holding it. Production decode currently runs 0.38 tok/s at ~4.46 GB — above the v0 table row above, which is the point of keeping that row labelled as a floor. Next: land experts-4bit upstream, then the staged `ExpertsNbit` diff (the class ships in the package today; what is queued is the upstream-facing version). On the serving side the cold engine still wants its AVX2 host kernel — today it takes bnb's AVX-512 kernel where `avx512f` is present and otherwise a pure-torch decode, which is correct but slow — and the residency rail wants to go format-agnostic so the MXFP4 lane rides the same engines. On the training side, timing the expert GEMM directly: the one measurement that would settle the open decode question above. Upstream diffs and production serving telemetry available on request. ## Contact [jordan@cerinamroth.com](mailto:jordan@cerinamroth.com) · independent consulting: [jordananderson.work](https://jordananderson.work) experts-4bit — ML-systems efficiency research by Cerin Amroth Research. · [Home](https://cerinamroth.com/) · [ML systems](https://cerinamroth.com/ml/) · [bitsandbytes PR #1965](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1965) © 2026 Cerin Amroth Security Research, LLC (d/b/a Cerin Amroth Research) · CERIN AMROTH SECURITY RESEARCH™ Current documentation: [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/) · [ML systems](https://cerinamroth.com/ml/) · this page was previously served at `/ml/experts-4bit/`.