experts-4bit

Large Mixture-of-Experts LLMs, fine-tuned and served on accessible NVIDIA GPUs.

MoE won the open-weights frontier and outgrew the GPUs. Qwen3-, Mixtral- and DeepSeek-class models activate few parameters per token but must hold every expert resident, so a 30B-class MoE wants 20+ GB for weights alone against the 8–16 GB on accessible cards — and fine-tuning stacks optimizer state on top. The losers are local and private deployments: teams whose data cannot leave the building are priced out of exactly the models built for them.

experts-4bit (e4b) quantizes the fused expert weights that stock bitsandbytes silently skips, composing with its 4-bit stack rather than forking it — so it inherits the ecosystem. Recompute-in-backward keeps training memory flat as expert count grows.

What it does

• Fine-tunes and serves past VRAM
  Layer-granular expert offload covers both. Pipelined residency keeps each layer’s hottest experts VRAM-resident and streams the cold tail from pinned host RAM; the cold engine computes that tail on the host CPU instead (bit-exact, correctness-tier until its AVX2 kernel lands). Five architectures including gpt-oss run end-to-end on real weights. The dial that matters: pick hot sets from a routing histogram, not by index — informed top-8 on a 128-expert Gemma-4 layer is 6% of the pool yet covers half of all routed selections.
• Trains through the fused kernel — enable_fast_train
  The sibling kernel is differentiable as of its 0.2.4, so fine-tuning no longer leaves the 4-bit format in the backward pass. On Qwen3-30B-A3B at all 48 layers with offload and gradient checkpointing: 0.95% from the reference path on held-out eval loss at 0.768× its peak VRAM — it uses less — with 192/192 frozen expert tensors byte-identical afterward over 16.31 GB hashed, under a byte-flip control proving the check can fail. The 48-layer gate is published as FAIL beside its passing re-run: our first fixture trained one repeated sequence and the reference arm memorised it.
• Generalizes to N-bit — ExpertsNbit
  The same fused stack at selectable precision — nf4/fp4, int8/fp8, bf16/fp16 passthrough — with a test-pinned fidelity ordering, so the precision knob is a measured trade rather than a vibe. The support matrix.
• Upstream-first, and it meets people where they are
  Submitted to bitsandbytes as PR #1965 — a deliberately clean 4-bit diff whose test file collects 59 tests on a CPU+CUDA box (22 functions, device-parametrized; 34 on a CPU-only machine) including composition probes. The N-bit generalization is staged behind it. An axolotl expert_offload plugin is open as PR #3797, and the public QLoRA kit experts4bit-qlora ships on PyPI with a provenance attestation on every release.

Measured

Everything below ran on a single RTX A2000 (12 GB) in a NAS's PCIe 3.0 ×8 slot — not a tuned bench rig — except where noted. Sources are public: the axolotl PR and the kit's METHODOLOGY.md.

Seed-matched A/B — the controlled experiment

OLMoE-1B-7B QLoRA, identical seed and data order in both arms.

Peak VRAM −57%, load footprint −77%, and the loss curves overlay — convergence is preserved. The cost is the per-block H2D copy, ~+11% s/step uncontended. Per-step JSONL, charts and the A/B runner are published in ab-telemetry/, including an axolotl-native re-run with DDP arms.

Fitting and serving on one small card

The 26B/30B rows decode on a card where resident decode simply OOMs, and adapters serve over the exact NF4 base they trained against — no GGUF/AWQ re-quantization shifting the error surface. Quality holds: QLoRA on frozen NF4 experts improves OLMoE's held-out Alpaca eval loss 1.4813 → 1.0290, and the streaming loader never materializes the bf16 model in CPU or GPU RAM (verified under a 3 GB container cap).

The decode column is the v0 offload path and the engine has moved past it. It is the oldest data on this page, kept because it is the honest floor — what you get with no tuning at all on a 12 GB card in a NAS. The pipelined engine (enable_pipelined_residency) supersedes it, and picking hot experts from a routing histogram lifts it further: on the same A2000, informed hot sets take OLMoE from 3.448 tok/s naive to 4.099 tok/s at K=4 (+19%, coverage 7.3% → 29.1%). Bigger cards move it much further — Qwen3-30B-A3B reaches 1.65 tok/s in 4.41 GB with prefetched offload and 5.19 tok/s resident at 20.04 GB — but those are different hosts (A5000 / A100 80GB) and, because offload decode is transfer-bound, none of these rates transfers between boxes. That is why they are reported per host rather than as a headline number.

What these numbers do not say

• The 7.16 GB and 9.13 GB figures are different configurations, not a failed replication
  Worth being exact, because “unreplicated” would be the wrong word. 7.16 GB was measured on the A2000 at SEQ=128; the 9.13 GB figure is a newer card at seq 192, and a five-dataset matrix at seq 512 put it at 9.126–9.153 GB, flat to 0.3%. Activation memory scales with sequence length, so those are answers to different questions — nobody has re-run the original configuration on a comparable card, which is a gap in coverage rather than a contradiction. What all of them agree on: the same model needs 27.30 GB unoffloaded, so offload is the difference between running and not. Quote each figure with its sequence length.
• enable_fast’s 3.65× is the expert projection, not end-to-end generation
  We tried to reproduce it end-to-end and could not. An interleaved A/B with both paths warmed and the flag toggled in place read medians of 1.006× and 1.026× at batch 1 over 24-token decode — against a 20–22% noise floor that dwarfs the effect. Two earlier attempts were discarded by their own controls. Per-token overhead outside the expert GEMM dominates at batch 1, so a real kernel win need not survive Amdahl there. The end-to-end decode question is open, not settled in our favour; testing it requires timing the expert GEMM directly.
• The energy penalty applies to storage-only 4-bit — the fused path reverses it
  Stated as a flat “memory technology, not an energy one,” this would now be out of date. It is true of plain NF4 storage: the GEMM still runs in bf16 and you pay dequant, so on a GPU that already fits the model 4-bit costs 1.2–2.3× energy per token. But e4b ships the fused path for both directions — enable_fast for inference and enable_fast_train for training — and through grouped-nf4-gemm that path deletes the dequantize round trip entirely: energy below baseline in 104 of 112 confirmatory cells at 1.16–2.73× decode, and fused training measured 1.65× faster per step at 0.768× the peak memory. So: a capacity technology that becomes an energy and speed win wherever the fused kernel covers the module, and stays a pure capacity trade where it doesn't. The training speedups are single same-process A/B pairs — a direction, not a barred band.
• The residency gain belongs to the host, not the model
  Informed hot sets buy +19% to +120% decode over the all-cold floor at equal VRAM — on bandwidth-limited hosts. A hot expert only pays when the transfer it avoids costs more than the resident path's own overhead, so the same dial measures ~+40% on a thin-link A2000 and roughly nothing on a fat-PCIe L40S.
• The training matrix is complete at 20/20 — and one of its gates was withdrawn
  All twenty registered cells ran: two 30B-class models, five structurally different synthetic datasets, both arms, 200 steps each. Parity holds everywhere, on both metrics the protocol actually registers — |Δ final train loss| ≤ 0.05 and median step-wise |Δ| ≤ 0.05. Worst cell across twenty is 0.03653. Cost: 1.52–1.81× faster per step at 0.75–0.81× peak VRAM and 0.86–0.92× the energy, with the memory ratio flat to three decimals within each model — a residency policy, not a data-dependent effect.

  An earlier version of this page cited “bit-exactness 192/192 in all ten” and a 0.00723 worst case. Both are withdrawn. That gate hashed getattr(module, "gate_up_proj"), which under expert offload is a 0-element placeholder — so it compared sha256(b"") with itself, 192 times per cell, and could not have failed. The 192 counted tensors, not bytes. And 0.00723 was the eval delta, which is not the quantity either protocol registers. The second model’s ten cells use the corrected gate and hash 12,846,366,720 bytes per cell — a figure derived from the checkpoint config before any receipt existed, matching to the byte — with zero empty tensors and a byte-flip control that must fire before a cell may start.

  The registered “best result” question is adjudicated, and the honest answer is neither path wins. Stronger than that: the same cell re-run on a second RTX 4090, same seeds and config, flips the winner — reference by 5.83% on one host, fused by 6.00% on the other. Both clear the registered 0.99% separability threshold by roughly six-fold and the sign still reverses, so that threshold — inherited from the first model’s zero-adapter floor — understates real run-to-run variance here by about an order of magnitude. A per-dataset winner is not a measurable quantity at this sample size. The fused path reproduces the reference; it does not compete with it.

  Second-model receipts · first ten, with the withdrawal · the stamped protocol.

In production, and next

e4b is not a lab demo. It serves scheduled Qwen3-30B-class jobs today from a single RTX A2000 (12 GB) on shared infrastructure that also carries unrelated production GPU workloads, and the current release is what is running — the live endpoint logged its most recent generations under 0.6.5 on the day it shipped. Three mechanics make that safe on one small card: every run appends tokens, peak VRAM and full version fingerprints to a ledger; each warm-up re-generates a golden canary and records its sha256, so a drifted endpoint is caught before it takes traffic; and a reaper returns the VRAM when idle, so a 30B model shares a GPU with unrelated jobs instead of holding it. Production decode currently runs 0.38 tok/s at ~4.46 GB — above the v0 table row above, which is the point of keeping that row labelled as a floor.

Next: land experts-4bit upstream, then the staged ExpertsNbit diff (the class ships in the package today; what is queued is the upstream-facing version). On the serving side the cold engine still wants its AVX2 host kernel — today it takes bnb's AVX-512 kernel where avx512f is present and otherwise a pure-torch decode, which is correct but slow — and the residency rail wants to go format-agnostic so the MXFP4 lane rides the same engines. On the training side, timing the expert GEMM directly: the one measurement that would settle the open decode question above. Upstream diffs and production serving telemetry available on request.

Contact

jordan@cerinamroth.com  ·  independent consulting: jordananderson.work

Current documentation: experts4bit-qlora · ML systems · this page was previously served at /ml/experts-4bit/.