experts-4bit

Large Mixture-of-Experts LLMs, fine-tuned and served on accessible NVIDIA GPUs.

experts-4bit (e4b) is 4-bit quantization of the fused expert weights in Mixture-of-Experts models, built to compose with the existing bitsandbytes 4-bit stack — it inherits the ecosystem instead of forking it. Recompute-in-backward keeps fine-tuning memory flat as expert count grows, so the same modest GPU that serves a model can also adapt it.

The problem

MoE won the open-weights frontier — and outgrew the GPUs. Qwen3-, Mixtral-, and DeepSeek-class architectures activate few parameters per token but must hold every expert resident, so expert weights dominate the memory bill: a 30B-class MoE wants 20+ GB for weights alone, while accessible NVIDIA workstation and consumer cards ship with 8–16 GB. Fine-tuning stacks optimizer state, gradients, and activations on top. The losers are local and private deployments — teams whose data cannot leave the building are priced out of exactly the models built for them.

What e4b does

• 4-bit quantization of fused MoE expert weights
  Composable with the bitsandbytes 4-bit stack (quantization types, double-quant, existing kernels) rather than a parallel format.
• Recompute-in-backward for fine-tuning
  Dequantized experts are recomputed inside the backward pass instead of materialized, keeping training memory flat as expert count grows.
• Generalizes to N-bit — ExpertsNbit
  Shipping in the kit today: the same fused stack at selectable precision — nf4/fp4 (4-bit packed), int8/fp8 (8-bit blockwise), bf16/fp16 (passthrough) — with a test-pinned fidelity ordering (fp16 < bf16 < int8 < fp8 < nf4 < fp4 reconstruction error), so the precision knob is a measured trade, not a vibe. What each mode does and doesn't promise: the support matrix.
• Upstream-first
  Submitted to bitsandbytes as PR #1965 — deliberately a clean, reviewable 4-bit diff with 59 tests including composition probes. The N-bit generalization is staged as the follow-up.
• Serves past VRAM — four instruments, honestly tiered
  The QLoRA kit’s serving side (as of 0.6.x): enable_fast routes frozen-expert inference through the fused kernel (measured 3.65× at bs=1); pipelined residency keeps each layer’s hottest experts VRAM-resident and streams the cold tail from pinned host RAM (K is config — empty set is pure streaming, all experts is fully resident); the cold engine computes the cold tail on the host CPU instead (bit-exact decode, correctness-tier until its fast AVX2 kernel lands); layer-granular expert offload covers training and serving past VRAM. Five architectures including gpt-oss run end-to-end on real weights. The measured law that makes residency work: pick hot sets from a routing histogram, not by index — informed top-8 on a 128-expert Gemma-4 layer is 6% of the pool yet covers half of all routed selections, and decode gain tracks that coverage on every architecture tried (+19% to +120% over the all-cold floor at equal VRAM; receipts in the repo).
• Meets practitioners where they are
  Integrations target the fine-tuning stacks people already run: an axolotl expert_offload plugin is open as PR #3797 (unsloth next), and a public QLoRA kit, experts4bit-qlora (on PyPI, with a trusted-publishing provenance attestation on every release), exercises the stack end-to-end.

In production

e4b is not a lab demo. It serves a Qwen3-30B-class MoE today from a single RTX A2000 (12 GB) on shared infrastructure that also carries unrelated production GPU workloads.

• Per-generation receipts
  Every run logs tokens, peak VRAM, and full version fingerprints to an append-only ledger.
• Golden-canary regression pinning
  Warm-ups verify bit-identical reference outputs before the endpoint takes traffic.
• Scale-to-zero scheduling
  VRAM is reclaimed when idle and re-warmed on schedule — small-GPU economics enforced in practice.

Measured

Everything below ran on a single RTX A2000 (12 GB) in a NAS's PCIe 3.0 ×8 slot — not a tuned bench rig. Sources are public: the axolotl integration PR and the QLoRA kit's METHODOLOGY.md.

Seed-matched A/B — axolotl expert_offload plugin

The controlled experiment: OLMoE-1B-7B QLoRA, identical seed and data order in both arms (axolotl PR #3797).

Peak VRAM −57%, load footprint −77%, and the loss curves overlay — convergence is preserved. Throughput cost is the per-block H2D copy, ~+11% s/step uncontended on this card. Per-step JSONL, charts, config, and the A/B runner are published in ab-telemetry/; the axolotl-native re-run — the same A/B through axolotl train on the PR branch, including DDP arms — is in ab-telemetry/axolotl-ab/.

Fine-tuning — QLoRA over frozen NF4 experts

Quality holds: QLoRA on the frozen NF4 experts improves OLMoE's held-out Alpaca eval loss 1.4813 → 1.0290. The streaming loader never materializes the bf16 model in CPU or GPU RAM (verified under a 3 GB container RAM cap).

Serving — decode on the same 12 GB card

Adapters serve over the exact NF4 base they trained against — no GGUF/AWQ re-quantization shifting the error surface. The 26B/30B rows decode on a card where resident decode simply OOMs.

Honest caveat: e4b is a memory technology, not an energy one. On a GPU that already fits the model, 4-bit costs 1.2–2.3× energy per token (NF4 is storage-only; the GEMM still runs in bf16, plus dequant). The win appears when memory is the binding constraint — then it is the difference between running and not, and up to 4.4× lower energy per token from the batching the freed memory unlocks.

Attacking that caveat: the sibling kernel project, grouped-nf4-gemm, deletes the dequantize-then-matmul round trip entirely — the expert GEMM runs directly on the NF4 bytes. Versus that standard path it decodes 1.16–2.73× faster with energy below baseline in 104 of 112 confirmatory cells, adjudicated under protocols timestamped before the data existed. The methodology writeup: Receipts-driven engineering.

As of July 18 the two ship as one: pip install "experts4bit-qlora[fast]" then enable_fast(model) routes the frozen expert projections through that fused kernel — measured 3.65× over the reference per-expert loop at batch-1 decode on the sm_86 reference card, with automatic fallback to the exact reference path for training and for any module the kernel does not cover. The caveat is no longer a caveat when the model fits; it is a flag you turn on.

Roadmap

Land experts-4bit upstream; follow with the staged ExpertsNbit upstream diff. On the serving side: the fast host decode kernel for the cold engine (AVX2), a format-agnostic residency rail so the kernel’s MXFP4 lane rides the same engines, and routing-informed hot sets promoted from bench driver to library API. The upstream diffs and production serving telemetry are available on request.

Contact

jordan@cerinamroth.com  ·  independent consulting: jordananderson.work