experts4bit-qlora

Train and serve Mixture-of-Experts models that do not fit in VRAM: fused 4-bit experts, QLoRA, CPU/NVMe offload, and fast inference on consumer NVIDIA GPUs.

Also discoverable as e4b, e4b-qlora, experts4bit, expertsnbit, experts-mxfp4. These are lookup-only aliases from the system manifest; experts4bit-qlora is the canonical project.

Direct answer

Use this package when load_in_4bit=True does not actually quantise a model's fused expert tensors. transformers v5 stores a MoE's experts as one fused three-dimensional parameter per layer rather than ordinary nn.Linear modules, so the standard bitsandbytes module walker can leave most of the expert weights in bf16 and the model still OOMs; this package quantises exactly that fused stack, fine-tunes it with per-expert LoRA, keeps experts in host RAM or on NVMe when they do not fit, and serves the result on one consumer NVIDIA GPU.

Use this when

• A Mixture-of-Experts model loaded with load_in_4bit still occupies bf16-sized VRAM and OOMs, because transformers stores its experts as one fused 3-D parameter per layer that bitsandbytes' nn.Linear walker never touches. — load_moe_4bit_streaming quantises exactly the fused expert stack on the way to the GPU (Experts4bit, the 4-bit face of ExpertsNbit), never materialising it in bf16; verify_moe_4bit(model, strict=True) raises if any stack is still high precision. solution page
• Fine-tune (QLoRA / LoRA) a fused-expert MoE whose expert weights PEFT and the bitsandbytes walker never see. — ExpertsLoRA wraps each frozen 4-bit expert stack with per-expert LoRA; load_moe_4bit_streaming installs it; enable_fast_train(model, dgrad=True) routes the step through grouped-nf4-gemm's grouped kernels, enable_batched_train(model) is the no-extras fallback; python -m experts4bit_qlora.train is the env-configured trainer. model_families lists exactly the families whose fast_train path is `supported` in training_support (the headline path, enable_fast_train, with a PASS receipt on real weights); every other path is stated per family there, never as a flat flag. solution page
• The quantised experts still do not fit the GPU: run a Mixture-of-Experts model larger than VRAM by keeping experts resident somewhere else and streaming them per layer. — load_moe_4bit_streaming(..., offload=True) homes each layer's frozen 4-bit experts in pinned host RAM and streams one layer at a time (prefetch=True overlaps); enable_dense_offload streams the dense side; hot_sets_from_profile picks the resident experts by routing frequency; enable_pipelined_residency trades spare VRAM for speed at serve time. solution page
• Experts do not even fit host RAM: serve or train from an on-disk expert arena (NVMe) with a pinned-DRAM hot tier. — Bake the experts into an arena with grouped-nf4-gemm (nvme_bake_nf4 for NF4, the relocation bake for native MXFP4), then enable_nvme_residency (serving), enable_mxfp4_nvme_residency (native MXFP4 serving) or enable_nvme_train_residency (training, gradient checkpointing required) serve each layer's cold experts from the arena while hot_rows stay pinned in DRAM. solution page
• Serve a large fused-expert MoE on one consumer NVIDIA GPU with the decode path measured against the model's own attention. — Two surfaces. The measured decode path is the paged runner (PagedModelRunner: paged fp8 KV cache and paged attention) with the experts on grouped-nf4-gemm's kernels -- NF4 experts on the grouped GEMM (enable_fast is the library entry point for that kernel; the harness attaches it through its residency engines), int4-b32 experts through enable_serve_experts_int4, and the opt-in calibrated int4 attention/head, fused glue rounds and router epilogue -- exercised by the in-tree bench harness bench/hybrid-g9/step_decomp.py, which produced the throughput and parity claims. The HTTP shim (python -m experts4bit_qlora.serve) is a reference-path deployment: load_moe_4bit_streaming with the experts streamed from pinned host RAM by default, stock model.generate, per-expert LoRA adapters hot-swapped per request; it attaches a kernel-backed engine only under E4B_RESIDENCY=pipelined and does not use the paged runner or enable_fast. solution page
• Work with models whose experts are released as native MXFP4 (gpt-oss, DeepSeek-V4): load them faithfully, keep the released bytes in an NVMe arena, train on them, and serve them without re-quantisation. — The loader dequantises MXFP4 (e2m1 blocks + e8m0 scales) bit-identically to the reference; enable_mxfp4_nvme_residency relocates the released bytes into an arena and serves them through grouped-nf4-gemm's native MXFP4 kernels; enable_nvme_train_residency trains against an MXFP4 arena; the native MXFP4 expert store for the paged engine is an opt-in lane. solution page

Do not use this when

• dense (non-MoE) models: use bitsandbytes directly
• a general-purpose serving engine or a vLLM replacement
• models that already fit in bf16 and need no offload: 4-bit here is a memory trade, not a speed-up, and on the measured comparator it cost energy (claim e4b.train.energy-honest.scoped-a2000)
• silent fallbacks: every enable_* returns a count or a non-empty handle list, or raises; callers must assert it
• operating systems other than Linux, or GPUs without CUDA

Install

Routes from docs/capabilities.json at the pinned commit; the first entry of the register is the primary route.

pip install "experts4bit-qlora[train]"

Which path?

Smallest verified example

Needs a CUDA GPU, network access and the checkpoint download; the strict verification is the observable result (it raises if any expert stack is still high precision). Syntax-checked in the repository's CI; executed on rented hardware lanes with receipts.

import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16, r=8, alpha=16, quant_type="nf4",
)
verify_moe_4bit(model, strict=True)   # raises RuntimeError naming the first high-precision expert stack

What it supports

Modes

inference, offload, quantization, serving, streaming, training

Formats

bf16, fp16, fp4, fp8, int4, int8, mxfp4, nf4

Model families

deepseek_v2; deepseek_v4 (Flash, Pro); gemma4_text; gpt_oss; granitemoe; mixtral; olmoe; qwen3_5_moe; qwen3_moe; qwen3_next

Operating system

Linux

Python

>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

Accelerator

NVIDIA CUDA GPU (bitsandbytes 4-bit); NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)

Public entry points

cli:python -m experts4bit_qlora.serve, cli:python -m experts4bit_qlora.train, experts4bit_qlora.engines.batched:enable_batched_train, experts4bit_qlora.engines.capture:capture_decode, experts4bit_qlora.engines.dense_offload:enable_dense_offload, experts4bit_qlora.engines.expert_profile:hot_sets_from_profile, experts4bit_qlora.engines.fast:enable_fast, experts4bit_qlora.engines.fast:enable_fast_train, experts4bit_qlora.engines.int4_attn_calib:enable_serve_attn_int4_calib, experts4bit_qlora.engines.int4_experts:enable_serve_experts_int4, experts4bit_qlora.engines.nvme_experts:enable_mxfp4_nvme_residency, experts4bit_qlora.engines.nvme_experts:enable_nvme_residency, experts4bit_qlora.engines.nvme_train:enable_nvme_train_residency, experts4bit_qlora.engines.offload:enable_expert_offload, experts4bit_qlora.engines.paged_runner:PagedModelRunner, experts4bit_qlora.engines.pipelined:enable_pipelined_residency, experts4bit_qlora.loader:load_moe_4bit_streaming, experts4bit_qlora.lora:ExpertsLoRA, experts4bit_qlora.verify:verify_moe_4bit, experts4bit_qlora:Experts4bit, experts4bit_qlora:ExpertsNbit, flag:E4B_EXPERT_PROFILE

Owns (system manifest)

• model loading (streaming loader, fused expert stacks, Experts4bit/ExpertsNbit bases)
• quantization orchestration (NF4/MXFP4 routes, int4 expert store, calibration drivers)
• adapters (ExpertsLoRA)
• training (trainer, fast/batched/hybrid paths, checkpointing)
• residency integration (host-RAM and NVMe engines bound to a loaded model)
• serving (paged runner, decode stack, HTTP shim)
• model-level measurement, quality gates and the claims register for model-level numbers

Related package

grouped-nf4-gemm — one system, two independently released packages: the runtime package depends on the kernel package through its optional `fast` extra; the kernel package never depends on the runtime package Dependency direction: experts4bit-qlora -> grouped-nf4-gemm.

Limitations — what it does not promise

• Unsupported model_type raises NotImplementedError; identity-expert families (LongCat-Flash) are refused by name
• 4-bit is a memory trade when the model already fits, and cost energy on the measured comparator (one card, one bitsandbytes development build): claim e4b.train.energy-honest.scoped-a2000
• Dense (non-MoE) models are not the target; use bitsandbytes directly
• enable_fast_train returns 0 (does not raise) when the kernel package is absent: assert the return value
• dgrad=True needs grouped-nf4-gemm>=0.7.0 (below the current [fast] floor in pyproject.toml, so any [fast] install has it); on older cuts it is downgraded with a RuntimeWarning
• Loss parity is measured on the registered criteria in docs/METHODOLOGY.md, not asserted for every dataset
• model_families is evidence-gated: a family is listed when its fast_train path -- the headline path, enable_fast_train -- PASSES the registered loss-parity criteria on real weights with a receipt in this repository (training_support is the per-path record; bench/train-parity-20260905/tp1/ is lane tp1, all six serving families). olmoe is confirmed on real weights and a registered text; qwen3_moe and gemma4_text carry tp1 PASSes beside their flagship matrices; mixtral entered on its tp1 fused PASS under offload; granitemoe entered on its corrected-counter re-run's PASS (its first attempt was a harness error, kept as its own row); gpt_oss is refused by every training enabler and stays out
• gpt_oss is not a training family here: the loader builds its experts bare (no ExpertsLoRA), and both enable_fast_train and enable_batched_train return zero on it -- REFUSED rows on tp1 (e4b.train.parity.tp1.gptoss.fused.2026-09-05, e4b.train.parity.tp1.gptoss.batched.2026-09-05). Attention-only QLoRA over its frozen experts trains (e4b.train.parity.tp1.gptoss.attn_only.2026-09-05); the only route that trains its experts is grouped-nf4-gemm's experimental mxfp4_qlora.ExpertsMxfp4LoRA under the mxfp4-moe-training-and-residency capability -- experimental, never licensed
• enable_batched_train falls back to the reference forward per call above _PAD_WASTE_LIMIT (engines/batched.py): the patch count stays positive while some or all layers run the per-expert loop on some steps. In the shipped code tp1 measured this fallback was uncounted, and three arms read VOID on exactly that -- OLMoE, Qwen3 and Gemma-4 (e4b.train.parity.tp1.olmoe.batched.2026-09-05, e4b.train.parity.tp1.qwen3.batched.2026-09-05, e4b.train.parity.tp1.gemma4.batched.2026-09-05); it engaged everywhere only on Mixtral's eight-expert and Granite's forty-expert shapes. As of the release carrying that bundle the fallback is countable: batched_fallback_stats(model) reports it (#402; those VOID rows are the reason it exists). Assert kernel engagement, not the return value alone
• enable_fast, enable_fast_train and enable_batched_train refuse a wrapper whose base violates the stock-epilogue contract (EpilogueContractError, #402) instead of silently computing act(gate)*up over a base that applies biases, clamps or another gating rule -- a refusal is loud, a wrong epilogue is not; gpt-oss is the family this protects, and DeepSeek-V4 supplies its epilogue through _apply_gate and is admitted
• Head-to-head against Unsloth's 4-bit MoE QLoRA path on one identical training problem (lane p38, 2026-09-05, one rented RTX 5090, Qwen3-30B-A3B, the clinical fixture tokenised once, seq 512, r 8 / alpha 16, identical targets, optimizer, batch, steps, precision and eval; bench/h2h-20260905/p38/): the shipped fused dgrad path with NF4 attention is faster per step, lower in peak VRAM and lower in energy per step than Unsloth on that box, and the two are comparable on held-out loss at sixty steps (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05, ...quality-n60) -- BUT the two-hundred-step curve favours Unsloth: its held-out loss is lower at the end of the curve while e4b's flattens from step sixty (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.curve-n200). Quote the position only beside that row. Candidate causes (the eval schedule, the checkpointing mode, the transformers/peft versions of the two stacks, the expert adapter's precision -- bf16 on this side, the loader passes the model dtype to ExpertsLoRA -- against fp32 on Unsloth's) are not established. One workload, one box, one family; nothing here licenses a path (tp1 does) and no general speed claim is made
• tp2/P40 (2026-09-06, one rented RTX 5090, receipts in bench/h2h-20260906/tp2/) extends the Unsloth head-to-head to all six families on one box and one fixture, and its rows are statuses, not one number: positions exist only for Qwen3-30B-A3B (s/step ratio Unsloth/e4b 1.457, e4b faster per step, quality COMPARABLE; it agrees with P38's 1.413 inside the pre-registered anchor band, e4b.train.h2h.unsloth.qwen3.5090.2026-09-06) and Mixtral-8x7B, whose row leads with the footprint trade: e4b trained its experts under CPU offload at roughly one-ninth of Unsloth's resident allocator peak (the registered design; the footprint row e4b.train.h2h.unsloth.mixtral.5090.2026-09-06.footprint -- a trainable-on-smaller-cards capability, quality COMPARABLE) while Unsloth ran resident, its only mode, and what that VRAM buys it is speed per step (s/step ratio 0.361, e4b.train.h2h.unsloth.mixtral.5090.2026-09-06 -- a footprint-vs-speed trade, not a kernel deficit). No position on the other four, and the coverage rows are results, not empty cells: on Granite the comparator attached LoRA to the attention only, a fraction of e4b's trainable count with ExpertsLoRA on every MoE layer (e4b.train.h2h.unsloth.granite.5090.2026-09-06.coverage), and on OLMoE it died at MoE-LoRA engage while both e4b arms trained (e4b.train.h2h.unsloth.olmoe.5090.2026-09-06.coverage); gpt-oss refuses on both sides (P5 held), and Gemma-4's e4b attention-4-bit arms died on the projection-count check (#412; the bf16-attention fast_train path stays as tp1 left it) while Unsloth's arm trained. P38's two-hundred-step curve row (Unsloth lower at the end of the curve) still stands beside any Qwen3 position; per-arm rows are e4b.train.h2h.unsloth.<family>.5090.2026-09-06.arm.*
• enable_pipelined_residency patches an ExpertsLoRA wrapper's base and only runs when the wrapper delegates (eval mode, no_grad, an adapter that contributes nothing): assert the count and check the served path
• enable_hot_residency is deprecated since 0.6.2
• Speed absolutes depend on the host link; only ratios travel between hosts (claim e4b.host.ratios-travel-absolutes-do-not)
• No shipped tool bakes the training arena from a bf16 checkpoint yet (docs/STATUS.md, open)
• enable_nvme_train_residency refuses hot_rows below the expert count and requires gradient checkpointing
• NVMe residency is a batch-throughput path; cold-expert latency is bounded by the device link
• Not a general-purpose vLLM replacement: on the same box with identical prompt ids vLLM 0.28.0 is ahead of this package's NF4 control at both batch sizes (claim e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05); the ratio against the licensed stack is not quoted on that lane because its licensed arms were void on that box under the pre-registered pack-fingerprint rule (the streamed calibration did not reproduce the licensed pack's expert-matrix counts), the registered K8 gate run on that box's pack failed its second text (claim e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05.gate), so the recipe's speed there stays unlicensed and the streamed calibration's host-dependence is an open item; the 2026-09-03 comparison is superseded
• Gemma-4 has no quality instrument at this resolution (claim e4b.parity.gemma4.no-reference); gpt-oss raw-text perplexity cannot rank arms
• Granite's int4-expert rows fail the registered K8 budget (claim IDs carry the numbers); its licensed stack keeps NF4 experts with the folds and epilogue (e4b.serve.census.bo7.granite.b1.5090.2026-09-05, licensed by e4b.serve.buildout.granite.b1.5090.2026-09-04); the per-family positions the census quotes are in docs/STATUS.md -- Qwen3's licensed stack is the streamed calibrated pack (e4b.serve.buildout.bo6c.qwen3.all-calibexp-streamed-64k.k8.2026-09-05), OLMoE's and Mixtral's position is NF4, Gemma-4's is the exact round-1 fold on NF4 with no quality instrument, gpt-oss's is its NF4 reference arm
• Single-stream speed absolutes are measured-private where the register says so; ratios travel, absolutes do not
• The streamed 64k calibration recipe does not reproduce its licence across hosts: re-derived on another RTX 5090 host with the same checkpoint, text and knobs it produced a different pack (a handful of expert matrices classified differently at the min_rows threshold) that fails the registered gate on C4 validation (claim e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05.gate); the licence of e4b.serve.buildout.bo6c.qwen3.all-calibexp-streamed-64k.k8.2026-09-05 stands on its box as measured, and a pack derived elsewhere carries no licence until gated on that box -- tracked as issue #405
• A uniform int4 grid cannot hold e2m1 levels: re-quantising MXFP4 experts to int4-b32 fails the quality gate (docs/STATUS.md)
• gpt-oss raw-text perplexity is not an instrument for ranking an exact arm against a noisy one; the native store's speed is quoted with its quality gate open (docs/SERVING-THROUGHPUT.md)
• MXFP4 arena training needs grouped-nf4-gemm>=0.12.0 (F8_E8M0 scales), below the current [fast] floor in pyproject.toml
• gpt-oss expert training is EXPERIMENTAL and unlicensed. Lane tp1 (bench/train-parity-20260905/tp1/) ran grouped-nf4-gemm's run_mxfp4_20b_qlora (mxfp4_qlora.ExpertsMxfp4LoRA over the released e2m1/e8m0 bytes, faithful clamped GLU and biases) on its own text: the step-0 canary against transformers' dequant path passed, provenance pre == post held over every native expert tensor, and the loss fell -- the row is e4b.train.parity.tp1.gptoss.mxfp4.2026-09-05. No parity pair exists, the run is never compared to the e4b arms, and nothing licenses it; the e4b enablers refuse gpt-oss (zero patched), and the arena_train=True wrap of gpt-oss computes a generic epilogue without the per-expert biases and clamp -- do not train gpt-oss experts through it
• enable_mxfp4_nvme_residency REFUSES bias-carrying (gpt-oss-shaped) modules (#402): it passed no biases and defaulted to the DeepSeek-V4 epilogue, so binding a gpt-oss stack to a native-byte arena computed an unfaithful forward (#397); the native MXFP4 store for gpt-oss serving is the paged engine's route (enable_serve_experts_int4 on gpt-oss), and gpt-oss training against an arena has no route -- training_support.gpt_oss.nvme_train is refused, with that code reference

Current measured position

Representative active claims, selected from docs/claims.json at the source commit; retired and superseded claims never appear here, and the register's tiers are counted above (claims tiered in the public register: 161 measured · 6 measured-private · 3 open · 7 superseded · 6 retired). Each card links its receipt at the pinned commit.

Current status and compatibility

v0.35.3 · current status → — the position page is docs/STATUS.md at the pinned commit (latest on main, unpinned; not the source of any fact rendered here); the site's status and compatibility pages are generated from the same commit, and the evidence words are defined once in the system manifest's vocabulary.

Problems this solves

• Why does `load_in_4bit` still OOM on a Mixture-of-Experts model? — bitsandbytes' 4-bit walker replaces only nn.Linear and skips a fused MoE's 3-D expert stacks, so load_moe_4bit_streaming quantises exactly those stacks and verify_moe_4bit proves it.
• How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)? — Choose between the convenient QLoRA path, which decodes MXFP4 and re-quantises to NF4, and the native-byte path, which keeps the released blocks and scales in an arena.
• How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena? — Bind pinned host RAM or a baked NVMe arena to a real model with the streaming loader and the nvme_residency, mxfp4_nvme_residency and nvme_train_residency engines.
• How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)? — ExpertsLoRA adds a trainable per-expert low-rank delta before each routed expert's activation over a frozen NF4 stack PEFT cannot target; enable_fast_train runs it on grouped kernels.
• How can I run a Mixture-of-Experts model larger than my GPU's VRAM? — Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena.
• How do I serve a large MoE on a consumer GPU (RTX 5090 class)? — The paged decode runner with experts on grouped-nf4-gemm's kernels is the measured serving path on one RTX 5090 class; the HTTP shim is a separate reference-path deployment.

Canonical links

• GitHub: https://github.com/pjordanandrsn/experts4bit-qlora · issues · releases
• PyPI: https://pypi.org/project/experts4bit-qlora/
• Pinned at 0c2a256dcdc2 (the source of every fact on this page): docs/capabilities.json · docs/STATUS.md · docs/claims.json · docs/SOLUTIONS.md · docs/system-manifest.json
• Latest on main (unpinned; not the source of any fact rendered here), beside each pinned document above: docs/capabilities.json · docs/STATUS.md · docs/claims.json · docs/SOLUTIONS.md · pinned counterparts: docs/claims.json@0c2a256dcdc2
• Machine-readable: this page as JSON · as Markdown · packages.json · system-manifest.json · system-manifest.provenance.json · /ml/llms.txt