`experts4bit-qlora`

Fine-tune and serve large Mixture-of-Experts models on one GPU, with 4-bit experts in GPU memory, RAM or SSD.

v0.58.0 · current status → · Install · Try the example · Check the limits

What it does

Use this runtime to load, fine-tune, and serve fused MoE models in 4-bit. transformers v5 stores a MoE's experts as one fused three-dimensional parameter per layer rather than ordinary nn.Linear modules, so the standard bitsandbytes module walker can leave most of the expert weights in bf16 and the model still OOMs; this package quantises exactly that fused stack, fine-tunes it with per-expert LoRA, keeps experts in host RAM or on NVMe when they do not fit, and serves the result on one consumer NVIDIA GPU.

Choose a task

Task guide	Status	Conditions
Why does `load_in_4bit` still OOM on a Mixture-of-Experts model?	supported	Models and limits
How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)?	supported	Models and limits
How can I run a Mixture-of-Experts model larger than my GPU's VRAM?	supported	Models and limits
How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena?	supported	Models and limits
How do I serve a large MoE on a consumer GPU (RTX 5090 class)?	supported	Models and limits
How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)?	experimental	Models and limits

Install

Routes from docs/capabilities.json at the pinned commit; the first entry of the register is the primary route.

pip install "experts4bit-qlora[train]"

Primary route: streaming loader + trainer.

Alternatives:

pip install "experts4bit-qlora[train,fast]" — loader plus accelerated grouped-kernel path
pip install "experts4bit-qlora[fast]" — residency/NVMe/fast-kernel route
pip install "experts4bit-qlora[serve]" — HTTP shim (optional)

Environment (from the capability register; one list for all of these routes, so not every item applies to each): OS: Linux · Python: >=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested) · Accelerator: NVIDIA CUDA GPU (bitsandbytes 4-bit); NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm) · Requires: an expert arena baked with grouped-nf4-gemm (nvme_bake_nf4 for NF4, the MXFP4 relocation bake for native MXFP4); bitsandbytes>=0.43; grouped-nf4-gemm>=0.43.0 ([fast] extra; the floor is pyproject.toml's, validated by CI); local NVMe or a fast block device; pinned host RAM for the hot tier; torch>=2.2; transformers>=5.0 for the streaming loader ([train] extra); triton>=3.4 (Linux-only distribution)

Smallest verified example

Needs a CUDA GPU, network access and the checkpoint download; the strict verification is the observable result (it raises if any expert stack is still high precision). Syntax-checked in the repository's CI; executed on rented hardware lanes with receipts.

import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16, r=8, alpha=16, quant_type="nf4",
)
verify_moe_4bit(model, strict=True)   # raises RuntimeError naming the first high-precision expert stack

How each capability addresses the problem

Use this when

A Mixture-of-Experts model loaded with load_in_4bit still occupies bf16-sized VRAM and OOMs, because transformers stores its experts as one fused 3-D parameter per layer that bitsandbytes' nn.Linear walker never touches. — load_moe_4bit_streaming quantises exactly the fused expert stack on the way to the GPU (Experts4bit, the 4-bit face of ExpertsNbit), never materialising it in bf16; verify_moe_4bit(model, strict=True) raises if any stack is still high precision. solution page
Fine-tune (QLoRA / LoRA) a fused-expert MoE whose expert weights PEFT and the bitsandbytes walker never see. — ExpertsLoRA wraps each frozen 4-bit expert stack with per-expert LoRA; load_moe_4bit_streaming installs it; enable_fast_train(model, dgrad=True) routes the step through grouped-nf4-gemm's grouped kernels, enable_batched_train(model) is the no-extras fallback; python -m experts4bit_qlora.train is the env-configured trainer, on the fused path with FAST_TRAIN=1 (implied by LAYER_MAJOR=1) and on the reference loop otherwise. model_families lists exactly the families whose fast_train path is `supported` in training_support (the headline path, enable_fast_train, with a PASS receipt on real weights); every other path is stated per family there, never as a flat flag. solution page
The quantised experts still do not fit the GPU: run a Mixture-of-Experts model larger than VRAM by keeping experts resident somewhere else and streaming them per layer. — load_moe_4bit_streaming(..., offload=True) homes each layer's frozen 4-bit experts in pinned host RAM and streams one layer at a time (prefetch=True overlaps); enable_dense_offload streams the dense side; hot_sets_from_profile picks the resident experts by routing frequency; enable_pipelined_residency trades spare VRAM for speed at serve time. In training, layer_major_step runs one step's micro-batches layer by layer, so each offloaded layer is staged once per step instead of once per micro-batch: its speed with enable_fast_train(dgrad=True) under offload is claims e4b.train.p137.layer-major-offload.qwen3.4090.2026-10-18, e4b.train.p140.layer-major-offload.qwen3.5090.2026-10-18 (Qwen3-30B-A3B) and e4b.train.p144.layer-major-offload.mixtral.4090.2026-10-18 (Mixtral-8x7B); from the CLI trainer at its default recipe (OFFLOAD_EXPERTS=1 LAYER_MAJOR=1) it is claim e4b.train.p146.layer-major-cli.qwen3.4090.2026-10-18. solution page
Experts do not even fit host RAM: serve or train from an on-disk expert arena (NVMe) with a pinned-DRAM hot tier. — Bake the experts into an arena with grouped-nf4-gemm (nvme_bake_nf4 for NF4, the relocation bake for native MXFP4), then enable_nvme_residency (serving), enable_mxfp4_nvme_residency (native MXFP4 serving) or enable_nvme_train_residency (training, gradient checkpointing required) serve each layer's cold experts from the arena while hot_rows stay pinned in DRAM. solution page
Serve a large fused-expert MoE on one consumer NVIDIA GPU with the decode path measured against the model's own attention. — Two surfaces. The measured decode path is the paged runner (PagedModelRunner: paged fp8 KV cache and paged attention) with the experts on grouped-nf4-gemm's kernels -- NF4 experts on the grouped GEMM (enable_fast is the library entry point for that kernel; the harness attaches it through its residency engines), int4-b32 experts through enable_serve_experts_int4, and the opt-in calibrated int4 attention/head, fused glue rounds and router epilogue -- exercised by the in-tree bench harness bench/hybrid-g9/step_decomp.py, which produced the throughput and parity claims. The HTTP shim (python -m experts4bit_qlora.serve) is a reference-path deployment: load_moe_4bit_streaming with the experts streamed from pinned host RAM by default, stock model.generate, per-expert LoRA adapters hot-swapped per request; it attaches a kernel-backed engine only under E4B_RESIDENCY=pipelined and does not use the paged runner or enable_fast. solution page
Work with models whose experts are released as native MXFP4 (gpt-oss, DeepSeek-V4): load them faithfully, keep the released bytes in an NVMe arena, train on them, and serve them without re-quantisation. — The loader dequantises MXFP4 (e2m1 blocks + e8m0 scales) bit-identically to the reference; enable_mxfp4_nvme_residency relocates the released bytes into an arena and serves them through grouped-nf4-gemm's native MXFP4 kernels; enable_nvme_train_residency trains against an MXFP4 arena; the native MXFP4 expert store for the paged engine is an opt-in lane. solution page

Do not use this when

dense (non-MoE) models: use bitsandbytes directly
a general-purpose serving engine or a vLLM replacement
models that already fit in bf16 and need no offload: 4-bit here is a memory trade, not a speed-up, and on the measured comparator it cost energy (claim e4b.train.energy-honest.5090.2026-10-05)
silent fallbacks: every enable_* returns a count or a non-empty handle list, or raises; callers must assert it
operating systems other than Linux, or GPUs without CUDA

Limitations — what it does not promise

Support depends on the model and execution path. Check the model and hardware conditions before choosing a route; the full repository limits for each capability are below.

quantize-fused-moe-experts · full limits

Unsupported model_type raises NotImplementedError; identity-expert families (LongCat-Flash) are refused by name
4-bit is a memory trade when the model already fits, and cost energy on the measured comparator (one rented card, one projection, medians of three passes): claim e4b.train.energy-honest.5090.2026-10-05
Dense (non-MoE) models are not the target; use bitsandbytes directly

qlora-fused-moe-experts · full limits

enable_fast_train returns 0 (does not raise) when the kernel package is absent: assert the return value
dgrad=True needs grouped-nf4-gemm>=0.7.0 (below the current [fast] floor in pyproject.toml, so any [fast] install has it); on older cuts it is downgraded with a RuntimeWarning
Loss parity is measured on the registered criteria in docs/METHODOLOGY.md, not asserted for every dataset
model_families is evidence-gated: a family is listed when its fast_train path -- the headline path, enable_fast_train -- PASSES the registered loss-parity criteria on real weights with a receipt in this repository (training_support is the per-path record; bench/train-parity-20260905/tp1/ is lane tp1, all six serving families). olmoe is confirmed on real weights and a registered text; qwen3_moe and gemma4_text carry tp1 PASSes beside their flagship matrices; mixtral entered on its tp1 fused PASS under offload; granitemoe entered on its corrected-counter re-run's PASS (its first attempt was a harness error, kept as its own row); gpt_oss is refused by every training enabler and stays out
gpt_oss is not a training family here: the loader builds its experts bare (no ExpertsLoRA) by default, and both enable_fast_train and enable_batched_train return zero on it -- REFUSED rows on tp1 (e4b.train.parity.tp1.gptoss.fused.2026-09-05, e4b.train.parity.tp1.gptoss.batched.2026-09-05). Attention-only QLoRA over its frozen experts trains (e4b.train.parity.tp1.gptoss.attn_only.2026-09-05); its experts train only on unlicensed routes: ExpertsLoRA's unfused reference loop behind E4B_GPTOSS_EXPERT_LORA=1 (CPU-graded against a transformers-matching oracle, unmeasured on hardware; enable_fast_train and enable_batched_train refuse that wrapper), and grouped-nf4-gemm's experimental mxfp4_qlora.ExpertsMxfp4LoRA under the mxfp4-moe-training-and-residency capability -- experimental, never licensed
enable_batched_train falls back to the reference forward per call above _PAD_WASTE_LIMIT (engines/batched.py): the patch count stays positive while some or all layers run the per-expert loop on some steps. In the shipped code tp1 measured this fallback was uncounted, and three arms read VOID on exactly that -- OLMoE, Qwen3 and Gemma-4 (e4b.train.parity.tp1.olmoe.batched.2026-09-05, e4b.train.parity.tp1.qwen3.batched.2026-09-05, e4b.train.parity.tp1.gemma4.batched.2026-09-05); it engaged everywhere only on Mixtral's eight-expert and Granite's forty-expert shapes. As of the release carrying that bundle the fallback is countable: batched_fallback_stats(model) reports it (#402; those VOID rows are the reason it exists). Assert kernel engagement, not the return value alone
enable_fast, enable_fast_train and enable_batched_train refuse a wrapper whose base violates the stock-epilogue contract (EpilogueContractError, #402) instead of silently computing act(gate)*up over a base that applies biases, clamps or another gating rule -- a refusal is loud, a wrong epilogue is not; gpt-oss is the family this protects, and DeepSeek-V4 supplies its epilogue through _apply_gate and is admitted
Head-to-head against Unsloth's 4-bit MoE QLoRA path on one identical training problem (lane p38, 2026-09-05, one rented RTX 5090, Qwen3-30B-A3B, the clinical fixture tokenised once, seq 512, r 8 / alpha 16, identical targets, optimizer, batch, steps, precision and eval; bench/h2h-20260905/p38/): the shipped fused dgrad path with NF4 attention is faster per step, lower in peak VRAM and lower in energy per step than Unsloth on that box, and the two are comparable on held-out loss at sixty steps (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05, ...quality-n60) -- BUT the two-hundred-step curve favours Unsloth: its held-out loss is lower at the end of the curve while e4b's flattens from step sixty (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.curve-n200). Quote the position only beside that row. Candidate causes (the eval schedule, the checkpointing mode, the transformers/peft versions of the two stacks, the expert adapter's precision -- bf16 on this side, the loader passes the model dtype to ExpertsLoRA -- against fp32 on Unsloth's) are not established. One workload, one box, one family; nothing here licenses a path (tp1 does) and no general speed claim is made
tp2/P40 (2026-09-06, one rented RTX 5090, receipts in bench/h2h-20260906/tp2/) extends the Unsloth head-to-head to all six families on one box and one fixture, and its rows are statuses, not one number: positions exist only for Qwen3-30B-A3B (s/step ratio Unsloth/e4b 1.457, e4b faster per step, quality COMPARABLE; it agrees with P38's 1.413 inside the pre-registered anchor band, e4b.train.h2h.unsloth.qwen3.5090.2026-09-06) and Mixtral-8x7B, whose row leads with the footprint trade: e4b trained its experts under CPU offload at roughly one-ninth of Unsloth's resident allocator peak (the registered design; the footprint row e4b.train.h2h.unsloth.mixtral.5090.2026-09-06.footprint -- a trainable-on-smaller-cards capability, quality COMPARABLE) while Unsloth ran resident, its only mode, and what that VRAM buys it is speed per step (s/step ratio 0.361, e4b.train.h2h.unsloth.mixtral.5090.2026-09-06 -- a footprint-vs-speed trade, not a kernel deficit). No position on the other four, and the coverage rows are results, not empty cells: on Granite the comparator attached LoRA to the attention only, a fraction of e4b's trainable count with ExpertsLoRA on every MoE layer (e4b.train.h2h.unsloth.granite.5090.2026-09-06.coverage), and on OLMoE it died at MoE-LoRA engage while both e4b arms trained (e4b.train.h2h.unsloth.olmoe.5090.2026-09-06.coverage); gpt-oss refuses on both sides (P5 held), and Gemma-4's e4b attention-4-bit arms died on the projection-count check (#412, fixed since by #435; the arms have run since in P56 and tp4, all 115 projections converted, and are not licensed because tp1's constant parity band fails every accelerated path on this model, the kernel-free one included: #713; the bf16-attention fast_train path stays as tp1 left it) while Unsloth's arm trained. P38's two-hundred-step curve row (Unsloth lower at the end of the curve) still stands beside any Qwen3 position; per-arm rows are e4b.train.h2h.unsloth.<family>.5090.2026-09-06.arm.*
Attention projection targeting (add_attention_lora and TRAIN_ATTN_4BIT / quantize_attention_projections_4bit) is by module STRUCTURE, never family name: q_proj, k_proj and o_proj must be supported linears; v_proj may be a supported linear or absent/None (Gemma-4 attention_k_eq_v layers). The expected count is len(candidates) from detect_attention_projections, never 4 * n_layers. A layer with q_proj/o_proj but no k_proj is refused. A projection that carries a bias is refused (gpt-oss). Gemma-4 attention-4-bit is supported since lane P67 (2026-09-24, #713): with this detector its arms convert all 115 projections and run VALID (P56 ladder-3, tp4-c-4, P67), and read against the family's own floor -- the reference against its own reorderings (e4b.train.p67.gemma4.attn4-reorder-floor.5090.2026-09-24) instead of tp1's constant 0.05 band against zero -- the fused arm is indistinguishable from that reordering noise (e4b.train.p67.gemma4.fused-attn4.floor-band.5090.2026-09-24)
Double-quantized expert absmax, the CLI trainer's default for resident training since TC1 amendments 28 / 31 and enable_fast_train's since amendment 56 (compress_expert_absmax_(model) in code; E4B_ABSMAX_DQ=0, or enable_fast_train(model, absmax_dq=False), turns it off; for python -m experts4bit_qlora.train =1 requires it): stores each ExpertsLoRA-wrapped NF4 stack's fp32 absmax (one fp32 per 64 weights) in bitsandbytes' nested form (quantize_4bit(compress_statistics=True)'s offset, 8-bit codes and one fp32 per 256 values), about a quarter of the bytes; the absmax VALUES change (lossy against fp32, identical to bitsandbytes' double-quant), and with it off nothing changes. Resident training only: enable_fast_train and the ExpertsLoRA reference loop expand one layer's absmax just in time; expert offload, enable_batched_train, the hot/cold/pipelined/hybrid/NVMe engines and the gpt-oss / DeepSeek-V4 per-expert forwards refuse a compressed model (AbsmaxCompressedError), so attach those engines before enable_fast_train. Its step cost and memory are registered: e4b.train.absmax-dq.mixtral.5090.2026-10-05, e4b.train.absmax-dq.qwen3.5090.2026-10-05, e4b.train.absmax-dq.packed-4k.5090.2026-10-07
estimate_qlora_footprint's activation line is a stated heuristic, not a measurement, and the estimate excludes the CUDA context and allocator fragmentation; compare it with a measured peak before trusting it on a new card or recipe

run-moe-larger-than-vram · full limits

enable_pipelined_residency patches an ExpertsLoRA wrapper's base and only runs when the wrapper delegates (eval mode, no_grad, an adapter that contributes nothing): assert the count and check the served path
enable_hot_residency is deprecated since 0.6.2
Speed absolutes depend on the host link; only ratios travel between hosts (claim e4b.host.ratios-travel-absolutes-do-not)

offload-moe-experts-to-cpu-or-nvme · full limits

The training arena is a separate GPU-side bake (grouped-nf4-gemm's nvme_bake_nf4); e4b.open.tr2-repro-gap, reproducing the TR2 training receipt, is still open in the register (docs/STATUS.md)
enable_nvme_train_residency refuses hot_rows below the expert count and requires gradient checkpointing
NVMe residency is a batch-throughput path; cold-expert latency is bounded by the device link

serve-moe-on-consumer-gpu · full limits

Not a general-purpose vLLM replacement: on the same box with identical prompt token ids vLLM 0.30.0 decodes faster than this package's current int4 stack at both B=1 and B=16 (claims e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b1.5090.2026-09-22, e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b16.5090.2026-09-22), bounded to graph decode on one RTX 5090 box with one prompt set; quality quoted, never equated; the 2026-09-05 comparison against vLLM 0.28.0 and the NF4 control (e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05) is history
Gemma-4 has no quality instrument at this resolution (claim e4b.parity.gemma4.no-reference); gpt-oss raw-text perplexity cannot rank arms
Granite's int4-expert rows fail the registered K8 budget (claim IDs carry the numbers); its licensed stack keeps NF4 experts with the folds and epilogue (e4b.serve.census.bo7.granite.b1.5090.2026-09-05, licensed by e4b.serve.buildout.granite.b1.5090.2026-09-04); the per-family positions the census quotes are in docs/STATUS.md -- Qwen3's licensed stack is the streamed calibrated pack (e4b.serve.buildout.bo6c.qwen3.all-calibexp-streamed-64k.k8.2026-09-05), OLMoE's and Mixtral's position is NF4, Gemma-4's is the exact round-1 fold on NF4 with no quality instrument, gpt-oss's is its NF4 reference arm
Single-stream speed absolutes are measured-private where the register says so; ratios travel, absolutes do not
The licensed Qwen3 int4 expert pack travels as bytes: the streamed 64k calibration reproduced it byte-for-byte on one box and across boxes and a release boundary (claims e4b.serve.p55x.qwen3.pack-deterministic-samebox.2026-09-22, e4b.serve.p55x.qwen3.pack-reproduces-crossbox.2026-09-22), and loaded back by fingerprint it passes the registered gate on both texts (e4b.serve.p55x.qwen3.all-calibexp-streamed-64k.k8.2026-09-22); P37's divergent pack, which failed on C4 validation (e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05.gate), was an outlier. The fingerprint covers the experts only: the calibrated attention projections are re-derived on every load, and with RTN attention the same pinned experts fail C4 validation (e4b.serve.p55x.qwen3.rtn-attn.k8.2026-09-22) -- tracked as issue #674
estimate_serve_footprint prices serve_paged.build_engine's all-VRAM placement and the solver's VRAM/DRAM/NVMe tiers as the server builds them without a routing profile; its prefill/decode working set is a stated heuristic, and CUDA graph pools, the prefill graph's pool, the CPU tier's compute buffers, the CUDA context and fragmentation are excluded. It has been checked on two cards only (lane P109's Qwen3-30B-A3B receipts on an RTX 5090, OLMoE-1B-7B on an RTX A2000 at both placements), where it sat slightly under the measured allocator peak; the CHANGELOG gives the figures

mxfp4-moe-training-and-residency · full limits

A uniform int4 grid cannot hold e2m1 levels: re-quantising MXFP4 experts to int4-b32 fails the quality gate (docs/STATUS.md)
gpt-oss raw-text perplexity is not an instrument for ranking an exact arm against a noisy one; the native store's speed is quoted with its quality gate open (docs/SERVING-THROUGHPUT.md)
MXFP4 arena training needs grouped-nf4-gemm>=0.12.0 (F8_E8M0 scales), below the current [fast] floor in pyproject.toml
gpt-oss expert training is EXPERIMENTAL and unlicensed. Lane tp1 (bench/train-parity-20260905/tp1/) ran grouped-nf4-gemm's run_mxfp4_20b_qlora (mxfp4_qlora.ExpertsMxfp4LoRA over the released e2m1/e8m0 bytes, faithful clamped GLU and biases) on its own text: the step-0 canary against transformers' dequant path passed, provenance pre == post held over every native expert tensor, and the loss fell -- the row is e4b.train.parity.tp1.gptoss.mxfp4.2026-09-05. No parity pair exists, the run is never compared to the e4b arms, and nothing licenses it; the e4b enablers refuse gpt-oss (zero patched), and the arena_train=True wrap of gpt-oss computes a generic epilogue without the per-expert biases and clamp -- do not train gpt-oss experts through it
enable_mxfp4_nvme_residency REFUSES bias-carrying (gpt-oss-shaped) modules (#402): it passed no biases and defaulted to the DeepSeek-V4 epilogue, so binding a gpt-oss stack to a native-byte arena computed an unfaithful forward (#397); the native MXFP4 store for gpt-oss serving is the paged engine's route (enable_serve_experts_int4 on gpt-oss), and gpt-oss training against an arena has no route -- training_support.gpt_oss.nvme_train is refused, with that code reference

Current measured position

Representative active claims, selected from docs/claims.json at the source commit; retired and superseded claims never appear here. The whole register at that commit holds 460 measured · 8 measured-private · 3 open · 19 superseded · 9 retired; each card below is an active claim and links its receipt at the pinned commit.

TC1 amendment 69 (#835), one rented RTX 5090 (tc1-5090-137, AMD EPYC 7713, Vast machine 55913): TC1's field recipe (alpaca, seq 2048, micro-batch 2 x accum 4) over 60 load-gated steps with both frameworks on one stack (venv-unsloth, torch 2.12.1), e4b's matched arm at every current default against Unsloth 2026.9.14's matched arm (grouped_mm), two draws each in ABBA order, every arm profiled. Unsloth's device time per profiled step is 1.924x e4b's (3259.1 / 1693.9 ms; P205 HELD, [1.7, 2.7]); s/step 9.780 against 3.489, a wall ratio of 2.803 on this host (P206 HELD, [2.3, 3.3]). Unsloth runs 14.09x e4b's CPU ops per step (P207 HELD, >= 8) and its GPU is busy for 0.333 of its timed step against e4b's 0.486, so the wall ratio is 1.457x the device ratio (P208 HELD, >= 1.05). Held-out at N 0.75643 against 0.75761 (P209 HELD, within 0.01). e4b peaks 26.16 GB against Unsloth's 24.27.measured

e4b.train.h2h.unsloth.qwen3.5090.2026-10-08.field-device-ratio · RTX 5090 (sm_120), one rented Vast.ai verified/secure host (tc1-5090-137: instance 54809007, AMD EPYC 7713, Vast machine 55913); e4b @ 737f02f + grouped-nf4-gemm @ 8371c38, torch 2.12.1+cu130; Unsloth 2026.9.14 · measured 2026-10-08 · receipt (pinned)

QLoRA on frozen NF4 experts improves a held-out Alpaca eval from 1.4813 to 1.0290 on OLMoE-1B-7B.measured

e4b.train.olmoe-converges · allenai/OLMoE-1B-7B · measured 2026-07-04 · receipt (pinned)

With OFFLOAD_EXPERTS=1 the frozen experts stream from pinned CPU RAM one layer at a time, so a fused MoE whose 4-bit experts exceed the card still QLoRA-trains on 12 GB: Qwen3-30B-A3B peaks at 7.16 GB and Gemma-4-26B-A4B at 8.47 GB, both of which OOM without offload. Cost is about +11% s/step at OLMoE scale.measured

e4b.offload.fits-30b-class · · measured 2026-07-02 · receipt (pinned)

Throughput census (bo7, the shipped 0.35.0 / 0.30.0 code): single-stream (B=1) paged decode of Qwen/Qwen3-30B-A3B's LICENSED serving stack -- streamed GPTQ-calibrated int4 experts at 64k C4-validation tokens + C4-calibrated int4 attention + round-1/2 folds + router epilogue + decode glue, the configuration bo6c licensed on both texts (e4b.serve.buildout.bo6c.qwen3.all-calibexp-streamed-64k.k8.2026-09-05; pack artifact 11512 gptq / 776 rtn) -- on one rented RTX 5090: 238.1 tok/s (4.20 ms/step), x2.067 vs e4b's own NF4 control on the same box (8.68 ms = 115.2 tok/s); anchor-class projection 159.2 x 2.067 = about 329 tok/s, a PROJECTION from a class that was never certified.measured

e4b.serve.census.bo7.qwen3.b1.5090.2026-09-05 · RTX 5090 (sm_120, driver 595.84, 32607 MiB) on an AMD EPYC 7Q83 64-core host (128 CPUs, 251 GB, container cgroup memory.max 183,318,347,776 B = 170.7 GiB), Vast.ai instance 49916675; the 5090 class carries ~8.5% inter-box dispersion and B=1 is host-bound, so cross-host ratios are indicative only · measured 2026-09-05 · receipt (pinned)

Two arithmetically equivalent forwards of an MoE model disagree because rounding flips router top-k choices: 4.52% of (layer, token) choices on gpt-oss-20b and 6.77% on Qwen3-30B, with the disagreement carried by the flipped tokens (KL 0.0504 vs 0.0013). Every parity delta must be read against a per-model measured floor, never against zero.measured

e4b.parity.moe-routing-flip-floor · RTX 5090, rented · measured 2026-09-03 · receipt (pinned)

On a card that already fits the model, 4-bit expert projections cost more GPU energy per op than native bf16 on the measured comparator: one OLMoE-dims gate_up projection on one rented RTX 5090 (500 W power limit, driver 570.133.07), bitsandbytes 0.50.2, the unchanged harness, medians of three passes. bnb.matmul_4bit costs 1.748x native's total J/op at decode M=1, 1.601x at prefill M=512 and 1.965x at train fwd+bwd M=32; dequantize-then-linear costs 3.293x, 1.539x and 1.405x. The registered prediction that matmul_4bit would read near 1x at decode is refuted: 1.748x, at half native's power and 0.28 of its op rate. Batch is where the energy goes: the fused 4-bit MoE forward's J/token at batch 4096 is 0.063 of batch 64's (about 16x lower).measured

e4b.train.energy-honest.5090.2026-10-05 · one rented RTX 5090 (32 GB; power limit 500 W of 600 W max; driver 570.133.07; PCIe gen 4 x16) on an AMD EPYC 7B12 host (503 GB), Vast.ai verified-secure · measured 2026-10-05 · receipt (pinned)

Current status and compatibility

v0.58.0 · current status → — the position page is docs/STATUS.md at the pinned commit (latest on main, unpinned; not the source of any fact rendered here); the site's status and compatibility pages are generated from the same commit, and the evidence words are defined once in the system manifest's vocabulary.

Problems this solves

Why does `load_in_4bit` still OOM on a Mixture-of-Experts model? — bitsandbytes' 4-bit walker replaces only nn.Linear and skips a fused MoE's 3-D expert stacks, so load_moe_4bit_streaming quantises exactly those stacks and verify_moe_4bit proves it.
How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)? — Choose an MXFP4 loading or native-byte execution route by model and task; gpt-oss arena training is refused, while its separate experimental expert-training path remains unlicensed.
How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena? — Bind pinned host RAM or a baked NVMe arena to a real model with the streaming loader and the nvme_residency, mxfp4_nvme_residency and nvme_train_residency engines.
How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)? — ExpertsLoRA adds a trainable per-expert low-rank delta before each routed expert's activation over a frozen NF4 stack PEFT cannot target; enable_fast_train runs it on grouped kernels.
How can I run a Mixture-of-Experts model larger than my GPU's VRAM? — Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena.
How do I serve a large MoE on a consumer GPU (RTX 5090 class)? — The paged decode runner with experts on grouped-nf4-gemm's kernels is the measured serving path on one RTX 5090-class GPU; the HTTP shim is a separate reference-path deployment.

Package identity, supported platforms and source links

Canonical PyPI package: experts4bit-qlora
Canonical repository: github.com/pjordanandrsn/experts4bit-qlora
Import: experts4bit_qlora
License: MIT
Repository version: 0.58.0 (at the pinned commit)
Latest PyPI version: 0.58.0 — published
Source commit: f15cb7d695cf (2026-10-24) · files at this commit
Claim register: claims tiered in the public register: 460 measured · 8 measured-private · 3 open · 19 superseded · 9 retired — docs/claims.json (pinned)
Current status: v0.58.0 · current status → · capability table · docs/STATUS.md (pinned)
System manifest: runtime package of one system with grouped-nf4-gemm — system-manifest.json (verbatim, identical in both repositories; provenance)

Also discoverable as e4b, e4b-qlora, experts4bit, expertsnbit, experts-mxfp4. These are lookup-only aliases from the system manifest; experts4bit-qlora is the canonical project.

What it supports

Modes

inference, offload, quantization, serving, streaming, training

Formats

bf16, fp16, fp4, fp8, int4, int8, mxfp4, nf4

Model families

deepseek_v2; deepseek_v4 (Flash, Pro); ernie4_5_moe; gemma4_text; gpt_oss; granitemoe; granitemoehybrid; lfm2_moe; mixtral; nemotron_h; olmoe; qwen3_5_moe; qwen3_moe; qwen3_next

Operating system

Linux

Python

>=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested)

Accelerator

NVIDIA CUDA GPU (bitsandbytes 4-bit); NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)

Public entry points

cli:python -m experts4bit_qlora.serve, cli:python -m experts4bit_qlora.train, experts4bit_qlora.arch.topology:describe_moe, experts4bit_qlora.engines.batched:enable_batched_train, experts4bit_qlora.engines.capture:capture_decode, experts4bit_qlora.engines.dense_offload:enable_dense_offload, experts4bit_qlora.engines.expert_profile:hot_sets_from_profile, experts4bit_qlora.engines.fast:enable_fast, experts4bit_qlora.engines.fast:enable_fast_train, experts4bit_qlora.engines.int4_attn_calib:enable_serve_attn_int4_calib, experts4bit_qlora.engines.int4_experts:enable_serve_experts_int4, experts4bit_qlora.engines.layer_major:layer_major_step, experts4bit_qlora.engines.nvme_experts:enable_mxfp4_nvme_residency, experts4bit_qlora.engines.nvme_experts:enable_nvme_residency, experts4bit_qlora.engines.nvme_train:enable_nvme_train_residency, experts4bit_qlora.engines.offload:enable_expert_offload, experts4bit_qlora.engines.paged_runner:PagedModelRunner, experts4bit_qlora.engines.pipelined:enable_pipelined_residency, experts4bit_qlora.loader:load_moe_4bit_streaming, experts4bit_qlora.lora:ExpertsLoRA, experts4bit_qlora.recipe:estimate_qlora_footprint, experts4bit_qlora.recipe:prepare_qlora_training, experts4bit_qlora.serve_recipe:estimate_serve_footprint, experts4bit_qlora.verify:verify_moe_4bit, experts4bit_qlora:Experts4bit, experts4bit_qlora:ExpertsNbit, flag:E4B_EXPERT_PROFILE

Owns (system manifest)

model loading (streaming loader, fused expert stacks, Experts4bit/ExpertsNbit bases)
quantization orchestration (NF4/MXFP4 routes, int4 expert store, calibration drivers)
adapters (ExpertsLoRA)
training (trainer, fast/batched/hybrid paths, checkpointing)
residency integration (host-RAM and NVMe engines bound to a loaded model)
serving (paged runner, decode stack, HTTP shim)
model-level measurement, quality gates and the claims register for model-level numbers

Related package

grouped-nf4-gemm — one system, two independently released packages: the runtime package depends on the kernel package through its optional `fast` extra; the kernel package never depends on the runtime package. Dependency direction: experts4bit-qlora -> grouped-nf4-gemm.

Canonical links

GitHub: https://github.com/pjordanandrsn/experts4bit-qlora · issues · releases
PyPI: https://pypi.org/project/experts4bit-qlora/
Pinned at f15cb7d695cf (the source of every fact on this page): docs/capabilities.json · docs/STATUS.md · docs/claims.json · docs/SOLUTIONS.md · docs/system-manifest.json
Latest on main (unpinned; not the source of any fact rendered here), beside each pinned document above: docs/capabilities.json · docs/STATUS.md · docs/claims.json · docs/SOLUTIONS.md · pinned counterparts: docs/claims.json@f15cb7d695cf
Machine-readable: this page as JSON · as Markdown · packages.json · system-manifest.json · system-manifest.provenance.json · /ml/llms.txt

rendered from experts4bit-qlora 0.58.0 · source: pjordanandrsn/experts4bit-qlora@f15cb7d695cf · checked 2026-10-24