How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)?

Solved by experts4bit-qlora · this page in the repository (pinned 0c2a256dcdc2, the source of this rendering; latest on main, unpinned; not the source of any fact rendered here)

Install routes

From docs/capabilities.json at the pinned commit.

pip install "experts4bit-qlora[train]"

Load with load_moe_4bit_streaming, which installs a frozen NF4 Experts4bit stack wrapped in trainable per-expert ExpertsLoRA adapters for every MoE layer, then train with python -m experts4bit_qlora.train or your own loop. Add enable_fast_train(model, dgrad=True) from the [fast] extra to run the step through grouped-nf4-gemm's grouped kernels, and assert its return value.

The abstraction that fails, and what replaces it

PEFT expects nn.Linear targets. A fused MoE checkpoint stores each layer's experts as stacked 3-D projections — gate_up_proj [E, 2I, H] and down_proj [E, H, I] — so there is no per-expert Linear for target_modules to match. A delta hung on the stack as a whole would also compute the wrong function: the low-rank term has to be applied per routed expert before that expert's activation (act(W_e x + B[e] A[e] x), not act(W_e x) + d). ExpertsLoRA supplies the adapter abstraction — stacked per-expert A[e] / B[e] over the frozen packed base, the delta injected pre-activation — and grouped-nf4-gemm supplies the fused grouped forward and, with dgrad=True, the optional input-gradient kernel.

Ordinary PEFT:     nn.Linear per projection  <-  LoRA A/B hooked on each Linear
Fused checkpoint:  experts.gate_up_proj [E, 2I, H], experts.down_proj [E, H, I]   (no per-expert Linear)
        |
        v
Experts4bit frozen packed base   +   ExpertsLoRA A[e] / B[e], one pair per expert
        |
        v
per-routed-expert delta, applied BEFORE the activation:  act(W_e x + B[e] A[e] x)
        |
        v
grouped forward (grouped-nf4-gemm)   +   optional dgrad (enable_fast_train(model, dgrad=True))

Measured result

Claim e4b.train.flagship-matrix (measured, receipts in this repository): across two 30B-class MoEs — Qwen3-30B-A3B and Gemma-4-26B-A4B — five datasets each, 200 steps per cell, the fused training path runs 1.52–1.81× faster per step at 0.75–0.81× peak VRAM and 0.86–0.92× energy per step, with held-out loss parity on both registered criteria and the frozen 4-bit stack verified bit-identical over 16.31 GB hashed. That ratio is fused-vs-reference with the same offload and checkpointing in both arms. A per-step multiple quoted against the per-expert Python loop on a smaller model (the OLMoE per-expert-loop comparison) has a different denominator and different models; it is not this number, and the two do not compose.

Claim family e4b.train.parity.tp1.* (measured, receipts in ../../bench/train-parity-20260905/tp1/; lane tp1, 2026-09-05, the shipped code on one rented RTX 5090, real weights through the direct loader, 60 steps on the registered clinical text, verdicts in the same registered units with every row classified OK / REFUSED / HARNESS_ERROR / ALARM / OOM / NOT_RUN / EXPERIMENTAL, cost reported and never gated): enable_fast_train(dgrad=True) PASSES on OLMoE-1B-7B-Instruct (e4b.train.parity.tp1.olmoe.fused.2026-09-05, the first fused-vs-reference reading on a registered text with real weights for that family), on Qwen3-30B-A3B resident on the 32 GB card (…qwen3.fused…), on Gemma-4-26B-A4B-it — the -it checkpoint, loaded without #344 on that host — with the step-wise median inside the band by a small margin (…gemma4.fused…), and on Mixtral-8x7B-Instruct under offload=True at half the reference loop's peak VRAM (…mixtral.fused…; the family entered model_families on it); and on Granite-3.1-3B-A800M — the family's first direct real-weight load — on the corrected-counter re-run (…granite.fused…; the first attempt is a kept HARNESS_ERROR row, …granite.fused.attempt1…, a closure bug in the harness's kernel counter, not the shipped code; the family entered model_families on the re-run); enable_batched_train PASSES on Granite (…granite.batched…) and Mixtral (…mixtral.batched…) and is VOID on OLMoE, Qwen3 and Gemma-4 — the kernel was not reached on every layer (below); gpt-oss's fused and batched arms are REFUSED (below). A PASS there is a PASS on one text; no convergence claim and no cross-family ratio is made.

Claim family e4b.train.h2h.unsloth.<family>.5090.2026-09-06 (measured, receipts in ../../bench/h2h-20260906/tp2/; lane tp2 / P40, 2026-09-06, one rented RTX 5090, the pre-registration verbatim in the bundle): the head-to-head against Unsloth, per family — P38's fixture exactly on all six families, one box, one session, e4b at the shipped cut a user installs today (0.35.1 + grouped-nf4-gemm 0.30.2 from PyPI, NF4 attention via TRAIN_ATTN_4BIT) against Unsloth 2026.9.2; every attempt a row, a position only where both primary arms are VALID, the quality reading beside it. Positions on two families: Qwen3-30B-A3B, s/step ratio Unsloth/e4b 1.457 (5.986 vs 4.108 s — e4b faster per step; peak 21.371 vs 23.141 GB; 383.9 vs 561.4 J/step; held-out COMPARABLE, Δ +0.0152 ≤ 0.05) — +3.1% from P38's 1.413, inside the pre-registered ±10% anchor band: the sign and size of the per-step position reproduce on a second box (…qwen3.5090.2026-09-06, ….quality-n60, ….e4b-internal-parity); and Mixtral-8x7B, where the footprint trade leads the row: the e4b arm trained its experts under CPU offload at a 3.223 GB peak (the registered design for this family, tp1's offload=True; a trainable-on-smaller-cards capability, its own row …mixtral.5090.2026-09-06.footprint) while Unsloth ran resident (its only mode) at 29.163 GB, and what that VRAM buys it is speed per step — ratio 0.361, a footprint-vs-speed trade, not a kernel deficit; e4b's energy is lower (298.3 vs 350.2 J/step); held-out COMPARABLE, Δ −0.0087; Unsloth's predicted resident OOM did not happen (…mixtral.5090.2026-09-06, ….quality-n60, ….e4b-internal-parity). Statuses and coverage rows are the result on the other four: Granite — the comparator could not train the experts: Unsloth's arm completed but is VOID (it attached LoRA to the attention only, 2,621,440 trainable against e4b's 49,807,360 with ExpertsLoRA on all 32 MoE layers; its MoE-LoRA path never engaged on granitemoe; …granite.5090.2026-09-06.coverage, ….arm.unsloth.ckpt_unsloth), e4b's arms VALID (fused ×5.59 its reference); OLMoE — the comparator could not train the family at all: Unsloth's process died at MoE-LoRA engage before its first receipt write (HARNESS_ERROR; …olmoe.5090.2026-09-06.coverage, ….arm.unsloth.ckpt_unsloth), e4b's arms VALID (fused ×3.73); gpt-oss — three refusals as pre-registered (e4b attention-4-bit refuses on 96/96 bias-carrying projections, the fused path patches nothing, Unsloth's MXFP4 load conversion fails; …gptoss….arm.*); Gemma-4 — both e4b attention-4-bit arms died on the converter's own count check (#412; the bf16-attention fast_train path stays as tp1 left it) while Unsloth's arm is OK · VALID at 3.510 s/step (…gemma4….arm.unsloth.ckpt_unsloth). e4b's internal fused-vs-reference parity PASSES on all four families that ran both arms (×5.59 / ×3.73 / ×2.69 / ×1.23 per step; informational, tp1 owns the licence). Nothing supersedes P38 — two boxes, two measurements, never averaged — and P38's 200-step curve row still stands beside any Qwen3 position.

Claim family e4b.train.h2h.unsloth.qwen3.5090.2026-09-05 (measured, receipts in ../../bench/h2h-20260905/p38/; lane p38, 2026-09-05, one rented RTX 5090, the pre-registration verbatim in the bundle): the head-to-head against Unsloth, end-to-end, on one identical training problem — Qwen3-30B-A3B at one pinned revision, the registered clinical fixture tokenised once and asserted by sha in every arm, seq 512, r 8 / α 16 on attention q/k/v/o and every expert, the router frozen, the same optimizer, LR, batch, steps, precision and held-out eval on both sides; this package's fused dgrad path with NF4 attention (enable_fast_train(dgrad=True) + the shipped TRAIN_ATTN_4BIT mechanism) against Unsloth 2026.9.2's 4-bit MoE QLoRA path. At 60 steps e4b is faster per step (s/step ratio Unsloth/e4b 1.413: 2.151 vs 1.522 s), lower in peak VRAM (21.371 vs 23.141 GB) and energy per step (157.1 vs 224.7 J), reaches a held-out loss of 0.32 sooner (92.5 vs 130.3 s), and the two are comparable on held-out loss (0.2923 vs 0.2975, |Δ| 0.0052 ≤ the 0.05 reading threshold; …quality-n60). At 200 steps the curves separate in Unsloth's favour (0.2713 vs 0.2881, …curve-n200) — e4b's flattens near 0.29 from step 60 while Unsloth's keeps falling; that row is quoted beside the position wherever the position is quoted, and its causes (the eval schedule, the checkpointing mode, the two stacks' transformers/peft versions, the expert adapter's precision — bf16 here because the loader passes the model dtype to ExpertsLoRA, fp32 on Unsloth's side) are candidates, not established. One workload (≈86 tokens per step, batch 1, resident), one box, one family; no general speed claim; nothing licensed (…e4b-internal-parity is the informational fused-vs-reference PASS on that box, tp1 owns the licence).

Symptoms

• "QLoRA a Qwen3 MoE on a 24 GB GPU" / "fine-tune fused MoE experts with LoRA on a consumer GPU".
• "experts4bit-qlora vs Unsloth for MoE QLoRA" / "is Unsloth faster than e4b on a 4-bit MoE" — the head-to-head above, one identical problem on one box.
• PEFT does not see the expert parameters: target_modules matches no nn.Linear under mlp.experts, because the experts are a fused 3-D tensor (gate_up_proj, down_proj).
• Stock load_in_4bit leaves the experts in bf16 and the run OOMs (bitsandbytes-moe-load-in-4bit-still-ooms.md).
• Training works but every step crawls through a per-expert Python loop.
• enable_fast_train returned 0 (or something falsy) and nothing changed.
• You need loss parity with a bf16 run, not just a falling curve.

Why it happens

Adapter libraries attach low-rank deltas to nn.Linear. A fused expert stack has no per-expert Linear to hook, and the adapter must land before the nonlinearity for each routed expert. ExpertsLoRA therefore re-implements the expert math: for each expert e the frozen 4-bit projections get a trainable scaling * (x @ A[e].T) @ B[e].T term, with stacked A/B per expert, while the base's projection re-dequantises in backward so no dequantised-expert activation is held (experts4bit_qlora/lora.py, ../BITSANDBYTES.md). The reference forward is a per-expert loop by design; the speed comes from the kernel side.

Which project solves it

experts4bit-qlora owns the adapter (ExpertsLoRA), the streaming loader, the trainer and the training-side offload. grouped-nf4-gemm (GitHub, PyPI) owns the kernels: enable_fast_train routes the ExpertsLoRA training forward through its nf4_qlora.fused_grouped_lora, and dgrad=True routes the backward through its single-launch dgrad kernel. [fast] is the seam. enable_batched_train is the no-extras fallback when the kernel package will not build.

Install

pip install "experts4bit-qlora[train]"   # minimum/reference training: loader + trainer
pip install "experts4bit-qlora[fast]"    # accelerated grouped-kernel path: + grouped-nf4-gemm for enable_fast_train

Smallest correct example

Needs: GPU + network + model download.

import torch
from experts4bit_qlora import enable_fast_train, load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16, r=8, alpha=16,
    quant_type="nf4", offload=True,          # offload=True: experts in pinned host RAM, one layer on the GPU
)
verify_moe_4bit(model, strict=True)
model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})  # required with offload
n = enable_fast_train(model, dgrad=True)
assert n > 0, "still on the per-expert loop: grouped-nf4-gemm missing?"
# ... your optimiser over the parameters whose name contains "lora" ...

The assertion is there because acceleration must be observed, not assumed: enable_fast_train returns 0 rather than raising when the kernel package is absent, and from the caller's side 0 looks exactly like the per-expert loop it was meant to replace. Keep the assert in your own loop.

The shipped trainer does the same from environment variables (there are no CLI flags; --help prints them):

STEPS=150 R=8 TRAIN_EXPERTS=1 OFFLOAD_EXPERTS=1 OUT=./out python -m experts4bit_qlora.train

Expected result

verify_moe_4bit(model, strict=True) returns without raising. enable_fast_train returns the number of ExpertsLoRA modules patched — one per MoE layer — and 0 means you are silently on the reference loop (../../CONTRIBUTING.md explains why every enabler returns a count). The trainer writes the adapter (adapter_best.pt) under OUT; python -m experts4bit_qlora.infer serves it over the same NF4 base with ADAPTER=./out/adapter_best.pt.

Supported scope

• Families, per path and from evidence — the capability's model_families (olmoe, qwen3_moe, gemma4_text, mixtral, granitemoe) is exactly the families whose fast_train cell below is supported:

Each cell is one of supported (completed under the registered protocol with a PASS/OK receipt), refused (with the reason), void (ran, unreadable), harness_error, not_tested, experimental, n/a — per path, never a flat flag; the machine-readable form, with the claim id behind every supported / void / refused cell, is training_support in ../capabilities.json, validated by scripts/check_capabilities.py, and model_families is exactly the families whose fast_train is supported. Row statuses in the tp1 receipt are one of OK / REFUSED / HARNESS_ERROR / ALARM / OOM / NOT_RUN / EXPERIMENTAL with the parity verdict (PASS / FAIL / VOID) as a separate column. On top of that table, the attention-4-bit configuration (TRAIN_ATTN_4BIT, reference_attn4/fused_attn4) has its own receipts since tp2/P40 (2026-09-06): supported on granitemoe, olmoe, qwen3_moe and mixtral (e4b.train.h2h.unsloth.<family>.5090.2026-09-06.arm.e4b.*, internal parity PASS on each); not supported on gemma4_text pending #412 (quantize_attention_projections_4bit converted 100 projections, expected 120 — both arms died before a step; the bf16-attention fast_train cell above is unchanged); refused on gpt_oss (96 of 96 attention projections carry a bias, and the converter refuses bias-carrying projections). The per-family reasons live in training_support's path entries. DeepSeek-V4 is admitted by structure (the base supplies its clamped epilogue via _apply_gate, ../DEEPSEEK-V4.md) and has no training receipt. The per-family training table with every row's status is the tp1 section of ../ARCHITECTURE_SUPPORT.md; the loader's families are in bitsandbytes-moe-load-in-4bit-still-ooms.md.

• Storage: nf4 is the benchmarked QLoRA default; fp4 / int8 / fp8 / bf16 / fp16 carry a tested LoRA-step contract only (../STORAGE-MODES.md).
• OFFLOAD_EXPERTS=1 / offload=True is what makes a 30B-class MoE train on a 12 GB or 24 GB card (run-moe-larger-than-vram.md).
• Environment: Linux, NVIDIA CUDA, torch>=2.2, bitsandbytes>=0.43, transformers>=5.0; CI tests Python 3.11. [fast] installs grouped-nf4-gemm at the fast extra's floor in pyproject.toml (grouped-nf4-gemm >= 0.30.0 at this commit; validated by CI), which needs Triton (Linux-only) on an sm_80-or-newer GPU.

Limitations

• enable_fast_train returns 0 rather than raising when nf4_qlora is missing; dgrad=True on a kernel cut older than grouped-nf4-gemm 0.7.0 is turned off with a RuntimeWarning. Assert the count.
• train.main() does not enable the fast path for you (../../CONTRIBUTING.md).
• The fused path changes the expert summation order (opt-in on purpose) and its model-level accuracy is measured, not assumed (engines/fast.py docstring).
• enable_batched_train wins only a toy microbench; at Qwen3-30B width it gives no speed-up and the highest peak memory (engines/batched.py, e4b.train.fast-train-dgrad) — width-dependent: on tp1 it is the faster of the two paths measured at Granite-3B width (e4b.train.parity.tp1.granite.batched.2026-09-05). It and enable_fast_train are mutually exclusive on a module.
• What to check on a batched arm — engagement, not the count. enable_batched_train falls back to the reference forward per call whenever the padded gather exceeds _PAD_WASTE_LIMIT (engines/batched.py), so its return value stays positive while some layers run the per-expert loop on some steps. In the code tp1 measured (0.35.0) that fallback was silent and uncounted, and three batched arms are VOID on exactly this — OLMoE, Qwen3 and Gemma-4 (e4b.train.parity.tp1.olmoe.batched.2026-09-05, …qwen3.batched…, …gemma4.batched…; on 9 of Gemma-4's 60 steps no layer reached the kernel) — while Mixtral's 8-expert and Granite's 40-expert shapes engaged everywhere. As of 0.35.1 the fallback is counted: read batched_fallback_stats(model) after a step (#402; those VOID rows are why it exists) and require at least two kernel calls per patched layer per step; a VOID row carries no parity number, however its loss curve reads.
• A wrapper whose base breaks the stock-epilogue contract is refused, loudly. enable_fast, enable_fast_train and enable_batched_train raise EpilogueContractError (0.35.1, #402) rather than compute act(gate) * up over a base that applies biases, clamps or another gating rule; DeepSeek-V4 supplies its epilogue through _apply_gate and is admitted, gpt-oss is the family the refusal protects.
• What to check on gpt-oss — the count is zero, by design. The loader builds its experts bare (no ExpertsLoRA), so enable_fast_train and enable_batched_train both return 0 — REFUSED rows on tp1 (e4b.train.parity.tp1.gptoss.fused.2026-09-05, .batched). Attention-only QLoRA over the frozen experts trains (e4b.train.parity.tp1.gptoss.attn_only.2026-09-05); the only route that trains its experts is the kernel package's experimental ExpertsMxfp4LoRA (mxfp4-moe-training-and-residency.md), never licensed.
• Non-checkpointed offload training is unsupported and fails loudly; the trainer always enables gradient checkpointing.
• gpt-oss: trainable LoRA over its biased, clamped experts needs a gpt-oss-aware adapter, which is a separate change (arch/gptoss.py).
• The registered training speed-up claim is retired: restated against a current baseline, roughly half of it is upstream's own fused loop (e4b.retired.13.47x-training-speedup, retired; ../STATUS.md, "What changed").
• The Unsloth head-to-head is one workload per box, and its 200-step curve favours Unsloth. The 60-step position (e4b faster per step, lower peak VRAM and energy, comparable held-out loss) is never quoted without the 200-step row (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.curve-n200: Unsloth's held-out loss lower at the end of the curve). ≈86 tokens per step at batch 1 on a resident 30B MoE; larger batches and longer sequences are not measured; the candidate causes of the separation are not established. The other families are now measured on one box (tp2/P40, 2026-09-06, above) and are statuses, not one number: two positions (Qwen3 1.457, e4b faster; Mixtral 0.361, Unsloth faster resident at 29.163 GB against e4b's expert-offload 3.223 GB — the designs are named wherever that ratio is quoted), a VOID, a HARNESS_ERROR, and refusals on both sides for gpt-oss; no cross-family ratio and nothing licensed.

Related

• bitsandbytes-moe-load-in-4bit-still-ooms.md · run-moe-larger-than-vram.md · offload-moe-experts-to-cpu-or-nvme.md · mxfp4-moe-training-and-residency.md
• ../CHOOSING.md · ../METHODOLOGY.md · ../STATUS.md · ../../README.md

Evidence

Register: ../claims.json; all of these are measured with receipts in this repository.

• e4b.train.olmoe-converges — QLoRA on frozen NF4 experts improves a held-out eval on OLMoE-1B-7B.
• e4b.train.ablation — adapter placement: experts+attention is the best eval, attention-only the efficiency pick; training the router hurts.
• e4b.train.fast-train-dgrad — enable_fast_train(dgrad=True) is the fastest lane at real width; the batched path wins only the microbench.
• e4b.train.flagship-matrix — two 30B-class MoEs by five datasets: fused path faster per step at lower VRAM and energy, loss parity on the registered criteria, frozen stack bit-identical.
• e4b.offload.fits-30b-class — with expert offload, Qwen3-30B-A3B and Gemma-4-26B-A4B QLoRA-train on 12 GB.
• e4b.train.parity.tp1.olmoe.fused.2026-09-05, e4b.train.parity.tp1.qwen3.fused.2026-09-05, e4b.train.parity.tp1.gemma4.fused.2026-09-05, e4b.train.parity.tp1.mixtral.fused.2026-09-05, e4b.train.parity.tp1.granite.fused.2026-09-05, e4b.train.parity.tp1.granite.batched.2026-09-05, e4b.train.parity.tp1.mixtral.batched.2026-09-05 — tp1 (real weights, the shipped code, one RTX 5090): the fused path passes on every family that has one and the batched path on Granite and Mixtral, in the registered units; e4b.train.parity.tp1.<family>.matrix.2026-09-05 per family (load + verify + every arm's row, including the VOID and the refusals); every row of the lane is measured.
• e4b.train.h2h.unsloth.qwen3.5090.2026-09-05 — the head-to-head against Unsloth's 4-bit MoE QLoRA path, end-to-end, on one identical training problem (one rented RTX 5090): the s/step ratio, peak VRAM, energy and time-to-target at 60 steps; e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.quality-n60 — held-out loss comparable at 60 steps; e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.curve-n200 — Unsloth's held-out loss lower at 200 steps, quoted beside the position; e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.e4b-internal-parity — the fused-vs-reference PASS on that box (informational); one row per arm under e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.arm.*.
• e4b.train.h2h.unsloth.<family>.5090.2026-09-06 — tp2/P40, the head-to-head per family (all six, one rented RTX 5090, P38's fixture): positions on Qwen3 (e4b.train.h2h.unsloth.qwen3.5090.2026-09-06, 1.457 — the cross-lane anchor, +3.1% from P38's 1.413) and Mixtral (e4b.train.h2h.unsloth.mixtral.5090.2026-09-06, 0.361 — led by the footprint trade: Unsloth resident vs e4b expert-offload), each with a .quality-n60 COMPARABLE row; e4b.train.h2h.unsloth.mixtral.5090.2026-09-06.footprint — the 3.223 GB offload peak against the 29.163 GB resident peak at comparable quality, the trainable-on-smaller-cards result; .coverage rows on Granite and OLMoE (the comparator could not train the experts there — attention-only LoRA / a crash at MoE-LoRA engage — with its own log lines quoted); .e4b-internal-parity PASS rows on Granite, OLMoE, Qwen3 and Mixtral; and one row per attempt under e4b.train.h2h.unsloth.<family>.5090.2026-09-06.arm.*, including Granite's Unsloth VOID, OLMoE's Unsloth HARNESS_ERROR, gpt-oss's three REFUSED rows and Gemma-4's two void_attn4 rows (#412).

Common wrong approaches

• Applying PEFT's LoRA to the model and reading a loss curve that only trains attention: PEFT never sees the fused experts.
• Calling enable_fast_train without asserting its return: 0 means the step is still on the per-expert loop.

Source and freshness

This page is a rendering of docs/solutions/qlora-fused-moe-experts.md at commit 0c2a256dcdc2 (sha256 23c02bb6a36826f5…). Numbers are never copied here: every measured statement cites a claim ID in docs/claims.json at that commit. Repository-relative links resolve to this site's pages where the document is published and to the pinned commit otherwise.