# experts4bit-qlora

Train and serve Mixture-of-Experts models that do not fit in VRAM: fused 4-bit experts, QLoRA, CPU/NVMe offload, and fast inference on consumer NVIDIA GPUs.

Canonical package: `experts4bit-qlora` (https://pypi.org/project/experts4bit-qlora/). Repository: https://github.com/pjordanandrsn/experts4bit-qlora. Import: experts4bit_qlora. Aliases (lookup only, from the system manifest): e4b, e4b-qlora, experts4bit, expertsnbit, experts-mxfp4.

System manifest: runtime package; one system, two independently released packages: the runtime package depends on the kernel package through its optional `fast` extra; the kernel package never depends on the runtime package Dependency direction: experts4bit-qlora -> grouped-nf4-gemm. Manifest: https://cerinamroth.com/ml/system-manifest.json (provenance https://cerinamroth.com/ml/system-manifest.provenance.json).

Owns: model loading (streaming loader, fused expert stacks, Experts4bit/ExpertsNbit bases); quantization orchestration (NF4/MXFP4 routes, int4 expert store, calibration drivers); adapters (ExpertsLoRA); training (trainer, fast/batched/hybrid paths, checkpointing); residency integration (host-RAM and NVMe engines bound to a loaded model); serving (paged runner, decode stack, HTTP shim); model-level measurement, quality gates and the claims register for model-level numbers

Claim register: claims tiered in the public register: 161 measured · 6 measured-private · 3 open · 7 superseded · 6 retired — https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/claims.json

Current status: v0.35.3 — https://cerinamroth.com/ml/status/#experts4bit-qlora

## Direct answer

Use this package when load_in_4bit=True does not actually quantise a model's fused expert tensors. transformers v5 stores a MoE's experts as one fused three-dimensional parameter per layer rather than ordinary nn.Linear modules, so the standard bitsandbytes module walker can leave most of the expert weights in bf16 and the model still OOMs; this package quantises exactly that fused stack, fine-tunes it with per-expert LoRA, keeps experts in host RAM or on NVMe when they do not fit, and serves the result on one consumer NVIDIA GPU.

## Use this when

- A Mixture-of-Experts model loaded with load_in_4bit still occupies bf16-sized VRAM and OOMs, because transformers stores its experts as one fused 3-D parameter per layer that bitsandbytes' nn.Linear walker never touches.
- Fine-tune (QLoRA / LoRA) a fused-expert MoE whose expert weights PEFT and the bitsandbytes walker never see.
- The quantised experts still do not fit the GPU: run a Mixture-of-Experts model larger than VRAM by keeping experts resident somewhere else and streaming them per layer.
- Experts do not even fit host RAM: serve or train from an on-disk expert arena (NVMe) with a pinned-DRAM hot tier.
- Serve a large fused-expert MoE on one consumer NVIDIA GPU with the decode path measured against the model's own attention.
- Work with models whose experts are released as native MXFP4 (gpt-oss, DeepSeek-V4): load them faithfully, keep the released bytes in an NVMe arena, train on them, and serve them without re-quantisation.

## Do not use this when

- dense (non-MoE) models: use bitsandbytes directly
- a general-purpose serving engine or a vLLM replacement
- models that already fit in bf16 and need no offload: 4-bit here is a memory trade, not a speed-up, and on the measured comparator it cost energy (claim e4b.train.energy-honest.scoped-a2000)
- silent fallbacks: every enable_* returns a count or a non-empty handle list, or raises; callers must assert it
- operating systems other than Linux, or GPUs without CUDA

## Install

```bash
pip install "experts4bit-qlora[train]"
```

Primary route: streaming loader + trainer.

Alternatives:

- `pip install "experts4bit-qlora[fast]"` — accelerated grouped-kernel path
- `pip install "experts4bit-qlora[serve]"` — HTTP shim (optional)

Environment: OS: Linux · Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested) · Accelerator: NVIDIA CUDA GPU (bitsandbytes 4-bit); NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)

## Problems this solves

- [Why does `load_in_4bit` still OOM on a Mixture-of-Experts model?](https://cerinamroth.com/ml/solutions/bitsandbytes-moe-load-in-4bit-still-ooms/): bitsandbytes' 4-bit walker replaces only nn.Linear and skips a fused MoE's 3-D expert stacks, so load_moe_4bit_streaming quantises exactly those stacks and verify_moe_4bit proves it.
- [How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)?](https://cerinamroth.com/ml/solutions/mxfp4-moe-training-and-residency/): Choose between the convenient QLoRA path, which decodes MXFP4 and re-quantises to NF4, and the native-byte path, which keeps the released blocks and scales in an arena.
- [How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena?](https://cerinamroth.com/ml/solutions/offload-moe-experts-to-cpu-or-nvme/): Bind pinned host RAM or a baked NVMe arena to a real model with the streaming loader and the nvme_residency, mxfp4_nvme_residency and nvme_train_residency engines.
- [How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)?](https://cerinamroth.com/ml/solutions/qlora-fused-moe-experts/): ExpertsLoRA adds a trainable per-expert low-rank delta before each routed expert's activation over a frozen NF4 stack PEFT cannot target; enable_fast_train runs it on grouped kernels.
- [How can I run a Mixture-of-Experts model larger than my GPU's VRAM?](https://cerinamroth.com/ml/solutions/run-moe-larger-than-vram/): Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena.
- [How do I serve a large MoE on a consumer GPU (RTX 5090 class)?](https://cerinamroth.com/ml/solutions/serve-large-moe-on-a-consumer-gpu/): The paged decode runner with experts on grouped-nf4-gemm's kernels is the measured serving path on one RTX 5090 class; the HTTP shim is a separate reference-path deployment.

## Evidence

Active claim IDs: e4b.train.olmoe-converges, e4b.offload.fits-30b-class, e4b.serve.census.bo7.qwen3.b1.5090.2026-09-05, e4b.parity.moe-routing-flip-floor, e4b.train.energy-honest.scoped-a2000 — see https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/claims.json (pinned; latest on main, unpinned; not the source of any fact rendered here: https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/claims.json).

## Freshness

CURRENT · source: pjordanandrsn/experts4bit-qlora@0c2a256dcdc2 · rendered package: 0.35.3 · latest published package: 0.35.3

## Provenance

Source commit: 0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd (2026-09-05); repository version 0.35.3; latest PyPI 0.35.3 (published); synced 2026-09-06T04:07:55Z. Pinned: https://github.com/pjordanandrsn/experts4bit-qlora/tree/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd · capabilities https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/capabilities.json · status https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/STATUS.md · claims https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/claims.json · system manifest https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/system-manifest.json. Latest on main (unpinned; not the source of any fact rendered here): https://github.com/pjordanandrsn/experts4bit-qlora.