# How can I run a Mixture-of-Experts model larger than my GPU's VRAM?

Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena.

Solved by `experts4bit-qlora` (https://cerinamroth.com/ml/experts4bit-qlora/). Source: https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/solutions/run-moe-larger-than-vram.md (pinned 0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd; latest on main, unpinned; not the source of any fact rendered here: https://github.com/pjordanandrsn/experts4bit-qlora/blob/main/docs/solutions/run-moe-larger-than-vram.md)

Freshness: CURRENT · source: pjordanandrsn/experts4bit-qlora@0c2a256dcdc2 · rendered package: 0.35.3 · latest published package: 0.35.3

## Install routes

```bash
pip install "experts4bit-qlora[train]"
```

Primary route: host-RAM training route.

Alternatives:

- `pip install "experts4bit-qlora[fast]"` — residency/NVMe/fast-kernel route

Environment: OS: Linux · Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested) · Accelerator: NVIDIA CUDA GPU (bitsandbytes 4-bit)

<!-- summary: Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena. -->

**This is the decision page**: choose by workload and by the memory tier that ran out. Model-level integration — the loaders and the `experts4bit-qlora` engines that bind pinned host RAM or an NVMe arena to a real model — is [`offload-moe-experts-to-cpu-or-nvme.md`](https://cerinamroth.com/ml/solutions/offload-moe-experts-to-cpu-or-nvme/); the arena bake, reader and tier primitives themselves are the kernel package's page, [stream-moe-experts-from-host-or-nvme.md](https://cerinamroth.com/ml/solutions/stream-moe-experts-from-host-or-nvme/).

Start from what ran out. If the 4-bit experts exceed VRAM, `load_moe_4bit_streaming(..., offload=True)` homes them in pinned host RAM and streams one layer at a time; if the dense side does not fit, `enable_dense_offload`; if the experts do not fit host RAM either, an NVMe arena; if you have spare VRAM to trade at serve time, `enable_pipelined_residency` with hot sets chosen from a routing profile.

Four things to hold before choosing. **Fitting is not a speed claim.** **Layer-granular host streaming is the capacity floor** — it is what makes the model run at all. **Profiled or paged residency is the serving-performance path.** **Absolute throughput is host-specific**: only ratios travel between hosts (claim `e4b.host.ratios-travel-absolutes-do-not`).

## First choose the workload

| workload | start with | escalate to |
|---|---|---|
| QLoRA training | pinned-host 4-bit expert streaming (`offload=True`; `OFFLOAD_EXPERTS=1` in the trainer) | NVMe training residency (`enable_nvme_train_residency`) when host RAM also runs out |
| Serving / generation | streaming, or profiled residency (`hot_sets_from_profile` + `enable_pipelined_residency`) | the NVMe arena (`enable_nvme_residency`, `enable_mxfp4_nvme_residency`) when the expert store exceeds host RAM |

## What fits, from the register

Training and serving are different workloads measured on different hardware; the two examples below share nothing but the register, and neither implies a throughput.

- **Training** — claim `e4b.offload.fits-30b-class` (measured): with `OFFLOAD_EXPERTS=1`, Qwen3-30B-A3B peaks at 7.16 GB of GPU memory during a QLoRA step and Gemma-4-26B-A4B at 8.47 GB; both OOM without offload on the 12 GB card. Fit, not speed.
- **Serving** — claim `e4b.serve.deepseek-v4` (measured): DeepSeek-V4-Flash, 284B parameters, loads in about 10 s at 8.74 GiB peak VRAM and generates with its 147 GB expert arena served from disk. Fit, not speed: that path's decode rate is a property of the host link and is not quoted here.

```text
VRAM overflow
  -> experts:       pinned-host expert streaming
  -> dense side:    dense offload
  -> spare VRAM:    profile-ranked hot residency
Host-RAM overflow
  -> expert store:  NVMe arena -> serving residency / training residency + checkpointing
```

## Symptoms

- "OOM even after 4-bit" on a 30B-class MoE (Qwen3-30B-A3B, Gemma-4-26B-A4B) on a 12 GB or 24 GB card.
- "run a MoE larger than VRAM" / "the experts don't fit" / "stream experts from host RAM".
- Unsure which path to take: the reference per-expert loop, the batched trainer, the fused kernel, host-streamed experts, or NVMe.
- The card fits the experts but not the attention and embeddings (the dense side).

## Why it happens

A MoE's weights are mostly experts, and each token touches only its top-k of them, so the whole model never has to be resident: the dense side plus one layer's experts is enough for a forward, and at decode only the routed rows are needed. What ran out decides where the rest lives — pinned host RAM across PCIe, or an on-disk arena read at the device link — and frozen 4-bit storage keeps the bytes that move small.

## Which project solves it

**experts4bit-qlora** decides which expert bytes are where: the streaming loader with `offload=True`, `enable_dense_offload` for the non-expert weights, `hot_sets_from_profile` for choosing resident experts, `enable_pipelined_residency` for the serve-time hot/cold split, and the `enable_nvme_*` engines. **grouped-nf4-gemm** ([GitHub](https://github.com/pjordanandrsn/grouped-nf4-gemm), [PyPI](https://pypi.org/project/grouped-nf4-gemm/)) supplies the fused grouped GEMM the pipelined engine runs on and the arena reader and tier the NVMe engines bind to; `[fast]` is the seam.

| what ran out | call | needs |
|---|---|---|
| the experts do not fit VRAM | `load_moe_4bit_streaming(..., offload=True)` (`OFFLOAD_EXPERTS=1` in the CLIs) | `[train]` |
| the dense side does not fit | `enable_dense_offload(model, "cuda")`; `DenseDiskSource(path)` when host RAM cannot hold it either | — |
| the experts do not fit host RAM, serving | `enable_nvme_residency(...)` / `enable_mxfp4_nvme_residency(...)` | `[fast]` + arena |
| the experts do not fit host RAM, training | `enable_nvme_train_residency(...)` | `[fast]` + arena + grad ckpt |
| serving, spare VRAM to trade | `enable_pipelined_residency(model, hot_sets, k_slots=k)` | `[fast]` |
| small GPU, strong CPU | `enable_cold_engine(model, hot_sets, dequant="auto")` | — |

## Install

```bash
pip install "experts4bit-qlora[train]"   # host-RAM training route: loader + pinned-host expert streaming; no kernel package needed
pip install "experts4bit-qlora[fast]"    # residency/NVMe/fast-kernel route: + grouped-nf4-gemm for the residency engines and NVMe arenas
```

## Smallest correct example

Needs: GPU + network + model download.

```python
import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16, r=8, alpha=16, quant_type="nf4",
    offload=True, pin=True, prefetch=True,   # prefetch: next layer's H2D copy overlaps this layer (no_grad only)
)
verify_moe_4bit(model, strict=True)
model.eval()
# do NOT call model.to("cuda"): the experts live in pinned host RAM by design
```

Or from the CLI: `OFFLOAD_EXPERTS=1 BENCH_TOKENS=128 python -m experts4bit_qlora.infer`. Dense side too large too? `enable_dense_offload(model, "cuda")` composes with it.

## Expected result

`verify_moe_4bit(model, strict=True)` returns without raising, and `torch.cuda.max_memory_allocated()` during a forward stays near one layer's experts plus the dense side rather than the whole model. `enable_dense_offload` returns a non-empty list of per-layer handles, described by `dense_offload_report(handles)`. Every `enable_*` returns a count or a non-empty handle list, or raises — assert it.

## Supported scope

- Families: those of the loader ([`bitsandbytes-moe-load-in-4bit-still-ooms.md`](https://cerinamroth.com/ml/solutions/bitsandbytes-moe-load-in-4bit-still-ooms/)). Offload identity is tested on nf4 / int8 / bf16; fp4 / fp8 / fp16 ride the same code path ([`../STORAGE-MODES.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/STORAGE-MODES.md)).
- Offload training requires gradient checkpointing (`use_reentrant=False`); the shipped trainer always enables it.
- Environment: Linux, NVIDIA CUDA, torch>=2.2, bitsandbytes>=0.43, transformers>=5.0; CI tests Python 3.11. Residency engines need grouped-nf4-gemm at the `fast` extra's floor in `pyproject.toml` (grouped-nf4-gemm >= 0.30.0 at this commit; validated by CI), with Triton (Linux-only) on an sm_80-or-newer GPU.

## Limitations

- Bulk layer-granular offload is PCIe-bound for decode at 26–30B scale; the v0 decode grid in [`../INFERENCE.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/INFERENCE.md) is **superseded** for decode by the pipelined and paged engines (claim `e4b.retired.inference-md-decode-grid`).
- Pick hot sets from a routing histogram (`E4B_EXPERT_PROFILE`, `hot_sets_from_profile`), never by index: an index-ordered set is a uniform random draw. The gain is a property of the host link and did not replicate on a fat-PCIe box ([`../RESIDENCY-ENGINES.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/RESIDENCY-ENGINES.md)).
- `enable_pipelined_residency` accepts the `ExpertsLoRA` wrapper the loader installs and patches its base. The patch runs only while the wrapper delegates to the base (eval mode, `no_grad`, an adapter that provably contributes nothing — an untrained `B`); with a trained adapter it installs, never runs, and warns, which is why `experts4bit_qlora.serve` treats residency and trained adapters as mutually exclusive. Assert the count and check the served path ([`../CHOOSING.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/CHOOSING.md), [`../RESIDENCY-ENGINES.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/RESIDENCY-ENGINES.md)).
- `enable_hot_residency` is deprecated in favour of `enable_pipelined_residency`.
- Absolutes are host-specific; only ratios travel (claim `e4b.host.ratios-travel-absolutes-do-not`).

## Use this page when…

- **you are choosing** a path by workload and by the memory tier that ran out — this page, the decision/router page.
- **you are wiring a model** — `load_moe_4bit_streaming(..., offload=True)`, the arena loader (`arena=`, `arena_train=True`), `enable_nvme_residency` / `enable_mxfp4_nvme_residency` / `enable_nvme_train_residency` — [`offload-moe-experts-to-cpu-or-nvme.md`](https://cerinamroth.com/ml/solutions/offload-moe-experts-to-cpu-or-nvme/), the model-level integration page.
- **you are baking, reading or sizing the tier itself** — `nvme_bake_nf4`, `bake_expert_tensors`, the reader, `ColdTier`, `capacity_for_bytes` — the kernel package's [stream-moe-experts-from-host-or-nvme.md](https://cerinamroth.com/ml/solutions/stream-moe-experts-from-host-or-nvme/).

## Related

- [`qlora-fused-moe-experts.md`](https://cerinamroth.com/ml/solutions/qlora-fused-moe-experts/) — training on the offloaded experts.
- [`serve-large-moe-on-a-consumer-gpu.md`](https://cerinamroth.com/ml/solutions/serve-large-moe-on-a-consumer-gpu/) — the paged serving engine.
- [`../CHOOSING.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/CHOOSING.md) · [`../RESIDENCY-ENGINES.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/RESIDENCY-ENGINES.md) · [`../STATUS.md`](https://cerinamroth.com/ml/status/#experts4bit-qlora)

## Evidence

Register: [`../claims.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/0c2a256dcdc2cb0a83cf7692224a8aa716f61ecd/docs/claims.json).

- `e4b.offload.fits-30b-class` — measured: with `OFFLOAD_EXPERTS=1`, Qwen3-30B-A3B and Gemma-4-26B-A4B train on 12 GB where they OOM without it.
- `e4b.serve.informed-hot-sets` — measured: profile-ranked hot sets beat by-index at identical VRAM; the gain is a property of the host.
- `e4b.serve.deepseek-v4` — measured: DeepSeek-V4-Flash loads and generates with its experts served from an on-disk arena.
- `e4b.host.ratios-travel-absolutes-do-not` — measured: absolute s/step and tok/s are host-specific.
- `e4b.retired.inference-md-decode-grid` — superseded: the v0 offload decode grid is not the number to quote.