How can I run a Mixture-of-Experts model larger than my GPU's VRAM?

Solved by experts4bit-qlora · this page in the repository (pinned 0c2a256dcdc2, the source of this rendering; latest on main, unpinned; not the source of any fact rendered here)

Install routes

From docs/capabilities.json at the pinned commit.

pip install "experts4bit-qlora[train]"

This is the decision page: choose by workload and by the memory tier that ran out. Model-level integration — the loaders and the experts4bit-qlora engines that bind pinned host RAM or an NVMe arena to a real model — is offload-moe-experts-to-cpu-or-nvme.md; the arena bake, reader and tier primitives themselves are the kernel package's page, stream-moe-experts-from-host-or-nvme.md.

Start from what ran out. If the 4-bit experts exceed VRAM, load_moe_4bit_streaming(..., offload=True) homes them in pinned host RAM and streams one layer at a time; if the dense side does not fit, enable_dense_offload; if the experts do not fit host RAM either, an NVMe arena; if you have spare VRAM to trade at serve time, enable_pipelined_residency with hot sets chosen from a routing profile.

Four things to hold before choosing. Fitting is not a speed claim. Layer-granular host streaming is the capacity floor — it is what makes the model run at all. Profiled or paged residency is the serving-performance path. Absolute throughput is host-specific: only ratios travel between hosts (claim e4b.host.ratios-travel-absolutes-do-not).

First choose the workload

What fits, from the register

Training and serving are different workloads measured on different hardware; the two examples below share nothing but the register, and neither implies a throughput.

• Training — claim e4b.offload.fits-30b-class (measured): with OFFLOAD_EXPERTS=1, Qwen3-30B-A3B peaks at 7.16 GB of GPU memory during a QLoRA step and Gemma-4-26B-A4B at 8.47 GB; both OOM without offload on the 12 GB card. Fit, not speed.
• Serving — claim e4b.serve.deepseek-v4 (measured): DeepSeek-V4-Flash, 284B parameters, loads in about 10 s at 8.74 GiB peak VRAM and generates with its 147 GB expert arena served from disk. Fit, not speed: that path's decode rate is a property of the host link and is not quoted here.

VRAM overflow
  -> experts:       pinned-host expert streaming
  -> dense side:    dense offload
  -> spare VRAM:    profile-ranked hot residency
Host-RAM overflow
  -> expert store:  NVMe arena -> serving residency / training residency + checkpointing

Symptoms

• "OOM even after 4-bit" on a 30B-class MoE (Qwen3-30B-A3B, Gemma-4-26B-A4B) on a 12 GB or 24 GB card.
• "run a MoE larger than VRAM" / "the experts don't fit" / "stream experts from host RAM".
• Unsure which path to take: the reference per-expert loop, the batched trainer, the fused kernel, host-streamed experts, or NVMe.
• The card fits the experts but not the attention and embeddings (the dense side).

Why it happens

A MoE's weights are mostly experts, and each token touches only its top-k of them, so the whole model never has to be resident: the dense side plus one layer's experts is enough for a forward, and at decode only the routed rows are needed. What ran out decides where the rest lives — pinned host RAM across PCIe, or an on-disk arena read at the device link — and frozen 4-bit storage keeps the bytes that move small.

Which project solves it

experts4bit-qlora decides which expert bytes are where: the streaming loader with offload=True, enable_dense_offload for the non-expert weights, hot_sets_from_profile for choosing resident experts, enable_pipelined_residency for the serve-time hot/cold split, and the enable_nvme_* engines. grouped-nf4-gemm (GitHub, PyPI) supplies the fused grouped GEMM the pipelined engine runs on and the arena reader and tier the NVMe engines bind to; [fast] is the seam.

Install

pip install "experts4bit-qlora[train]"   # host-RAM training route: loader + pinned-host expert streaming; no kernel package needed
pip install "experts4bit-qlora[fast]"    # residency/NVMe/fast-kernel route: + grouped-nf4-gemm for the residency engines and NVMe arenas

Smallest correct example

Needs: GPU + network + model download.

import torch
from experts4bit_qlora import load_moe_4bit_streaming, verify_moe_4bit

model, config = load_moe_4bit_streaming(
    "Qwen/Qwen3-30B-A3B", "cuda", torch.bfloat16, r=8, alpha=16, quant_type="nf4",
    offload=True, pin=True, prefetch=True,   # prefetch: next layer's H2D copy overlaps this layer (no_grad only)
)
verify_moe_4bit(model, strict=True)
model.eval()
# do NOT call model.to("cuda"): the experts live in pinned host RAM by design

Or from the CLI: OFFLOAD_EXPERTS=1 BENCH_TOKENS=128 python -m experts4bit_qlora.infer. Dense side too large too? enable_dense_offload(model, "cuda") composes with it.

Expected result

verify_moe_4bit(model, strict=True) returns without raising, and torch.cuda.max_memory_allocated() during a forward stays near one layer's experts plus the dense side rather than the whole model. enable_dense_offload returns a non-empty list of per-layer handles, described by dense_offload_report(handles). Every enable_* returns a count or a non-empty handle list, or raises — assert it.

Supported scope

• Families: those of the loader (bitsandbytes-moe-load-in-4bit-still-ooms.md). Offload identity is tested on nf4 / int8 / bf16; fp4 / fp8 / fp16 ride the same code path (../STORAGE-MODES.md).
• Offload training requires gradient checkpointing (use_reentrant=False); the shipped trainer always enables it.
• Environment: Linux, NVIDIA CUDA, torch>=2.2, bitsandbytes>=0.43, transformers>=5.0; CI tests Python 3.11. Residency engines need grouped-nf4-gemm at the fast extra's floor in pyproject.toml (grouped-nf4-gemm >= 0.30.0 at this commit; validated by CI), with Triton (Linux-only) on an sm_80-or-newer GPU.

Limitations

• Bulk layer-granular offload is PCIe-bound for decode at 26–30B scale; the v0 decode grid in ../INFERENCE.md is superseded for decode by the pipelined and paged engines (claim e4b.retired.inference-md-decode-grid).
• Pick hot sets from a routing histogram (E4B_EXPERT_PROFILE, hot_sets_from_profile), never by index: an index-ordered set is a uniform random draw. The gain is a property of the host link and did not replicate on a fat-PCIe box (../RESIDENCY-ENGINES.md).
• enable_pipelined_residency accepts the ExpertsLoRA wrapper the loader installs and patches its base. The patch runs only while the wrapper delegates to the base (eval mode, no_grad, an adapter that provably contributes nothing — an untrained B); with a trained adapter it installs, never runs, and warns, which is why experts4bit_qlora.serve treats residency and trained adapters as mutually exclusive. Assert the count and check the served path (../CHOOSING.md, ../RESIDENCY-ENGINES.md).
• enable_hot_residency is deprecated in favour of enable_pipelined_residency.
• Absolutes are host-specific; only ratios travel (claim e4b.host.ratios-travel-absolutes-do-not).

Use this page when…

• you are choosing a path by workload and by the memory tier that ran out — this page, the decision/router page.
• you are wiring a model — load_moe_4bit_streaming(..., offload=True), the arena loader (arena=, arena_train=True), enable_nvme_residency / enable_mxfp4_nvme_residency / enable_nvme_train_residency — offload-moe-experts-to-cpu-or-nvme.md, the model-level integration page.
• you are baking, reading or sizing the tier itself — nvme_bake_nf4, bake_expert_tensors, the reader, ColdTier, capacity_for_bytes — the kernel package's stream-moe-experts-from-host-or-nvme.md.

Related

• qlora-fused-moe-experts.md — training on the offloaded experts.
• serve-large-moe-on-a-consumer-gpu.md — the paged serving engine.
• ../CHOOSING.md · ../RESIDENCY-ENGINES.md · ../STATUS.md

Evidence

Register: ../claims.json.

• e4b.offload.fits-30b-class — measured: with OFFLOAD_EXPERTS=1, Qwen3-30B-A3B and Gemma-4-26B-A4B train on 12 GB where they OOM without it.
• e4b.serve.informed-hot-sets — measured: profile-ranked hot sets beat by-index at identical VRAM; the gain is a property of the host.
• e4b.serve.deepseek-v4 — measured: DeepSeek-V4-Flash loads and generates with its experts served from an on-disk arena.
• e4b.host.ratios-travel-absolutes-do-not — measured: absolute s/step and tok/s are host-specific.
• e4b.retired.inference-md-decode-grid — superseded: the v0 offload decode grid is not the number to quote.

Common wrong approaches

• Pinning experts by index: a routed expert lands in an index-ordered hot set a few percent of the time; pick hot sets from a routing profile.
• Reading a residency speed-up on one host as a number that travels: absolutes depend on the host link, ratios travel.

Source and freshness

This page is a rendering of docs/solutions/run-moe-larger-than-vram.md at commit 0c2a256dcdc2 (sha256 7cd2e077e6d40730…). Numbers are never copied here: every measured statement cites a claim ID in docs/claims.json at that commit. Repository-relative links resolve to this site's pages where the document is published and to the pinned commit otherwise.