# MoE Training & Inference Guides — Cerin Amroth

Source: https://cerinamroth.com/ml/solutions/

[Home](https://cerinamroth.com/) › [ML systems](https://cerinamroth.com/ml/) › Solutions 

# Solutions — one page per problem

Choose the problem you are trying to solve. Each guide gives an install route, a working example, the result to expect, and the limits of the approach, with links to its evidence.

 

## `experts4bit-qlora` — [docs/SOLUTIONS.md](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/SOLUTIONS.md) at [605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d)

 | Problem | Package
 | [Why does `load_in_4bit` still OOM on a Mixture-of-Experts model?](https://cerinamroth.com/ml/solutions/bitsandbytes-moe-load-in-4bit-still-ooms/)
bitsandbytes' 4-bit walker replaces only nn.Linear and skips a fused MoE's 3-D expert stacks, so load_moe_4bit_streaming quantises exactly those stacks and verify_moe_4bit proves it. | [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/)
 | [How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)?](https://cerinamroth.com/ml/solutions/mxfp4-moe-training-and-residency/)
Choose an MXFP4 loading or native-byte execution route by model and task; gpt-oss arena training is refused, while its separate experimental expert-training path remains unlicensed. | [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/)
 | [How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena?](https://cerinamroth.com/ml/solutions/offload-moe-experts-to-cpu-or-nvme/)
Bind pinned host RAM or a baked NVMe arena to a real model with the streaming loader and the nvme_residency, mxfp4_nvme_residency and nvme_train_residency engines. | [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/)
 | [How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)?](https://cerinamroth.com/ml/solutions/qlora-fused-moe-experts/)
ExpertsLoRA adds a trainable per-expert low-rank delta before each routed expert's activation over a frozen NF4 stack PEFT cannot target; enable_fast_train runs it on grouped kernels. | [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/)
 | [How can I run a Mixture-of-Experts model larger than my GPU's VRAM?](https://cerinamroth.com/ml/solutions/run-moe-larger-than-vram/)
Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena. | [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/)
 | [How do I serve a large MoE on a consumer GPU (RTX 5090 class)?](https://cerinamroth.com/ml/solutions/serve-large-moe-on-a-consumer-gpu/)
The paged decode runner with experts on grouped-nf4-gemm's kernels is the measured serving path on one RTX 5090-class GPU; the HTTP shim is a separate reference-path deployment. | [`experts4bit-qlora`](https://cerinamroth.com/ml/experts4bit-qlora/)

## `grouped-nf4-gemm` — [docs/SOLUTIONS.md](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/SOLUTIONS.md) at [30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca)

 | Problem | Package
 | [How do I run paged decode attention over an FP8 KV cache for a quantized MoE serving path, with sliding windows, attention sinks and a custom scale?](https://cerinamroth.com/ml/solutions/fp8-paged-attention-for-moe-serving/)
fp8_paged_decode_attention runs paged flash-decode over an E4M3 KV cache with windows, sinks and a custom scale; the fp8 compute path is supported on sm_89+, the f32 path (the sm_80–sm_88 default) since #319's close. | [`grouped-nf4-gemm`](https://cerinamroth.com/ml/grouped-nf4-gemm/)
 | [How do I run a single-token INT4 decode GEMV over routed MoE experts, and pack calibrated (GPTQ) weights for it?](https://cerinamroth.com/ml/solutions/int4-decode-gemv/)
gemv_int4_b32 runs the batch-1 int4-b32 decode GEMV with exact integer accumulation, and gptq_pack_int4_b32 packs calibrated weights onto the same bytes; the served GEMV's throughput and the calibrated pack's quality are measured-private, while the small-M GEMM and the two opt-in GEMV variants are measured in public, and the split-K planner plans from (N, K) alone since K30 read its row-count term OFF. | [`grouped-nf4-gemm`](https://cerinamroth.com/ml/grouped-nf4-gemm/)
 | [How do I run MoE expert inference natively on MXFP4 (e2m1 + e8m0) weights, straight from the released checkpoint bytes?](https://cerinamroth.com/ml/solutions/native-mxfp4-moe-inference/)
gemm_mxfp4_grouped multiplies gpt-oss and Kimi-class experts on their released e2m1 blocks and e8m0 scales, with no requantization to NF4 and no bf16 materialization. | [`grouped-nf4-gemm`](https://cerinamroth.com/ml/grouped-nf4-gemm/)
 | [How do I run a grouped GEMM directly on NF4 packed MoE expert weights, without dequantizing to bf16 first?](https://cerinamroth.com/ml/solutions/nf4-grouped-gemm-without-bf16-materialization/)
gemm_4bit_grouped runs the routed MoE expert GEMM on bitsandbytes NF4 packed weights in one Triton launch, decoding nibbles in registers with fp32 accumulation and no bf16 expert tensor. | [`grouped-nf4-gemm`](https://cerinamroth.com/ml/grouped-nf4-gemm/)
 | [How do I run a MoE whose experts do not fit in VRAM, streaming them from pinned host RAM or an NVMe arena?](https://cerinamroth.com/ml/solutions/stream-moe-experts-from-host-or-nvme/)
The kernel-side storage primitives for experts that do not fit VRAM: the NVMe arena bake and O_DIRECT reader, the pinned-DRAM row tier and low-level residency, and the GPU-driven host gather. | [`grouped-nf4-gemm`](https://cerinamroth.com/ml/grouped-nf4-gemm/)
 | [How do I verify that the quantized expert bytes I serve or train on are the released checkpoint bytes?](https://cerinamroth.com/ml/solutions/verify-quantized-checkpoint-provenance/)
file_tensor_sha256, provenance_table and verify_arena_matches hash safetensors byte ranges against the loaded and baked bytes, and verify_provenance re-derives a run's hash table. | [`grouped-nf4-gemm`](https://cerinamroth.com/ml/grouped-nf4-gemm/)

 

rendered from experts4bit-qlora 0.58.1 · source: [pjordanandrsn/experts4bit-qlora@605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) · checked 2026-10-25

rendered from grouped-nf4-gemm 0.51.0 · source: [pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) · checked 2026-10-25

 

Sources and publication details

The generated ML documentation surfaces on this site (`/ml/` and its sub-pages, `/ml/packages.json`, both `llms.txt` files) are derived from the repositories below at pinned commits; each repository's `docs/claims.json` at that commit is the only source for numbers. Pages with their own source are listed separately.

- [pjordanandrsn/experts4bit-qlora@605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) (2026-10-24) · package `experts4bit-qlora` 0.58.1 · PyPI 0.58.1 (published) · documents: [`docs/SOLUTIONS.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/SOLUTIONS.md), [`docs/solutions/bitsandbytes-moe-load-in-4bit-still-ooms.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/bitsandbytes-moe-load-in-4bit-still-ooms.md), [`docs/solutions/mxfp4-moe-training-and-residency.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/mxfp4-moe-training-and-residency.md), [`docs/solutions/offload-moe-experts-to-cpu-or-nvme.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/offload-moe-experts-to-cpu-or-nvme.md), [`docs/solutions/qlora-fused-moe-experts.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/qlora-fused-moe-experts.md), [`docs/solutions/run-moe-larger-than-vram.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/run-moe-larger-than-vram.md), [`docs/solutions/serve-large-moe-on-a-consumer-gpu.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/serve-large-moe-on-a-consumer-gpu.md) · registers: none · synced 2026-10-25T00:58:32Z · [files at this commit](https://github.com/pjordanandrsn/experts4bit-qlora/tree/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) · [latest on main](https://github.com/pjordanandrsn/experts4bit-qlora) (unpinned; not the source of any fact rendered here)
- [pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) (2026-10-24) · package `grouped-nf4-gemm` 0.51.0 · PyPI 0.51.0 (published) · documents: [`docs/SOLUTIONS.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/SOLUTIONS.md), [`docs/solutions/fp8-paged-attention-for-moe-serving.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/fp8-paged-attention-for-moe-serving.md), [`docs/solutions/int4-decode-gemv.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/int4-decode-gemv.md), [`docs/solutions/native-mxfp4-moe-inference.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/native-mxfp4-moe-inference.md), [`docs/solutions/nf4-grouped-gemm-without-bf16-materialization.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/nf4-grouped-gemm-without-bf16-materialization.md), [`docs/solutions/stream-moe-experts-from-host-or-nvme.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/stream-moe-experts-from-host-or-nvme.md), [`docs/solutions/verify-quantized-checkpoint-provenance.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/verify-quantized-checkpoint-provenance.md) · registers: none · synced 2026-10-24T14:51:49Z · [files at this commit](https://github.com/pjordanandrsn/grouped-nf4-gemm/tree/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) · [latest on main](https://github.com/pjordanandrsn/grouped-nf4-gemm) (unpinned; not the source of any fact rendered here)