Solutions — one page per problem
Choose the problem you are trying to solve. Each guide gives an install route, a working example, the result to expect, and the limits of the approach, with links to its evidence.
experts4bit-qlora — docs/SOLUTIONS.md at 605d21e67958
| Problem | Package |
|---|---|
| Why does `load_in_4bit` still OOM on a Mixture-of-Experts model? bitsandbytes' 4-bit walker replaces only nn.Linear and skips a fused MoE's 3-D expert stacks, so load_moe_4bit_streaming quantises exactly those stacks and verify_moe_4bit proves it. | experts4bit-qlora |
| How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)? Choose an MXFP4 loading or native-byte execution route by model and task; gpt-oss arena training is refused, while its separate experimental expert-training path remains unlicensed. | experts4bit-qlora |
| How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena? Bind pinned host RAM or a baked NVMe arena to a real model with the streaming loader and the nvme_residency, mxfp4_nvme_residency and nvme_train_residency engines. | experts4bit-qlora |
| How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)? ExpertsLoRA adds a trainable per-expert low-rank delta before each routed expert's activation over a frozen NF4 stack PEFT cannot target; enable_fast_train runs it on grouped kernels. | experts4bit-qlora |
| How can I run a Mixture-of-Experts model larger than my GPU's VRAM? Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena. | experts4bit-qlora |
| How do I serve a large MoE on a consumer GPU (RTX 5090 class)? The paged decode runner with experts on grouped-nf4-gemm's kernels is the measured serving path on one RTX 5090-class GPU; the HTTP shim is a separate reference-path deployment. | experts4bit-qlora |
grouped-nf4-gemm — docs/SOLUTIONS.md at 30adb1bfa76f
| Problem | Package |
|---|---|
| How do I run paged decode attention over an FP8 KV cache for a quantized MoE serving path, with sliding windows, attention sinks and a custom scale? fp8_paged_decode_attention runs paged flash-decode over an E4M3 KV cache with windows, sinks and a custom scale; the fp8 compute path is supported on sm_89+, the f32 path (the sm_80–sm_88 default) since #319's close. | grouped-nf4-gemm |
| How do I run a single-token INT4 decode GEMV over routed MoE experts, and pack calibrated (GPTQ) weights for it? gemv_int4_b32 runs the batch-1 int4-b32 decode GEMV with exact integer accumulation, and gptq_pack_int4_b32 packs calibrated weights onto the same bytes; the served GEMV's throughput and the calibrated pack's quality are measured-private, while the small-M GEMM and the two opt-in GEMV variants are measured in public, and the split-K planner plans from (N, K) alone since K30 read its row-count term OFF. | grouped-nf4-gemm |
| How do I run MoE expert inference natively on MXFP4 (e2m1 + e8m0) weights, straight from the released checkpoint bytes? gemm_mxfp4_grouped multiplies gpt-oss and Kimi-class experts on their released e2m1 blocks and e8m0 scales, with no requantization to NF4 and no bf16 materialization. | grouped-nf4-gemm |
| How do I run a grouped GEMM directly on NF4 packed MoE expert weights, without dequantizing to bf16 first? gemm_4bit_grouped runs the routed MoE expert GEMM on bitsandbytes NF4 packed weights in one Triton launch, decoding nibbles in registers with fp32 accumulation and no bf16 expert tensor. | grouped-nf4-gemm |
| How do I run a MoE whose experts do not fit in VRAM, streaming them from pinned host RAM or an NVMe arena? The kernel-side storage primitives for experts that do not fit VRAM: the NVMe arena bake and O_DIRECT reader, the pinned-DRAM row tier and low-level residency, and the GPU-driven host gather. | grouped-nf4-gemm |
| How do I verify that the quantized expert bytes I serve or train on are the released checkpoint bytes? file_tensor_sha256, provenance_table and verify_arena_matches hash safetensors byte ranges against the loaded and baked bytes, and verify_provenance re-derives a run's hash table. | grouped-nf4-gemm |
rendered from experts4bit-qlora 0.58.1 · source: pjordanandrsn/experts4bit-qlora@605d21e67958 · checked 2026-10-25
rendered from grouped-nf4-gemm 0.51.0 · source: pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f · checked 2026-10-25