Solutions — one page per problem
Each page answers one ordinary problem in the same order: the direct answer, symptoms, why it happens, which package solves it, the canonical install, the smallest correct example, the observable result, scope, limitations, common wrong approaches, related pages and evidence by claim ID. Summaries are each document's own <!-- summary: … --> sentence, verbatim; the build refuses a current solution page without one, so nothing here is a first-sentence guess or a truncation.
experts4bit-qlora — docs/SOLUTIONS.md at 0c2a256dcdc2
| Problem | Package |
|---|---|
| Why does `load_in_4bit` still OOM on a Mixture-of-Experts model? bitsandbytes' 4-bit walker replaces only nn.Linear and skips a fused MoE's 3-D expert stacks, so load_moe_4bit_streaming quantises exactly those stacks and verify_moe_4bit proves it. | experts4bit-qlora |
| How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)? Choose between the convenient QLoRA path, which decodes MXFP4 and re-quantises to NF4, and the native-byte path, which keeps the released blocks and scales in an arena. | experts4bit-qlora |
| How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena? Bind pinned host RAM or a baked NVMe arena to a real model with the streaming loader and the nvme_residency, mxfp4_nvme_residency and nvme_train_residency engines. | experts4bit-qlora |
| How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)? ExpertsLoRA adds a trainable per-expert low-rank delta before each routed expert's activation over a frozen NF4 stack PEFT cannot target; enable_fast_train runs it on grouped kernels. | experts4bit-qlora |
| How can I run a Mixture-of-Experts model larger than my GPU's VRAM? Choose a residency path by workload and by the memory tier that ran out: pinned-host expert streaming, dense offload, profiled hot residency, or an NVMe arena. | experts4bit-qlora |
| How do I serve a large MoE on a consumer GPU (RTX 5090 class)? The paged decode runner with experts on grouped-nf4-gemm's kernels is the measured serving path on one RTX 5090 class; the HTTP shim is a separate reference-path deployment. | experts4bit-qlora |
grouped-nf4-gemm — docs/SOLUTIONS.md at e2af4cfb91b2
| Problem | Package |
|---|---|
| How do I run paged decode attention over an FP8 KV cache for a quantized MoE serving path, with sliding windows, attention sinks and a custom scale? fp8_paged_decode_attention runs paged flash-decode over an E4M3 KV cache with windows, sinks and a custom scale; the fp8 compute path is supported on sm_89+, the f32 path is open under #319. | grouped-nf4-gemm |
| How do I run a single-token INT4 decode GEMV over routed MoE experts, and pack calibrated (GPTQ) weights for it? gemv_int4_b32 runs the batch-1 int4-b32 decode GEMV with exact integer accumulation, and gptq_pack_int4_b32 packs calibrated weights onto the same bytes; its numbers are measured-private. | grouped-nf4-gemm |
| How do I run MoE expert inference natively on MXFP4 (e2m1 + e8m0) weights, straight from the released checkpoint bytes? gemm_mxfp4_grouped multiplies gpt-oss and Kimi-class experts on their released e2m1 blocks and e8m0 scales, with no requantization to NF4 and no bf16 materialization. | grouped-nf4-gemm |
| How do I run a grouped GEMM directly on NF4 packed MoE expert weights, without dequantizing to bf16 first? gemm_4bit_grouped runs the routed MoE expert GEMM on bitsandbytes NF4 packed weights in one Triton launch, decoding nibbles in registers with fp32 accumulation and no bf16 expert tensor. | grouped-nf4-gemm |
| How do I run a MoE whose experts do not fit in VRAM, streaming them from pinned host RAM or an NVMe arena? The kernel-side storage primitives for experts that do not fit VRAM: the NVMe arena bake and O_DIRECT reader, the pinned-DRAM row tier and low-level residency, and the GPU-driven host gather. | grouped-nf4-gemm |
| How do I verify that the quantized expert bytes I serve or train on are the released checkpoint bytes? file_tensor_sha256, provenance_table and verify_arena_matches hash safetensors byte ranges against the loaded and baked bytes, and verify_provenance re-derives a run's hash table. | grouped-nf4-gemm |
CURRENT · source: pjordanandrsn/experts4bit-qlora@0c2a256dcdc2 · rendered package: 0.35.3 · latest published package: 0.35.3
CURRENT · source: pjordanandrsn/grouped-nf4-gemm@e2af4cfb91b2 · rendered package: 0.30.2 · latest published package: 0.30.2