CERIN AMROTH · ML systems

Large Mixture-of-Experts models on smaller hardware

Cerin Amroth builds open-source tools to quantize, fine-tune, serve, and stream fused Mixture-of-Experts models without requiring the model to fit entirely in GPU memory — or, in the NVMe tier, host memory. Start from what does not fit, or from what is too slow.

How the two packages relate

Derived from docs/system-manifest.json, the cross-repository manifest that is byte-identical in both repositories at the pinned commits (provenance); nothing in this section is site-authored.

experts4bit-qlora  (experts4bit-qlora + grouped-nf4-gemm: runtime package)
        |
        |  depends on the kernel package through its optional `fast` extra (>=0.30.0)
        v
grouped-nf4-gemm  (kernel package; never depends on the runtime package)

one system, two independently released packages: the runtime package depends on the kernel package through its optional `fast` extra; the kernel package never depends on the runtime package Dependency direction: experts4bit-qlora -> grouped-nf4-gemm. Current floor: experts4bit-qlora >=0.35.0 → grouped-nf4-gemm >=0.30.0 (extra: fast).

RolePackageOwns
runtimeexperts4bit-qlora
  • model loading (streaming loader, fused expert stacks, Experts4bit/ExpertsNbit bases)
  • quantization orchestration (NF4/MXFP4 routes, int4 expert store, calibration drivers)
  • adapters (ExpertsLoRA)
  • training (trainer, fast/batched/hybrid paths, checkpointing)
  • residency integration (host-RAM and NVMe engines bound to a loaded model)
  • serving (paged runner, decode stack, HTTP shim)
  • model-level measurement, quality gates and the claims register for model-level numbers
kernelsgrouped-nf4-gemm
  • grouped packed GEMM (NF4 and native MXFP4) and its dgrad
  • decode GEMV (int4-b32, MXFP4) and split-K reduce
  • FP8 paged attention and the fp8 KV appends
  • decode glue kernels (rope, norm and residual folds, swiglu, combine)
  • pack and pure-torch reference operations, GPTQ packing, provenance verification
  • host/NVMe primitives (arena, reader, row tier, bake, residency primitives)
  • kernel-level measurement and the claims register for kernel-level numbers

Start with the need

NeedStart with
Quantize fused experts skipped by bitsandbytesexperts4bit-qlora
Fine-tune fused experts with QLoRAexperts4bit-qlora[train]
Serve through the package's serving layerexperts4bit-qlora[serve]
Enable the fused kernel pathexperts4bit-qlora[fast]
Call quantized expert kernels directlygrouped-nf4-gemm
Bake or verify an expert arenagrouped-nf4-gemm primitives
Understand claim methodologyResearch / Receipts-driven engineering

Current status

experts4bit-qlora v0.35.3 · current status → — claims tiered in the public register: 161 measured · 6 measured-private · 3 open · 7 superseded · 6 retired. grouped-nf4-gemm v0.30.2 · current status → — claims tiered in the public register: 11 confirmed · 9 measured · 3 measured-private · 1 projected · 3 open · 1 superseded · 2 retired. Tiers are the registers' own words; nothing here collapses them into one label. The position pages are the repositories' docs/STATUS.md; the site's status page and compatibility page are generated from the same pinned commits.

Evidence vocabulary (from the system manifest)

confirmed
a public receipt plus an independent confirmation cell (kernel register)
verified
a public receipt plus an independent check (runtime register)
measured
the run happened and the receipt is public in the repository
measured-private
the run happened and the receipt lives outside the repository; NOT publicly reproducible from the repository alone
projected
arithmetic from measured inputs, not a run
open
a statement the documents make that has no evidence either way yet; it backs no capability
superseded
replaced by a later claim that names it; kept as the record, never current
retired
withdrawn; kept as the record, never current

Solution pages

ProblemPackage
Why does `load_in_4bit` still OOM on a Mixture-of-Experts model?experts4bit-qlora
How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)?experts4bit-qlora
How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena?experts4bit-qlora
How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)?experts4bit-qlora
How can I run a Mixture-of-Experts model larger than my GPU's VRAM?experts4bit-qlora
How do I serve a large MoE on a consumer GPU (RTX 5090 class)?experts4bit-qlora
How do I run paged decode attention over an FP8 KV cache for a quantized MoE serving path, with sliding windows, attention sinks and a custom scale?grouped-nf4-gemm
How do I run a single-token INT4 decode GEMV over routed MoE experts, and pack calibrated (GPTQ) weights for it?grouped-nf4-gemm
How do I run MoE expert inference natively on MXFP4 (e2m1 + e8m0) weights, straight from the released checkpoint bytes?grouped-nf4-gemm
How do I run a grouped GEMM directly on NF4 packed MoE expert weights, without dequantizing to bf16 first?grouped-nf4-gemm
How do I run a MoE whose experts do not fit in VRAM, streaming them from pinned host RAM or an NVMe arena?grouped-nf4-gemm
How do I verify that the quantized expert bytes I serve or train on are the released checkpoint bytes?grouped-nf4-gemm

Canonical identity and machine-readable data

Historical narratives: Receipts-driven engineering and the experts-4bit narrative are dated research pages indexed at /research/; the project pages above are the current documentation.

CURRENT · source: pjordanandrsn/experts4bit-qlora@0c2a256dcdc2 · rendered package: 0.35.3 · latest published package: 0.35.3

CURRENT · source: pjordanandrsn/grouped-nf4-gemm@e2af4cfb91b2 · rendered package: 0.30.2 · latest published package: 0.30.2