# grouped-nf4-gemm

Triton kernels for 4-bit MoE inference: grouped NF4/MXFP4 GEMM, INT4 GEMV, FP8 paged attention, and CPU/NVMe expert streaming.

Canonical package: `grouped-nf4-gemm` (https://pypi.org/project/grouped-nf4-gemm/). Repository: https://github.com/pjordanandrsn/grouped-nf4-gemm. Import: nf4_grouped, mxfp4_grouped, int4_b32, fp8_paged_attn, fp8_kv, nvme_arena, nvme_reader, nvme_residency, nvme_bake_nf4, mxfp4_loader, gptq_pack, nf4_pack_ref, mxfp4_pack_ref, verify_provenance. Aliases (lookup only, from the system manifest): nf4gemm, gnf4, grouped-mxfp4-gemm.

System manifest: kernels package; one system, two independently released packages: the runtime package depends on the kernel package through its optional `fast` extra; the kernel package never depends on the runtime package Dependency direction: experts4bit-qlora -> grouped-nf4-gemm. Manifest: https://cerinamroth.com/ml/system-manifest.json (provenance https://cerinamroth.com/ml/system-manifest.provenance.json).

Owns: grouped packed GEMM (NF4 and native MXFP4) and its dgrad; decode GEMV (int4-b32, MXFP4) and split-K reduce; FP8 paged attention and the fp8 KV appends; decode glue kernels (rope, norm and residual folds, swiglu, combine); pack and pure-torch reference operations, GPTQ packing, provenance verification; host/NVMe primitives (arena, reader, row tier, bake, residency primitives); kernel-level measurement and the claims register for kernel-level numbers

Claim register: claims tiered in the public register: 11 confirmed · 9 measured · 3 measured-private · 1 projected · 3 open · 1 superseded · 2 retired — https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/claims.json

Current status: v0.30.2 — https://cerinamroth.com/ml/status/#grouped-nf4-gemm

## Direct answer

Use this package when dequantising each active expert into a high-precision weight before its matmul is the bottleneck. It computes the grouped expert GEMM directly over the packed NF4 (bitsandbytes gemm_4bit layout) or native MXFP4 (e2m1 + e8m0) representations, and ships the int4 decode GEMV, the fp8 paged attention, the decode glue and the host/NVMe primitives a 4-bit MoE serving path needs around it. It is a kernel package: experts4bit-qlora drives it.

## Use this when

- A per-expert 4-bit MoE loop pays a launch per active expert per projection, and wherever that loop dequantises (bitsandbytes releases before 0.50.0, cells outside its packed gemm_4bit inference forward, the conventional 4-bit backward) it also writes and re-reads a bf16 copy of each expert; compute the grouped expert GEMM directly on the NF4-packed weights in one launch instead.
- Serve or train a MoE whose experts are released as MXFP4 (gpt-oss, DeepSeek-V4, Kimi lineage) directly from the checkpoint's own bytes, without re-quantising them.
- Single-token (decode) expert and attention projections on 4-bit weights are bandwidth-bound; an int4 GEMV with 32-wide scales and a calibrated packer serves them near the memory ceiling.
- A 4-bit MoE serving path needs a paged decode attention over an fp8 KV cache -- sliding windows, attention sinks, a custom scale, per-layer KV geometry -- computed with fp8 tensor-core dots on the cards that have them.
- The same paged decode attention over the fp8 KV cache computed with f32 dots: the default on sm_80-sm_88 (no fp8 tensor-core dot), the fallback wherever the fp8 constraints fail with GNF4_ATTN_COMPUTE unset, and every explicit f32 request.
- The non-GEMM part of a batch-1 decode step is a tail of small launches (RMSNorm, residual add, rotary, router softmax and top-k, activation, top-k combine, split-K reduce); fold each into one launch.
- The expert weights do not fit in VRAM, or not even in host RAM: stream them from pinned host memory or from an on-disk expert arena on NVMe.
- Prove that the quantised bytes a kernel serves are the checkpoint's released bytes, unchanged, before quoting any result.

## Do not use this when

- dense (non-MoE) GEMMs: use cuBLAS, torch, or bitsandbytes' own 4-bit path -- since 0.50.0 its supported ordinary 2-D inference cells compute from the packed weights directly (torch.ops.bitsandbytes.gemm_4bit); there is no grouped routed-MoE contract upstream, which is what this package is
- a serving engine or a vLLM replacement: this is a kernel package driven by experts4bit-qlora
- operating systems other than Linux for the kernels (triton is Linux-only); ROCm/XPU are port targets, not supported
- models that already fit in bf16 with headroom: the grouped kernel loses at small shapes and to a CUDA-graphed per-expert loop at some decode shapes (claims gnf4.kernel.graphed-baseline-decode-loses)
- silent fallbacks: nothing falls back silently on CPU — nf4_grouped.gemm_4bit_grouped and dgrad_4bit_grouped refuse CPU tensors with an error that names dequant_ref; gemm_mxfp4_grouped and the int4_b32 kernels carry no device guard and fail inside the Triton launch, and the fp8_kv appends and the paged attention refuse with a CUDA+Triton message that names no reference

## Install

```bash
pip install grouped-nf4-gemm
```

Primary route: kernel package.

Alternatives:

- `pip install "experts4bit-qlora[fast]"` — through the model consumer

Environment: OS: Linux · Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested) · Accelerator: NVIDIA CUDA GPU for the serving tiers and for the default NF4 quantise bake (nvme_bake_nf4.bake_nf4 with quantize_fn=None); the relocation bake and verify (nvme_arena.bake / bake_expert_tensors / verify) need no GPU; NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target); NVIDIA CUDA GPU, sm_80 or newer (the composition was measured on the RTX 5090, sm_120); NVIDIA CUDA GPU, sm_80 or newer; this is the path sm_80-sm_88 take by default. Its reference tests were run on an RTX 5090 (sm_120) with the mode forced; no registered cell exists on sm_80-sm_88; NVIDIA CUDA GPU, sm_89 or newer (the fp8 tensor-core dot); measured on the RTX 5090 (sm_120) only -- sm_89 and sm_90 meet the requirement but no registered cell was run there; none: pure torch (a CUDA GPU is optional)

## Problems this solves

- [How do I run paged decode attention over an FP8 KV cache for a quantized MoE serving path, with sliding windows, attention sinks and a custom scale?](https://cerinamroth.com/ml/solutions/fp8-paged-attention-for-moe-serving/): fp8_paged_decode_attention runs paged flash-decode over an E4M3 KV cache with windows, sinks and a custom scale; the fp8 compute path is supported on sm_89+, the f32 path is open under #319.
- [How do I run a single-token INT4 decode GEMV over routed MoE experts, and pack calibrated (GPTQ) weights for it?](https://cerinamroth.com/ml/solutions/int4-decode-gemv/): gemv_int4_b32 runs the batch-1 int4-b32 decode GEMV with exact integer accumulation, and gptq_pack_int4_b32 packs calibrated weights onto the same bytes; its numbers are measured-private.
- [How do I run MoE expert inference natively on MXFP4 (e2m1 + e8m0) weights, straight from the released checkpoint bytes?](https://cerinamroth.com/ml/solutions/native-mxfp4-moe-inference/): gemm_mxfp4_grouped multiplies gpt-oss and Kimi-class experts on their released e2m1 blocks and e8m0 scales, with no requantization to NF4 and no bf16 materialization.
- [How do I run a grouped GEMM directly on NF4 packed MoE expert weights, without dequantizing to bf16 first?](https://cerinamroth.com/ml/solutions/nf4-grouped-gemm-without-bf16-materialization/): gemm_4bit_grouped runs the routed MoE expert GEMM on bitsandbytes NF4 packed weights in one Triton launch, decoding nibbles in registers with fp32 accumulation and no bf16 expert tensor.
- [How do I run a MoE whose experts do not fit in VRAM, streaming them from pinned host RAM or an NVMe arena?](https://cerinamroth.com/ml/solutions/stream-moe-experts-from-host-or-nvme/): The kernel-side storage primitives for experts that do not fit VRAM: the NVMe arena bake and O_DIRECT reader, the pinned-DRAM row tier and low-level residency, and the GPU-driven host gather.
- [How do I verify that the quantized expert bytes I serve or train on are the released checkpoint bytes?](https://cerinamroth.com/ml/solutions/verify-quantized-checkpoint-provenance/): file_tensor_sha256, provenance_table and verify_arena_matches hash safetensors byte ranges against the loaded and baked bytes, and verify_provenance re-derives a run's hash table.

## Evidence

Active claim IDs: gnf4.kernel.fused-more-accurate-than-dequant-bf16, gnf4.kernel.decode-speed-census, gnf4.kernel.graphed-baseline-decode-loses, gnf4.mxfp4.serve-tax-deleted, gnf4.flagship.235b-phaseA, gnf4.nvme.tier-batch-only — see https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/claims.json (pinned; latest on main, unpinned; not the source of any fact rendered here: https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/main/docs/claims.json).

## Freshness

CURRENT · source: pjordanandrsn/grouped-nf4-gemm@e2af4cfb91b2 · rendered package: 0.30.2 · latest published package: 0.30.2

## Provenance

Source commit: e2af4cfb91b2c29968989c5254bff55aa7da3347 (2026-09-05); repository version 0.30.2; latest PyPI 0.30.2 (published); synced 2026-09-06T00:07:26Z. Pinned: https://github.com/pjordanandrsn/grouped-nf4-gemm/tree/e2af4cfb91b2c29968989c5254bff55aa7da3347 · capabilities https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/capabilities.json · status https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/STATUS.md · claims https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/claims.json · system manifest https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/system-manifest.json. Latest on main (unpinned; not the source of any fact rendered here): https://github.com/pjordanandrsn/grouped-nf4-gemm.