grouped-nf4-gemm

Use this when

• A per-expert 4-bit MoE loop pays a launch per active expert per projection, and wherever that loop dequantises (bitsandbytes releases before 0.50.0, cells outside its packed gemm_4bit inference forward, the conventional 4-bit backward) it also writes and re-reads a bf16 copy of each expert; compute the grouped expert GEMM directly on the NF4-packed weights in one launch instead. — gemm_4bit_grouped runs one Triton launch for all active experts over the bitsandbytes gemm_4bit NF4 layout (LUT decode to fp32 in registers, blockwise absmax scaling, fp32 accumulation, bf16 epilogue); dequant_ref is the pure-torch oracle it is asserted against; dgrad_4bit_grouped is the one-launch training backward; gemm_nf4_grouped_smallm (K25; experts4bit-qlora's default for NF4 decode-batch rows since its P96, e4b.serve.p96.nf4-families.k25-windowed-k8.5090.2026-10-02) is K19's grouped tensor-core small-M GEMM on the same NF4 store, its codebook decoded by an exact select tree (or, with lut="prmt", K39's byte-permute decode, bit-identical; experts4bit-qlora's default on qwen3_moe since its P158, e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20) that lane K26 measured faster than the served NF4 GEMM (gnf4.kernel.k26-nf4-decode-ablation.5090.2026-10-01) for decode batches, its weight operand exactly dequant_ref rounded to bf16 (not the served kernel's TF32), so a consumer gates it on quality. On sm_90 the default training route (GNF4_TRAIN_GEMM=auto; nf4_route) dequantizes the present experts to bf16 (bit-equal to dequant_ref rounded to bf16) and runs the training forward and dgrad through torch._grouped_mm: experts4bit-qlora's TC1c amendment 6 measured the full Qwen3-30B-A3B training step on an H100 NVL at Unsloth/e4b 1.030 with it against 0.817 with the fused kernels, the matched set EQUIVALENT though not bit-identical (<= 0.0024 relative Frobenius per GEMM in the kernel replay). GNF4_TRAIN_GEMM=fused keeps the fused kernels, which every other card uses by default. Off sm_90, auto takes the dense route (each present expert as its own dequant + torch.mm, one expert's bf16 weight at a time) for a call with at most DENSE_AUTO_MAX_GROUPS present groups, and the fused kernels above that: experts4bit-qlora's TC1 amendment 22 read the dense route faster on a few-large-expert family (Mixtral-8x7B) and far slower on a many-expert one (Qwen3-30B-A3B, launch-bound). Not bit-identical to the fused kernels; GNF4_TRAIN_GEMM=fused restores them. solution page
• Serve or train a MoE whose experts are released as MXFP4 (gpt-oss, DeepSeek-V4, Kimi lineage) directly from the checkpoint's own bytes, without re-quantising them. — gemm_mxfp4_grouped is the table-swap of the NF4 kernel over native blocks [E, N, K//2] + e8m0 scales [E, N, K//32]; gemv_mxfp4_b32 is its decode-grade GEMV; gemm_mxfp4_grouped_smallm (K21; experts4bit-qlora's default for the native MXFP4 store's decode-batch rows since its P90, e4b.serve.p90.gptoss.mxfp4.k21-b16.5090.2026-10-01) is K19's grouped tensor-core small-M GEMM on the same store, for decode batches above the GEMV's rows; mxfp4_loader.to_kernel_shapes views the released tensors in kernel layout with no copy; mxfp4_qlora and mxfp4_native_load train LoRA over the frozen native stacks. solution page
• Single-token (decode) expert and attention projections on 4-bit weights are bandwidth-bound; an int4 GEMV with 32-wide scales and a calibrated packer serves them near the memory ceiling. — quant_x_rows quantises activation rows to int8 with per-32 scales; gemv_int4_b32 runs the split-K int4-b32 GEMV over packed [E, N, K//2] + fp16 scales [E, N, K//32] (reduce_partials folds the split-K sum and bf16 cast into one launch); pack_int4_b32 is the round-to-nearest packer and gptq_pack_int4_b32 + HessianAccumulator the calibrated one; gemm_int4_b32_grouped_captured is the capture-legal batched variant. gemv_int4_b32_grouped (K18) is the split-K GEMV with each program serving up to four rows of one expert, bitwise the served GEMV and measured slower (dormant). gemm_int4_b32_grouped_smallm (K19) runs K16's bf16 tensor-core arithmetic over the device tile table (build_group_tiles_fused, 16-row tiles) with the first projection's gather folded in, bit-identical to K16 per expert and across plans; its default plan (BLOCK_N 32, KC 256) is K20's best on the RTX 5090 (gnf4.kernel.k20-k19-plan-sweep.5090.2026-10-01); experts4bit-qlora's default for int4 decode rows above one since its P88 (e4b.serve.p88.qwen3.int4.k19-b16.5090.2026-10-01). solution page
• A 4-bit MoE serving path needs a paged decode attention over an fp8 KV cache -- sliding windows, attention sinks, a custom scale, per-layer KV geometry -- computed with fp8 tensor-core dots on the cards that have them. — fp8_kv quantises and appends KV blocks in e4m3 with per-(token, head) or grouped scales (the fused appends fp8_kv_append_t1 / fp8_kv_append_bt1 write quantize_kv_fp8's bytes exactly: one per-group helper, _e4m3_group, with an IEEE-rounded quotient, gated in fp32 on any CUDA card by test_group_math_is_the_reference_fp32_math -- e4b#771); fp8_paged_decode_attention is the paged decode kernel with sliding windows, sinks, custom scale and per-layer KV geometry, asserted against paged_attn_ref. With GNF4_ATTN_COMPUTE unset it selects the fp8 compute path wherever fp8_compute_unsupported returns None, an explicit request is never downgraded, and compute_counts records which path ran. solution page
• The same paged decode attention over the fp8 KV cache computed with f32 dots: the default on sm_80-sm_88 (no fp8 tensor-core dot), the fallback wherever the fp8 constraints fail with GNF4_ATTN_COMPUTE unset, and every explicit f32 request. — fp8_paged_decode_attention with compute='f32' (or selected by the capability-conditional default) runs the split kernel, or the packed kernel with pack_heads=True, dequantising e4m3 in registers and accumulating in f32; it imports and launches on any sm_80+ card and is asserted against the same paged_attn_ref. solution page
• The non-GEMM part of a batch-1 decode step is a tail of small launches (RMSNorm, residual add, rotary, router softmax and top-k, activation, top-k combine, split-K reduce); fold each into one launch. — int4_b32 carries the decode glue: rmsnorm_rows, rmsnorm_resid_rows, scaled_resid_add_rows, rope_norm_heads, rope_norm_qk (the query's and the key's in one launch), rope_heads, router_epilogue (softmax-then-top-k, or top-k on logits with an optional bias), swiglu_rows, combine_rows and reduce_partials; each has an in-tree parity test against a torch reference. On sm_90+ NVIDIA cards these kernels, the int4 decode GEMV and the activation quantise launch with programmatic dependent launch by default, for launches of at most 8 activation rows (GNF4_PDL, GNF4_PDL_MAX_ROWS; GNF4_PDL=0 turns it off): each waits for its predecessor before touching memory, so outputs are bitwise those with it off (kernel/test_pdl.py). solution page
• The expert weights do not fit in VRAM, or not even in host RAM: stream them from pinned host memory or from an on-disk expert arena on NVMe. — nvme_arena.bake / bake_expert_tensors relocate a checkpoint's expert tensors into a page-aligned arena with an index and manifest; nvme_bake_nf4.bake_nf4 quantises at bake; nvme_reader.ArenaReader reads it with O_DIRECT queues; nvme_residency.ColdTier and row_pool.RowPool hold the hot rows in pinned DRAM; arena_experts.ArenaExpertSource fetches per-layer fused stacks; mxfp4_residency serves native MXFP4 arenas. solution page
• Prove that the quantised bytes a kernel serves are the checkpoint's released bytes, unchanged, before quoting any result. — mxfp4_loader.file_tensor_sha256 hashes a safetensors tensor's data-section bytes without loading or dequantising; provenance_table builds the per-tensor hash table for a snapshot; verify_arena_matches raises on any mismatch (a provenance failure is not a tolerance); nvme_arena.verify checks an arena against its source; python -m verify_provenance is the CLI. solution page

• Loses to a CUDA-graphed per-expert baseline at some decode shapes (claim gnf4.kernel.graphed-baseline-decode-loses); shapes below the dispatch floor belong to the dequant path: nf4_grouped.decode_dispatch says so and the caller routes them there, while the op itself always runs fused (docs/STATUS.md)
• CUDA + Triton only; calling the kernel on CPU raises and names dequant_ref
• The grouped_mm training route needs compute capability 9.0 (torch 2.8's torch._grouped_mm target): GNF4_TRAIN_GEMM=auto takes it only there, and GNF4_TRAIN_GEMM=grouped_mm refuses elsewhere with the reason; it is not bit-identical to the fused kernels.
• The decoded training route (GNF4_TRAIN_GEMM=decoded: dequant_groups + one grouped bf16 GEMM launch per chunk of groups, the chunk's decode transient capped by GNF4_DECODED_MAX_BYTES, default 256 MiB) is opt-in only: GNF4_TRAIN_GEMM=auto never takes it. It is not bit-identical to the fused kernels or the dense route, and no speed is claimed for it. experts4bit-qlora's TC1 amendment 46 (one RTX 5090, experts4bit-qlora#1221) read the training step at decoded/fused 1.005 [0.979, 1.031] on OLMoE-1B-7B and 1.066 on Qwen3-30B-A3B, so auto does not take it.
• The fidelity claim is measured against fp32 accumulation on NVIDIA tensor cores; other backends must re-measure (docs/PORTABILITY.md)
• gemm_nf4_grouped_smallm's lut="prmt" (lane K39) is an opt-in byte-permute codebook decode in inline PTX, bit-identical to lut="tree" on NVIDIA targets (the same codebook values and the same operand layout); it is refused on ROCm, decodes through the tree under the interpreter, and this package claims no speed for it. experts4bit-qlora serves it by default on qwen3_moe since 0.56.0 (its served read P158, e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20)
• GNF4_GEMV_BW routes single-row decode calls through the bandwidth-targeted GEMV (lane K33). Its default, auto, engages at the expert shapes in _BW_SHAPES on >= 160-SM parts -- Qwen3-30B-A3B's (experts4bit-qlora's served lane P116, DEFAULT_ON: one request faster, 16 requests unchanged, within P110's teacher-forced quality bar), Granite-3.1-3b-a800m's and OLMoE-1B-7B's (P134, e4b.serve.p134.gemv-bw.granite.5090.2026-10-18, e4b.serve.p134.gemv-bw.olmoe.5090.2026-10-18) and Mixtral-8x7B's (P141, e4b.serve.p141.gemv-bw.mixtral.5090.2026-10-18) -- and at Qwen3-30B-A3B's on 128-SM sm_89 parts, the RTX 4090 (P133, e4b.serve.p133.gemv-bw.qwen3.4090.2026-10-18); every other shape and part keeps the scalar GEMV, and dot-pad runs only with GNF4_GEMV_BW=0. GNF4_GEMV_BW=1 forces it everywhere and 0 turns it off. It runs K33's selected plan per shape (_BW_PLANS; GNF4_GEMV_BW_PLAN overrides) with GNF4_PDL at its default. K33 read LEVER at kernel level on an RTX 5090 (claim gnf4.kernel.k33-nf4-decode-gemv-bw.5090.2026-10-07). Its codebook decode is exact fp32 either way: the PTX byte-permute table (prmt32) needs a compiled NVIDIA target, and the interpreter and non-PTX backends take an exact select tree (GNF4_GEMV_BW_DECODE=tree). It is not bit-identical to the scalar or dot-pad routes (a different reduction tree; the tolerance contract applies).
• Every speed figure is per card -- the census is sm_86, the head-to-head and the real-finetune cells are RTX 4090 / H100, the torch._grouped_mm cell is RTX 5090 -- and none is an architecture-wide statement; the registered comparator is this repository's own per-expert dequantize-then-GEMM loop, and no registered cell times bitsandbytes' own packed 2-D inference forward (0.50.0 and later)

• A uniform int4 grid cannot represent e2m1 levels: do not quantise-bake an already-MXFP4 checkpoint to NF4 or int4 (docs/solutions/native-mxfp4-moe-inference.md)
• gemv_mxfp4_b32 re-streams weights per row: it is a decode lever for a handful of rows; batched rows use the grouped GEMM or the consumer's NF4 path
• No ROCm/XPU
• The claims here are fidelity and provenance results from single runs per card (a rented H200 pod for serving, an L40S and an A2000 for training, the K3 oracle on CPU); the MXFP4 GEMM has no speed census of its own and gemv_mxfp4_b32 is inside no registered claim

• The int4-b32 GEMV and the calibrated packer are measured-private (claims gnf4.serve.int4-b32-gemv, gnf4.serve.gptq-pack-int4-b32): the receipts live in the consumer's private audit tree
• Split-K on the NF4 dot-pad decode GEMV (nf4_grouped, GNF4_GEMV_SPLITK, lane K7) was refuted once and ships dormant there (the retired claim gnf4.retired.splitk-gemv records the refutation); the int4-b32 GEMV's own split-K is the shipped, measured plan (int4_b32._plan, reduced by reduce_partials)
• Split-K on the int4-b32 GEMV is planned from (N, K) alone on every part: the row-count term (R >= 16, gated by SPLITK_R_TERM_MAX_SMS) is off since K30 read it OFF on a rented NVIDIA L4 (claim gnf4.kernel.k30-splitk-r-term.l4.2026-10-05), so the plan no longer depends on the SM count and two cards plan the same sk for the same call
• Whether int4 experts pass a quality gate is the consumer's per-family question (experts4bit-qlora docs/STATUS.md): small-active-parameter models fail it with round-to-nearest packing
• The single-stream decode anchor (claim gnf4.serve.decode-anchor-5090) is a consumer-measured anchor of the serving class on a knob-off basis, not a measurement of this lane, and is not cited here as its evidence
• K16: int4_smallm.gemm_int4_b32_smallm is a small-M single-projection GEMM -- in-register int4 dequant, per-block scaling inside the tile, bf16 tensor-core MMA over fat K chunks, split-K fused into the launch (kernel/PREREG-k16-smallm-int4-gemm.md). Measured on the RTX 5090 class at M=16 (claim gnf4.kernel.k16-smallm-int4-gemm.5090.2026-09-19): it beats this package's bf16 dequant path on every attention shape and meets the pre-registered Marlin bar; the single-scale control (P4) was not tested; P5 holds in the consumer (claim gnf4.kernel.k16-smallm-int4-gemm.5090.2026-09-19), and the consumer's route is its default (auto) from experts4bit-qlora 0.36.2
• K18: int4_b32.gemv_int4_b32_grouped tiles the call's rows expert-major in-register (histogram, cumsum, per-expert rank -- no sort, no host sync, capture-legal) so each program loads one expert's weight slice once for up to GROUPED_MT_DEFAULT=4 of its rows; it is torch.equal to gemv_int4_b32(..., fused_reduce=False). Measured on the RTX 5090 (claim gnf4.kernel.k18-grouped-expert-gemv.5090.2026-09-22): exact, and SLOWER than gemv_int4_b32 both on the recorded Qwen3-30B-A3B B=16 routing and at the small row counts, so it is not a lever and ships dormant as the evidence; use gemv_int4_b32. Nothing in this package or the consumer routes to it. The wrapper refuses E > GROUPED_E_MAX=4095 (the tiling's shared-memory histogram)
• build_group_tiles_fused(..., rank="cumsum", programs=P) (e4b#846, opt-in) splits the cumsum tile table over P programs (_tile_table_cumsum_mp): each takes every expert's count from one tl.histogram of the ids and ranks and writes only its own experts' rows, so no row has two writers; programs=1 (the default) is the one-program kernel, launched as before. The same integers at every P by test (kernel/test_tile_table_programs_interp.py: E 40/128/256, R 1-1024, P 2/3/8/64, interpreter and compiled on an RTX A2000). Unread for speed: experts4bit-qlora's lane reads it before any default moves.

• One card class measured: both claims were run on the RTX 5090 (sm_120). The sm_89+ requirement is the kernel's precondition, not a measured statement about Ada or Hopper parts
• Adds one e4m3 rounding on q and one on p, so its test bound is wider than the f32 path's (kernel/test_fp8_paged_attn.py)
• pack_heads=True falls back to the split fp8 kernel with a one-time RuntimeWarning where the packed tile exceeds the card's shared memory
• Served-path parity (Granite, gpt-oss, Gemma-4) is measured in experts4bit-qlora's private receipts; the evidence here is kernel-level parity against paged_attn_ref
• KV budgets in docs/context-budgets.md are rung-one (A2000) numbers; pending rows must not be promoted

• #319 is closed: its failures were the test's shape arms running the fp8 kernel on sm_89+ while judged at the f32 tolerance, never a defect in these kernels (claim gnf4.serve.f32-arms-ran-fp8); the earlier claim gnf4.open.f32-compute-modes-triton34 is retired
• Measured on two cards (sm_86 and sm_120) on one torch/Triton pair; other pairs are unmeasured. The old advice to gate lanes with -k "f8dot or pf8" excluded exactly the mislabelled arms and should not be followed (docs/STATUS.md)
• Explicit GNF4_ATTN_COMPUTE=f32 or compute='f32' selects this path on any card, sm_89+ included: the explicit request is honoured, never downgraded to fp8

• The composition timing and the paired perplexity deltas are measured-private (claim gnf4.serve.decode-glue-kernels): real runs through the consumer on the RTX 5090 whose receipts live in a private audit tree; what is checkable here is each kernel's parity test
• int4_b32 imports triton at module level and is not importable without it
• combine_rows and reduce_partials carry an accuracy contract, not a bitwise one: each is within the error bound of a correct fp32 summation in every census case on an RTX 5090, and neither is bitwise the torch chain it replaced (lane B393; gnf4.kernel.combine-rows-accuracy.5090.2026-09-23, gnf4.kernel.reduce-partials-slot-order.5090.2026-09-23); the kernel's output and torch's chain were each bit-identical on sm_86 and sm_120 at the census cases (corrected 2026-09-24)
• GNF4_PDL is on by default, capped at 8 activation rows, since experts4bit-qlora lane P113 read CAP_DEFAULT on SC1's int4 serving configuration (e4b.serve.p113.gnf4-pdl-capped.qwen3-int4.5090.2026-10-04; uncapped it cost the 16-request step there, which is why the cap is the default); lane K28 read the mechanism on the decode chain (gnf4.kernel.k28-pdl-decode-chain.5090.2026-10-04); it is inert on CPU, under the interpreter, on ROCm and below sm_90, and its speed elsewhere is unread
• dense_gemv.gemv_bf16_bw (lane K35) is experts4bit-qlora's P148 prototype kernel verbatim behind a frozen per-shape plan (any other shape runs the default plan, untimed): one activation row only (a multi-row call keeps cuBLAS), and it routes no model call by itself. Its tolerance contract against cuBLAS is checked on the card (kernel/test_dense_gemv.py); no speed is claimed here, because the consumer's served read is the timing of record
• dense_gemv.gemv_i8_bw (lane K36) is experts4bit-qlora's P148 int8 prototype kernel verbatim, with its quantizer quantize_rows_int8 (symmetric per-row absmax, s = absmax / 127) verbatim, behind a frozen plan at Qwen3-30B-A3B's int8 lm_head shape (any other shape runs the default plan, untimed): one activation row only, int8 weights with an fp32 per-row scale, and it routes no model call by itself. Its tolerance contract against cuBLAS on the bf16 head is checked on the card (kernel/test_dense_gemv_i8.py); the int8 grid's effect on a model's quality is the consumer's ruling (experts4bit-qlora P150), and no speed is claimed here: the consumer's served read is the timing of record
• dense_i8.linear_i8 (lane K37) is experts4bit-qlora's P149 prototype int8 kernels verbatim (a one-row GEMV and a 2-to-16-row small-M kernel) over K36's quantize_rows_int8 grid, re-exported, behind plans frozen at Mixtral-8x7B's attention projection shapes (any other shape runs the default plans, untimed): 1 to 16 activation rows only, int8 weights with an fp32 per-row scale, and it routes no model call by itself. Its tolerance contract against cuBLAS on the bf16 weights is checked on the card (kernel/test_dense_i8.py); the int8 grid's effect on a model's quality is the consumer's ruling (experts4bit-qlora P147), and its timing is the consumer's served read.

• NVMe is a batch tier: per-token latency is bounded by the device link (claim gnf4.nvme.tier-batch-only)
• Expert prefetch closed negative over four arcs (claim gnf4.flagship.prefetch-closed-negative)
• The cold-engine 'free floor' premise was refuted (claim gnf4.cold-engine.phase0-premise-refuted); read docs/INDEX.md before quoting any cold-engine number
• This entry is the primitive layer (arena, reader, row tier, low-level residency); which bytes live where and the end-to-end run of a MoE larger than VRAM are experts4bit-qlora's pages run-moe-larger-than-vram (decision/router) and offload-moe-experts-to-cpu-or-nvme (model-level integration)

• A quantize-at-bake arena (nvme_bake_nf4) is a documented two-hop transform: its manifest's bake_mode says it is bit-identical to the quantiser's output, not to a bf16 release
• Making the verifier pass on non-matching bytes is the highest-severity class in SECURITY.md; report privately

Also discoverable as nf4gemm, gnf4, grouped-mxfp4-gemm. These are lookup-only aliases from the system manifest; grouped-nf4-gemm is the canonical project.

What it supports

Modes

benchmark, calibration, inference, offload, packing, serving, streaming, training, verification

Formats

bf16, fp8, int4, mxfp4, nf4

Model families

DeepSeek-V4 lineage; Kimi K3 lineage (reference decode); Qwen3-235B-A22B (host-RAM flagship); any checkpoint whose experts the bake can enumerate; any expert or projection weight packed with pack_int4_b32 / gptq_pack_int4_b32 (driven by experts4bit-qlora's serve lanes); any fused-expert MoE whose experts are packed in the bitsandbytes NF4 gemm_4bit layout (Qwen3-MoE, OLMoE, Mixtral, GraniteMoe, Gemma-4 text tower via experts4bit-qlora); any safetensors checkpoint; MXFP4 expert tensors for gpt-oss / DeepSeek-V4 / Kimi lineage; driven by experts4bit-qlora's paged engine for Qwen3-MoE, OLMoE, GraniteMoe, gpt-oss, Gemma-4, Mixtral; gpt-oss (20b, 120b); gpt-oss / DeepSeek-V4 / Kimi lineage MXFP4 arenas

Operating system

Linux

Python

>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

Accelerator

NVIDIA CUDA GPU for the serving tiers and for the default NF4 quantise bake (nvme_bake_nf4.bake_nf4 with quantize_fn=None); the relocation bake and verify (nvme_arena.bake / bake_expert_tensors / verify) need no GPU; NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target); NVIDIA CUDA GPU, sm_80 or newer (the composition was measured on the RTX 5090, sm_120); NVIDIA CUDA GPU, sm_80 or newer; this is the path sm_80-sm_88 take by default. Reference tests measured on an RTX A2000 (sm_86) and an RTX 5090 (sm_120), torch 2.8.0+cu128 / triton 3.4.0, with the arms pinned to the compute mode they name: identical errors on both, every one a bf16 output ULP, none over tolerance (gnf4.serve.f32-arms-ran-fp8); NVIDIA CUDA GPU, sm_89 or newer (the fp8 tensor-core dot); measured on the RTX 5090 (sm_120) only -- sm_89 and sm_90 meet the requirement but no registered cell was run there; none: pure torch (a CUDA GPU is optional)

Public entry points

arena_experts:ArenaExpertSource, cli:python -m nvme_arena, cli:python -m nvme_bake_nf4, cli:python -m verify_provenance, dense_gemv:gemv_bf16_bw, dense_gemv:gemv_i8_bw, dense_gemv:quantize_rows_int8, dense_i8:linear_i8, flag:GNF4_ATTN_COMPUTE, flag:GNF4_GEMV_BW, fp8_kv:fp8_kv_append_bt1, fp8_kv:fp8_kv_append_t1, fp8_kv:quantize_kv_fp8, fp8_paged_attn:compute_counts, fp8_paged_attn:fp8_compute_unsupported, fp8_paged_attn:fp8_paged_decode_attention, fp8_paged_attn:paged_attn_available, fp8_paged_attn:paged_attn_ref, gptq_pack:HessianAccumulator, gptq_pack:gptq_pack_int4_b32, int4_b32:combine_rows, int4_b32:gemm_int4_b32_grouped_captured, int4_b32:gemv_int4_b32, int4_b32:gemv_int4_b32_grouped, int4_b32:quant_x_rows, int4_b32:reduce_partials, int4_b32:rmsnorm_resid_rows, int4_b32:rmsnorm_rows, int4_b32:rope_heads, int4_b32:rope_norm_heads, int4_b32:rope_norm_qk, int4_b32:router_epilogue, int4_b32:scaled_resid_add_rows, int4_b32:swiglu_rows, int4_pack_ref:dequant_int4_ref, int4_pack_ref:pack_int4_b32, int4_smallm:gemm_int4_b32_grouped_smallm, int4_smallm:gemm_int4_b32_smallm, int4_smallm:plan_smallm, int4_smallm:smallm_workspace, mxfp4_grouped:gemm_mxfp4_grouped, mxfp4_grouped:gemm_mxfp4_grouped_smallm, mxfp4_grouped:gemv_mxfp4_b32, mxfp4_loader:file_tensor_sha256, mxfp4_loader:provenance_table, mxfp4_loader:to_kernel_shapes, mxfp4_loader:verify_arena_matches, mxfp4_native_load:build_native_qlora_model, mxfp4_pack_ref:dequant_mxfp4, mxfp4_pack_ref:quantize_pack_mxfp4, mxfp4_qlora, mxfp4_residency:Mxfp4NvmeResidency, nf4_grouped:build_group_tiles_device, nf4_grouped:dequant_ref, nf4_grouped:dgrad_4bit_grouped, nf4_grouped:gemm_4bit_grouped, nf4_grouped:gemm_4bit_grouped_captured, nf4_grouped:repack_from_bnb, nf4_pack_ref:quantize_pack_nf4, nf4_qlora:fused_grouped_lora, nf4_qlora:gemm_4bit_grouped_train, nf4_route:decoded_dgrad, nf4_route:decoded_forward, nf4_route:dense_dgrad, nf4_route:dense_forward, nf4_route:dequant_groups, nf4_route:grouped_mm_dgrad, nf4_route:grouped_mm_forward, nf4_route:route_for, nf4_smallm:gemm_nf4_grouped_smallm, nf4_smallm:pair_lut, nvme_arena:bake, nvme_arena:bake_expert_tensors, nvme_arena:verify, nvme_bake_nf4:bake_nf4, nvme_reader:ArenaReader, nvme_reader:alloc_landing, nvme_residency:ColdTier, nvme_residency:capacity_for_bytes, row_pool:RowPool, verify_provenance:main

Owns (system manifest)

• grouped packed GEMM (NF4 and native MXFP4) and its dgrad
• decode GEMV (int4-b32, MXFP4) and split-K reduce
• FP8 paged attention and the fp8 KV appends
• decode glue kernels (rope, norm and residual folds, swiglu, combine)
• pack and pure-torch reference operations, GPTQ packing, provenance verification
• host/NVMe primitives (arena, reader, row tier, bake, residency primitives)
• kernel-level measurement and the claims register for kernel-level numbers

Related package

experts4bit-qlora — one system, two independently released packages: the runtime package depends on the kernel package through its optional `fast` extra; the kernel package never depends on the runtime package. Dependency direction: experts4bit-qlora -> grouped-nf4-gemm.

Canonical links

• GitHub: https://github.com/pjordanandrsn/grouped-nf4-gemm · issues · releases
• PyPI: https://pypi.org/project/grouped-nf4-gemm/
• Pinned at 30adb1bfa76f (the source of every fact on this page): docs/capabilities.json · docs/STATUS.md · docs/claims.json · docs/SOLUTIONS.md · docs/system-manifest.json
• Latest on main (unpinned; not the source of any fact rendered here), beside each pinned document above: docs/capabilities.json · docs/STATUS.md · docs/claims.json · docs/SOLUTIONS.md · pinned counterparts: docs/claims.json@30adb1bfa76f
• Machine-readable: this page as JSON · as Markdown · packages.json · system-manifest.json · system-manifest.provenance.json · /ml/llms.txt