Compatibility

Check the package versions together, then choose a capability to see its supported models, hardware requirements and limits. Each status applies only within those conditions; it does not promise support for every model or configuration.

Statuses and conditions come from the pinned docs/capabilities.json registers. A capability's status is the register's own word for the support state of the capability itself; the claim tiers its conditions cite are defined in the evidence vocabulary. A family or platform not listed in a capability's conditions is not supported.

Package compatibility

From the compatibility records of docs/system-manifest.json. The current record is the one whose consumer_versions range contains the rendered experts4bit-qlora version (0.58.1); its floor is validated against pyproject.toml at the pin, never trusted from the manifest, and the rendered grouped-nf4-gemm (0.51.0) satisfies it.

experts4bit-qlora >=0.55.0 → grouped-nf4-gemm >=0.43.0 (extra: fast)

Why: the bandwidth-targeted NF4 decode GEMV is the default from 0.43.0 (GNF4_GEMV_BW=auto, grouped-nf4-gemm#509): experts4bit-qlora P116's registered consequence floors the consumer's fast extra on that release (experts4bit-qlora bench/p116/RESULTS-p116.md:93-96)

Consumer CI pin: the consumer's CI installs the kernel package from the git commit of the v0.47.1 release · since 2026-10-17.

Historical records

Earlier floors, kept as the record; not current.

experts4bit-qlora >=0.35.0,<0.55.0 → grouped-nf4-gemm >=0.30.0 (extra: fast) (since 2026-09-04) — combine_rows (grouped-nf4-gemm#335) with the swiglu_rows / reduce_partials decode glue that experts4bit-qlora#385 integrates, and gptq_pack for the calibrated int4 expert store (experts4bit-qlora#384)
experts4bit-qlora 0.34.x → grouped-nf4-gemm >=0.28.0 (extra: fast) (since 2026-09-04) — rope_heads (the rotary-only fold for attention without a head norm, grouped-nf4-gemm#330) and gemv_mxfp4_b32 (the decode-grade MXFP4 GEMV)

`experts4bit-qlora` — docs/capabilities.json at 605d21e67958

Capability	Task / format	Status	Scope
quantize-fused-moe-experts experts4bit-qlora	quantization, inference, training nf4, fp4, int8, fp8, bf16, fp16	supported	Models, requirements and limits ↓
qlora-fused-moe-experts experts4bit-qlora	training nf4	supported	Models, requirements and limits ↓
run-moe-larger-than-vram experts4bit-qlora	offload, inference, training, serving nf4	supported	Models, requirements and limits ↓
offload-moe-experts-to-cpu-or-nvme experts4bit-qlora	offload, serving, training, streaming nf4, mxfp4	supported	Models, requirements and limits ↓
serve-moe-on-consumer-gpu experts4bit-qlora	serving, inference nf4, int4, fp8	supported	Models, requirements and limits ↓
mxfp4-moe-training-and-residency experts4bit-qlora	inference, training, offload, serving mxfp4, nf4	experimental	Models, requirements and limits ↓

quantize-fused-moe-experts · models, requirements and limits

Model families: olmoe; qwen3_moe; qwen3_5_moe; gemma4_text; granitemoe; gpt_oss; deepseek_v4 (Flash, Pro); qwen3_next; deepseek_v2; mixtral
OS / accelerator: Linux · NVIDIA CUDA GPU (bitsandbytes 4-bit)
Python: >=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested)

Unsupported model_type raises NotImplementedError; identity-expert families (LongCat-Flash) are refused by name
4-bit is a memory trade when the model already fits, and cost energy on the measured comparator (one rented card, one projection, medians of three passes): claim e4b.train.energy-honest.5090.2026-10-05
Dense (non-MoE) models are not the target; use bitsandbytes directly

qlora-fused-moe-experts · models, requirements and limits

Model families: olmoe; qwen3_moe; gemma4_text; mixtral; granitemoe; lfm2_moe; granitemoehybrid; ernie4_5_moe; nemotron_h; qwen3_5_moe
OS / accelerator: Linux · NVIDIA CUDA GPU (bitsandbytes 4-bit)
Python: >=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested)

enable_fast_train returns 0 (does not raise) when the kernel package is absent: assert the return value
dgrad=True needs grouped-nf4-gemm>=0.7.0 (below the current [fast] floor in pyproject.toml, so any [fast] install has it); on older cuts it is downgraded with a RuntimeWarning
Loss parity is measured on the registered criteria in docs/METHODOLOGY.md, not asserted for every dataset
model_families is evidence-gated: a family is listed when its fast_train path -- the headline path, enable_fast_train -- PASSES the registered loss-parity criteria on real weights with a receipt in this repository (training_support is the per-path record; bench/train-parity-20260905/tp1/ is lane tp1, all six serving families). olmoe is confirmed on real weights and a registered text; qwen3_moe and gemma4_text carry tp1 PASSes beside their flagship matrices; mixtral entered on its tp1 fused PASS under offload; granitemoe entered on its corrected-counter re-run's PASS (its first attempt was a harness error, kept as its own row); gpt_oss is refused by every training enabler and stays out
gpt_oss is not a training family here: the loader builds its experts bare (no ExpertsLoRA) by default, and both enable_fast_train and enable_batched_train return zero on it -- REFUSED rows on tp1 (e4b.train.parity.tp1.gptoss.fused.2026-09-05, e4b.train.parity.tp1.gptoss.batched.2026-09-05). Attention-only QLoRA over its frozen experts trains (e4b.train.parity.tp1.gptoss.attn_only.2026-09-05); its experts train only on unlicensed routes: ExpertsLoRA's unfused reference loop behind E4B_GPTOSS_EXPERT_LORA=1 (CPU-graded against a transformers-matching oracle, unmeasured on hardware; enable_fast_train and enable_batched_train refuse that wrapper), and grouped-nf4-gemm's experimental mxfp4_qlora.ExpertsMxfp4LoRA under the mxfp4-moe-training-and-residency capability -- experimental, never licensed
enable_batched_train falls back to the reference forward per call above _PAD_WASTE_LIMIT (engines/batched.py): the patch count stays positive while some or all layers run the per-expert loop on some steps. In the shipped code tp1 measured this fallback was uncounted, and three arms read VOID on exactly that -- OLMoE, Qwen3 and Gemma-4 (e4b.train.parity.tp1.olmoe.batched.2026-09-05, e4b.train.parity.tp1.qwen3.batched.2026-09-05, e4b.train.parity.tp1.gemma4.batched.2026-09-05); it engaged everywhere only on Mixtral's eight-expert and Granite's forty-expert shapes. As of the release carrying that bundle the fallback is countable: batched_fallback_stats(model) reports it (#402; those VOID rows are the reason it exists). Assert kernel engagement, not the return value alone
enable_fast, enable_fast_train and enable_batched_train refuse a wrapper whose base violates the stock-epilogue contract (EpilogueContractError, #402) instead of silently computing act(gate)*up over a base that applies biases, clamps or another gating rule -- a refusal is loud, a wrong epilogue is not; gpt-oss is the family this protects, and DeepSeek-V4 supplies its epilogue through _apply_gate and is admitted
Head-to-head against Unsloth's 4-bit MoE QLoRA path on one identical training problem (lane p38, 2026-09-05, one rented RTX 5090, Qwen3-30B-A3B, the clinical fixture tokenised once, seq 512, r 8 / alpha 16, identical targets, optimizer, batch, steps, precision and eval; bench/h2h-20260905/p38/): the shipped fused dgrad path with NF4 attention is faster per step, lower in peak VRAM and lower in energy per step than Unsloth on that box, and the two are comparable on held-out loss at sixty steps (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05, ...quality-n60) -- BUT the two-hundred-step curve favours Unsloth: its held-out loss is lower at the end of the curve while e4b's flattens from step sixty (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.curve-n200). Quote the position only beside that row. Candidate causes (the eval schedule, the checkpointing mode, the transformers/peft versions of the two stacks, the expert adapter's precision -- bf16 on this side, the loader passes the model dtype to ExpertsLoRA -- against fp32 on Unsloth's) are not established. One workload, one box, one family; nothing here licenses a path (tp1 does) and no general speed claim is made
tp2/P40 (2026-09-06, one rented RTX 5090, receipts in bench/h2h-20260906/tp2/) extends the Unsloth head-to-head to all six families on one box and one fixture, and its rows are statuses, not one number: positions exist only for Qwen3-30B-A3B (s/step ratio Unsloth/e4b 1.457, e4b faster per step, quality COMPARABLE; it agrees with P38's 1.413 inside the pre-registered anchor band, e4b.train.h2h.unsloth.qwen3.5090.2026-09-06) and Mixtral-8x7B, whose row leads with the footprint trade: e4b trained its experts under CPU offload at roughly one-ninth of Unsloth's resident allocator peak (the registered design; the footprint row e4b.train.h2h.unsloth.mixtral.5090.2026-09-06.footprint -- a trainable-on-smaller-cards capability, quality COMPARABLE) while Unsloth ran resident, its only mode, and what that VRAM buys it is speed per step (s/step ratio 0.361, e4b.train.h2h.unsloth.mixtral.5090.2026-09-06 -- a footprint-vs-speed trade, not a kernel deficit). No position on the other four, and the coverage rows are results, not empty cells: on Granite the comparator attached LoRA to the attention only, a fraction of e4b's trainable count with ExpertsLoRA on every MoE layer (e4b.train.h2h.unsloth.granite.5090.2026-09-06.coverage), and on OLMoE it died at MoE-LoRA engage while both e4b arms trained (e4b.train.h2h.unsloth.olmoe.5090.2026-09-06.coverage); gpt-oss refuses on both sides (P5 held), and Gemma-4's e4b attention-4-bit arms died on the projection-count check (#412, fixed since by #435; the arms have run since in P56 and tp4, all 115 projections converted, and are not licensed because tp1's constant parity band fails every accelerated path on this model, the kernel-free one included: #713; the bf16-attention fast_train path stays as tp1 left it) while Unsloth's arm trained. P38's two-hundred-step curve row (Unsloth lower at the end of the curve) still stands beside any Qwen3 position; per-arm rows are e4b.train.h2h.unsloth.<family>.5090.2026-09-06.arm.*
Attention projection targeting (add_attention_lora and TRAIN_ATTN_4BIT / quantize_attention_projections_4bit) is by module STRUCTURE, never family name: q_proj, k_proj and o_proj must be supported linears; v_proj may be a supported linear or absent/None (Gemma-4 attention_k_eq_v layers). The expected count is len(candidates) from detect_attention_projections, never 4 * n_layers. A layer with q_proj/o_proj but no k_proj is refused. A projection that carries a bias is refused (gpt-oss). Gemma-4 attention-4-bit is supported since lane P67 (2026-09-24, #713): with this detector its arms convert all 115 projections and run VALID (P56 ladder-3, tp4-c-4, P67), and read against the family's own floor -- the reference against its own reorderings (e4b.train.p67.gemma4.attn4-reorder-floor.5090.2026-09-24) instead of tp1's constant 0.05 band against zero -- the fused arm is indistinguishable from that reordering noise (e4b.train.p67.gemma4.fused-attn4.floor-band.5090.2026-09-24)
Double-quantized expert absmax, the CLI trainer's default for resident training since TC1 amendments 28 / 31 and enable_fast_train's since amendment 56 (compress_expert_absmax_(model) in code; E4B_ABSMAX_DQ=0, or enable_fast_train(model, absmax_dq=False), turns it off; for python -m experts4bit_qlora.train =1 requires it): stores each ExpertsLoRA-wrapped NF4 stack's fp32 absmax (one fp32 per 64 weights) in bitsandbytes' nested form (quantize_4bit(compress_statistics=True)'s offset, 8-bit codes and one fp32 per 256 values), about a quarter of the bytes; the absmax VALUES change (lossy against fp32, identical to bitsandbytes' double-quant), and with it off nothing changes. Resident training only: enable_fast_train and the ExpertsLoRA reference loop expand one layer's absmax just in time; expert offload, enable_batched_train, the hot/cold/pipelined/hybrid/NVMe engines and the gpt-oss / DeepSeek-V4 per-expert forwards refuse a compressed model (AbsmaxCompressedError), so attach those engines before enable_fast_train. Its step cost and memory are registered: e4b.train.absmax-dq.mixtral.5090.2026-10-05, e4b.train.absmax-dq.qwen3.5090.2026-10-05, e4b.train.absmax-dq.packed-4k.5090.2026-10-07
estimate_qlora_footprint's activation line is a stated heuristic, not a measurement, and the estimate excludes the CUDA context and allocator fragmentation; compare it with a measured peak before trusting it on a new card or recipe

run-moe-larger-than-vram · models, requirements and limits

Model families: olmoe; qwen3_moe; qwen3_5_moe; gemma4_text; granitemoe; gpt_oss; deepseek_v4 (Flash, Pro); qwen3_next; deepseek_v2; mixtral
OS / accelerator: Linux · NVIDIA CUDA GPU (bitsandbytes 4-bit)
Python: >=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested)

enable_pipelined_residency patches an ExpertsLoRA wrapper's base and only runs when the wrapper delegates (eval mode, no_grad, an adapter that contributes nothing): assert the count and check the served path
enable_hot_residency is deprecated since 0.6.2
Speed absolutes depend on the host link; only ratios travel between hosts (claim e4b.host.ratios-travel-absolutes-do-not)

offload-moe-experts-to-cpu-or-nvme · models, requirements and limits

Model families: qwen3_moe; olmoe; deepseek_v4 (Flash, Pro); gpt_oss
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)
Python: >=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested)

The training arena is a separate GPU-side bake (grouped-nf4-gemm's nvme_bake_nf4); e4b.open.tr2-repro-gap, reproducing the TR2 training receipt, is still open in the register (docs/STATUS.md)
enable_nvme_train_residency refuses hot_rows below the expert count and requires gradient checkpointing
NVMe residency is a batch-throughput path; cold-expert latency is bounded by the device link

serve-moe-on-consumer-gpu · models, requirements and limits

Model families: qwen3_moe; olmoe; granitemoe; gpt_oss; gemma4_text; mixtral
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)
Python: >=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested)

Not a general-purpose vLLM replacement: on the same box with identical prompt token ids vLLM 0.30.0 decodes faster than this package's current int4 stack at both B=1 and B=16 (claims e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b1.5090.2026-09-22, e4b.serve.h2h.vllm-0.30.0.p58.qwen3.b16.5090.2026-09-22), bounded to graph decode on one RTX 5090 box with one prompt set; quality quoted, never equated; the 2026-09-05 comparison against vLLM 0.28.0 and the NF4 control (e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05) is history
Gemma-4 has no quality instrument at this resolution (claim e4b.parity.gemma4.no-reference); gpt-oss raw-text perplexity cannot rank arms
Granite's int4-expert rows fail the registered K8 budget (claim IDs carry the numbers); its licensed stack keeps NF4 experts with the folds and epilogue (e4b.serve.census.bo7.granite.b1.5090.2026-09-05, licensed by e4b.serve.buildout.granite.b1.5090.2026-09-04); the per-family positions the census quotes are in docs/STATUS.md -- Qwen3's licensed stack is the streamed calibrated pack (e4b.serve.buildout.bo6c.qwen3.all-calibexp-streamed-64k.k8.2026-09-05), OLMoE's and Mixtral's position is NF4, Gemma-4's is the exact round-1 fold on NF4 with no quality instrument, gpt-oss's is its NF4 reference arm
Single-stream speed absolutes are measured-private where the register says so; ratios travel, absolutes do not
The licensed Qwen3 int4 expert pack travels as bytes: the streamed 64k calibration reproduced it byte-for-byte on one box and across boxes and a release boundary (claims e4b.serve.p55x.qwen3.pack-deterministic-samebox.2026-09-22, e4b.serve.p55x.qwen3.pack-reproduces-crossbox.2026-09-22), and loaded back by fingerprint it passes the registered gate on both texts (e4b.serve.p55x.qwen3.all-calibexp-streamed-64k.k8.2026-09-22); P37's divergent pack, which failed on C4 validation (e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05.gate), was an outlier. The fingerprint covers the experts only: the calibrated attention projections are re-derived on every load, and with RTN attention the same pinned experts fail C4 validation (e4b.serve.p55x.qwen3.rtn-attn.k8.2026-09-22) -- tracked as issue #674
estimate_serve_footprint prices serve_paged.build_engine's all-VRAM placement and the solver's VRAM/DRAM/NVMe tiers as the server builds them without a routing profile; its prefill/decode working set is a stated heuristic, and CUDA graph pools, the prefill graph's pool, the CPU tier's compute buffers, the CUDA context and fragmentation are excluded. It has been checked on two cards only (lane P109's Qwen3-30B-A3B receipts on an RTX 5090, OLMoE-1B-7B on an RTX A2000 at both placements), where it sat slightly under the measured allocator peak; the CHANGELOG gives the figures

mxfp4-moe-training-and-residency · models, requirements and limits

Model families: gpt_oss; deepseek_v4 (Flash, Pro)
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)
Python: >=3.11 tested in CI (pyproject says >=3.10; 3.10 is not tested)

A uniform int4 grid cannot hold e2m1 levels: re-quantising MXFP4 experts to int4-b32 fails the quality gate (docs/STATUS.md)
gpt-oss raw-text perplexity is not an instrument for ranking an exact arm against a noisy one; the native store's speed is quoted with its quality gate open (docs/SERVING-THROUGHPUT.md)
MXFP4 arena training needs grouped-nf4-gemm>=0.12.0 (F8_E8M0 scales), below the current [fast] floor in pyproject.toml
gpt-oss expert training is EXPERIMENTAL and unlicensed. Lane tp1 (bench/train-parity-20260905/tp1/) ran grouped-nf4-gemm's run_mxfp4_20b_qlora (mxfp4_qlora.ExpertsMxfp4LoRA over the released e2m1/e8m0 bytes, faithful clamped GLU and biases) on its own text: the step-0 canary against transformers' dequant path passed, provenance pre == post held over every native expert tensor, and the loss fell -- the row is e4b.train.parity.tp1.gptoss.mxfp4.2026-09-05. No parity pair exists, the run is never compared to the e4b arms, and nothing licenses it; the e4b enablers refuse gpt-oss (zero patched), and the arena_train=True wrap of gpt-oss computes a generic epilogue without the per-expert biases and clamp -- do not train gpt-oss experts through it
enable_mxfp4_nvme_residency REFUSES bias-carrying (gpt-oss-shaped) modules (#402): it passed no biases and defaulted to the DeepSeek-V4 epilogue, so binding a gpt-oss stack to a native-byte arena computed an unfaithful forward (#397); the native MXFP4 store for gpt-oss serving is the paged engine's route (enable_serve_experts_int4 on gpt-oss), and gpt-oss training against an arena has no route -- training_support.gpt_oss.nvme_train is refused, with that code reference

`grouped-nf4-gemm` — docs/capabilities.json at 30adb1bfa76f

Capability	Task / format	Status	Scope
grouped-nf4-gemm grouped-nf4-gemm	inference, training, benchmark nf4	supported	Models, requirements and limits ↓
native-mxfp4-moe-inference grouped-nf4-gemm	inference, training, serving mxfp4	supported	Models, requirements and limits ↓
int4-decode-gemv grouped-nf4-gemm	inference, serving, calibration, packing int4	measured-private	Models, requirements and limits ↓
fp8-paged-attention-fp8-compute grouped-nf4-gemm	serving, inference fp8, bf16	supported	Models, requirements and limits ↓
fp8-paged-attention-f32-compute grouped-nf4-gemm	serving, inference fp8, bf16	supported	Models, requirements and limits ↓
decode-glue-kernels grouped-nf4-gemm	serving, inference bf16	measured-private	Models, requirements and limits ↓
stream-experts-from-host-or-nvme grouped-nf4-gemm	offload, streaming, serving, training nf4, mxfp4	supported	Models, requirements and limits ↓
verify-checkpoint-provenance grouped-nf4-gemm	verification mxfp4, nf4, bf16	supported	Models, requirements and limits ↓

grouped-nf4-gemm · models, requirements and limits

Model families: any fused-expert MoE whose experts are packed in the bitsandbytes NF4 gemm_4bit layout (Qwen3-MoE, OLMoE, Mixtral, GraniteMoe, Gemma-4 text tower via experts4bit-qlora)
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target)
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

Loses to a CUDA-graphed per-expert baseline at some decode shapes (claim gnf4.kernel.graphed-baseline-decode-loses); shapes below the dispatch floor belong to the dequant path: nf4_grouped.decode_dispatch says so and the caller routes them there, while the op itself always runs fused (docs/STATUS.md)
CUDA + Triton only; calling the kernel on CPU raises and names dequant_ref
The grouped_mm training route needs compute capability 9.0 (torch 2.8's torch._grouped_mm target): GNF4_TRAIN_GEMM=auto takes it only there, and GNF4_TRAIN_GEMM=grouped_mm refuses elsewhere with the reason; it is not bit-identical to the fused kernels.
The decoded training route (GNF4_TRAIN_GEMM=decoded: dequant_groups + one grouped bf16 GEMM launch per chunk of groups, the chunk's decode transient capped by GNF4_DECODED_MAX_BYTES, default 256 MiB) is opt-in only: GNF4_TRAIN_GEMM=auto never takes it. It is not bit-identical to the fused kernels or the dense route, and no speed is claimed for it. experts4bit-qlora's TC1 amendment 46 (one RTX 5090, experts4bit-qlora#1221) read the training step at decoded/fused 1.005 [0.979, 1.031] on OLMoE-1B-7B and 1.066 on Qwen3-30B-A3B, so auto does not take it.
The fidelity claim is measured against fp32 accumulation on NVIDIA tensor cores; other backends must re-measure (docs/PORTABILITY.md)
gemm_nf4_grouped_smallm's lut="prmt" (lane K39) is an opt-in byte-permute codebook decode in inline PTX, bit-identical to lut="tree" on NVIDIA targets (the same codebook values and the same operand layout); it is refused on ROCm, decodes through the tree under the interpreter, and this package claims no speed for it. experts4bit-qlora serves it by default on qwen3_moe since 0.56.0 (its served read P158, e4b.serve.p158.k25-prmt.qwen3.5090.2026-10-20)
GNF4_GEMV_BW routes single-row decode calls through the bandwidth-targeted GEMV (lane K33). Its default, auto, engages at the expert shapes in _BW_SHAPES on >= 160-SM parts -- Qwen3-30B-A3B's (experts4bit-qlora's served lane P116, DEFAULT_ON: one request faster, 16 requests unchanged, within P110's teacher-forced quality bar), Granite-3.1-3b-a800m's and OLMoE-1B-7B's (P134, e4b.serve.p134.gemv-bw.granite.5090.2026-10-18, e4b.serve.p134.gemv-bw.olmoe.5090.2026-10-18) and Mixtral-8x7B's (P141, e4b.serve.p141.gemv-bw.mixtral.5090.2026-10-18) -- and at Qwen3-30B-A3B's on 128-SM sm_89 parts, the RTX 4090 (P133, e4b.serve.p133.gemv-bw.qwen3.4090.2026-10-18); every other shape and part keeps the scalar GEMV, and dot-pad runs only with GNF4_GEMV_BW=0. GNF4_GEMV_BW=1 forces it everywhere and 0 turns it off. It runs K33's selected plan per shape (_BW_PLANS; GNF4_GEMV_BW_PLAN overrides) with GNF4_PDL at its default. K33 read LEVER at kernel level on an RTX 5090 (claim gnf4.kernel.k33-nf4-decode-gemv-bw.5090.2026-10-07). Its codebook decode is exact fp32 either way: the PTX byte-permute table (prmt32) needs a compiled NVIDIA target, and the interpreter and non-PTX backends take an exact select tree (GNF4_GEMV_BW_DECODE=tree). It is not bit-identical to the scalar or dot-pad routes (a different reduction tree; the tolerance contract applies).
Every speed figure is per card -- the census is sm_86, the head-to-head and the real-finetune cells are RTX 4090 / H100, the torch._grouped_mm cell is RTX 5090 -- and none is an architecture-wide statement; the registered comparator is this repository's own per-expert dequantize-then-GEMM loop, and no registered cell times bitsandbytes' own packed 2-D inference forward (0.50.0 and later)

native-mxfp4-moe-inference · models, requirements and limits

Model families: gpt-oss (20b, 120b); DeepSeek-V4 lineage; Kimi K3 lineage (reference decode)
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target)
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

A uniform int4 grid cannot represent e2m1 levels: do not quantise-bake an already-MXFP4 checkpoint to NF4 or int4 (docs/solutions/native-mxfp4-moe-inference.md)
gemv_mxfp4_b32 re-streams weights per row: it is a decode lever for a handful of rows; batched rows use the grouped GEMM or the consumer's NF4 path
No ROCm/XPU
The claims here are fidelity and provenance results from single runs per card (a rented H200 pod for serving, an L40S and an A2000 for training, the K3 oracle on CPU); the MXFP4 GEMM has no speed census of its own and gemv_mxfp4_b32 is inside no registered claim

int4-decode-gemv · models, requirements and limits

Model families: any expert or projection weight packed with pack_int4_b32 / gptq_pack_int4_b32 (driven by experts4bit-qlora's serve lanes)
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target)
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

The int4-b32 GEMV and the calibrated packer are measured-private (claims gnf4.serve.int4-b32-gemv, gnf4.serve.gptq-pack-int4-b32): the receipts live in the consumer's private audit tree
Split-K on the NF4 dot-pad decode GEMV (nf4_grouped, GNF4_GEMV_SPLITK, lane K7) was refuted once and ships dormant there (the retired claim gnf4.retired.splitk-gemv records the refutation); the int4-b32 GEMV's own split-K is the shipped, measured plan (int4_b32._plan, reduced by reduce_partials)
Split-K on the int4-b32 GEMV is planned from (N, K) alone on every part: the row-count term (R >= 16, gated by SPLITK_R_TERM_MAX_SMS) is off since K30 read it OFF on a rented NVIDIA L4 (claim gnf4.kernel.k30-splitk-r-term.l4.2026-10-05), so the plan no longer depends on the SM count and two cards plan the same sk for the same call
Whether int4 experts pass a quality gate is the consumer's per-family question (experts4bit-qlora docs/STATUS.md): small-active-parameter models fail it with round-to-nearest packing
The single-stream decode anchor (claim gnf4.serve.decode-anchor-5090) is a consumer-measured anchor of the serving class on a knob-off basis, not a measurement of this lane, and is not cited here as its evidence
K16: int4_smallm.gemm_int4_b32_smallm is a small-M single-projection GEMM -- in-register int4 dequant, per-block scaling inside the tile, bf16 tensor-core MMA over fat K chunks, split-K fused into the launch (kernel/PREREG-k16-smallm-int4-gemm.md). Measured on the RTX 5090 class at M=16 (claim gnf4.kernel.k16-smallm-int4-gemm.5090.2026-09-19): it beats this package's bf16 dequant path on every attention shape and meets the pre-registered Marlin bar; the single-scale control (P4) was not tested; P5 holds in the consumer (claim gnf4.kernel.k16-smallm-int4-gemm.5090.2026-09-19), and the consumer's route is its default (auto) from experts4bit-qlora 0.36.2
K18: int4_b32.gemv_int4_b32_grouped tiles the call's rows expert-major in-register (histogram, cumsum, per-expert rank -- no sort, no host sync, capture-legal) so each program loads one expert's weight slice once for up to GROUPED_MT_DEFAULT=4 of its rows; it is torch.equal to gemv_int4_b32(..., fused_reduce=False). Measured on the RTX 5090 (claim gnf4.kernel.k18-grouped-expert-gemv.5090.2026-09-22): exact, and SLOWER than gemv_int4_b32 both on the recorded Qwen3-30B-A3B B=16 routing and at the small row counts, so it is not a lever and ships dormant as the evidence; use gemv_int4_b32. Nothing in this package or the consumer routes to it. The wrapper refuses E > GROUPED_E_MAX=4095 (the tiling's shared-memory histogram)
build_group_tiles_fused(..., rank="cumsum", programs=P) (e4b#846, opt-in) splits the cumsum tile table over P programs (_tile_table_cumsum_mp): each takes every expert's count from one tl.histogram of the ids and ranks and writes only its own experts' rows, so no row has two writers; programs=1 (the default) is the one-program kernel, launched as before. The same integers at every P by test (kernel/test_tile_table_programs_interp.py: E 40/128/256, R 1-1024, P 2/3/8/64, interpreter and compiled on an RTX A2000). Unread for speed: experts4bit-qlora's lane reads it before any default moves.

fp8-paged-attention-fp8-compute · models, requirements and limits

Model families: driven by experts4bit-qlora's paged engine for Qwen3-MoE, OLMoE, GraniteMoe, gpt-oss, Gemma-4, Mixtral
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_89 or newer (the fp8 tensor-core dot); measured on the RTX 5090 (sm_120) only -- sm_89 and sm_90 meet the requirement but no registered cell was run there
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

One card class measured: both claims were run on the RTX 5090 (sm_120). The sm_89+ requirement is the kernel's precondition, not a measured statement about Ada or Hopper parts
Adds one e4m3 rounding on q and one on p, so its test bound is wider than the f32 path's (kernel/test_fp8_paged_attn.py)
pack_heads=True falls back to the split fp8 kernel with a one-time RuntimeWarning where the packed tile exceeds the card's shared memory
Served-path parity (Granite, gpt-oss, Gemma-4) is measured in experts4bit-qlora's private receipts; the evidence here is kernel-level parity against paged_attn_ref
KV budgets in docs/context-budgets.md are rung-one (A2000) numbers; pending rows must not be promoted

fp8-paged-attention-f32-compute · models, requirements and limits

Model families: driven by experts4bit-qlora's paged engine for Qwen3-MoE, OLMoE, GraniteMoe, gpt-oss, Gemma-4, Mixtral
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer; this is the path sm_80-sm_88 take by default. Reference tests measured on an RTX A2000 (sm_86) and an RTX 5090 (sm_120), torch 2.8.0+cu128 / triton 3.4.0, with the arms pinned to the compute mode they name: identical errors on both, every one a bf16 output ULP, none over tolerance (gnf4.serve.f32-arms-ran-fp8)
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

#319 is closed: its failures were the test's shape arms running the fp8 kernel on sm_89+ while judged at the f32 tolerance, never a defect in these kernels (claim gnf4.serve.f32-arms-ran-fp8); the earlier claim gnf4.open.f32-compute-modes-triton34 is retired
Measured on two cards (sm_86 and sm_120) on one torch/Triton pair; other pairs are unmeasured. The old advice to gate lanes with -k "f8dot or pf8" excluded exactly the mislabelled arms and should not be followed (docs/STATUS.md)
Explicit GNF4_ATTN_COMPUTE=f32 or compute='f32' selects this path on any card, sm_89+ included: the explicit request is honoured, never downgraded to fp8

decode-glue-kernels · models, requirements and limits

Model families: driven by experts4bit-qlora's paged engine for Qwen3-MoE, OLMoE, GraniteMoe, gpt-oss, Gemma-4, Mixtral
OS / accelerator: Linux · NVIDIA CUDA GPU, sm_80 or newer (the composition was measured on the RTX 5090, sm_120)
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

The composition timing and the paired perplexity deltas are measured-private (claim gnf4.serve.decode-glue-kernels): real runs through the consumer on the RTX 5090 whose receipts live in a private audit tree; what is checkable here is each kernel's parity test
int4_b32 imports triton at module level and is not importable without it
combine_rows and reduce_partials carry an accuracy contract, not a bitwise one: each is within the error bound of a correct fp32 summation in every census case on an RTX 5090, and neither is bitwise the torch chain it replaced (lane B393; gnf4.kernel.combine-rows-accuracy.5090.2026-09-23, gnf4.kernel.reduce-partials-slot-order.5090.2026-09-23); the kernel's output and torch's chain were each bit-identical on sm_86 and sm_120 at the census cases (corrected 2026-09-24)
GNF4_PDL is on by default, capped at 8 activation rows, since experts4bit-qlora lane P113 read CAP_DEFAULT on SC1's int4 serving configuration (e4b.serve.p113.gnf4-pdl-capped.qwen3-int4.5090.2026-10-04; uncapped it cost the 16-request step there, which is why the cap is the default); lane K28 read the mechanism on the decode chain (gnf4.kernel.k28-pdl-decode-chain.5090.2026-10-04); it is inert on CPU, under the interpreter, on ROCm and below sm_90, and its speed elsewhere is unread
dense_gemv.gemv_bf16_bw (lane K35) is experts4bit-qlora's P148 prototype kernel verbatim behind a frozen per-shape plan (any other shape runs the default plan, untimed): one activation row only (a multi-row call keeps cuBLAS), and it routes no model call by itself. Its tolerance contract against cuBLAS is checked on the card (kernel/test_dense_gemv.py); no speed is claimed here, because the consumer's served read is the timing of record
dense_gemv.gemv_i8_bw (lane K36) is experts4bit-qlora's P148 int8 prototype kernel verbatim, with its quantizer quantize_rows_int8 (symmetric per-row absmax, s = absmax / 127) verbatim, behind a frozen plan at Qwen3-30B-A3B's int8 lm_head shape (any other shape runs the default plan, untimed): one activation row only, int8 weights with an fp32 per-row scale, and it routes no model call by itself. Its tolerance contract against cuBLAS on the bf16 head is checked on the card (kernel/test_dense_gemv_i8.py); the int8 grid's effect on a model's quality is the consumer's ruling (experts4bit-qlora P150), and no speed is claimed here: the consumer's served read is the timing of record
dense_i8.linear_i8 (lane K37) is experts4bit-qlora's P149 prototype int8 kernels verbatim (a one-row GEMV and a 2-to-16-row small-M kernel) over K36's quantize_rows_int8 grid, re-exported, behind plans frozen at Mixtral-8x7B's attention projection shapes (any other shape runs the default plans, untimed): 1 to 16 activation rows only, int8 weights with an fp32 per-row scale, and it routes no model call by itself. Its tolerance contract against cuBLAS on the bf16 weights is checked on the card (kernel/test_dense_i8.py); the int8 grid's effect on a model's quality is the consumer's ruling (experts4bit-qlora P147), and its timing is the consumer's served read.

stream-experts-from-host-or-nvme · models, requirements and limits

Model families: Qwen3-235B-A22B (host-RAM flagship); gpt-oss / DeepSeek-V4 / Kimi lineage MXFP4 arenas; any checkpoint whose experts the bake can enumerate
OS / accelerator: Linux · NVIDIA CUDA GPU for the serving tiers and for the default NF4 quantise bake (nvme_bake_nf4.bake_nf4 with quantize_fn=None); the relocation bake and verify (nvme_arena.bake / bake_expert_tensors / verify) need no GPU
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

NVMe is a batch tier: per-token latency is bounded by the device link (claim gnf4.nvme.tier-batch-only)
Expert prefetch closed negative over four arcs (claim gnf4.flagship.prefetch-closed-negative)
The cold-engine 'free floor' premise was refuted (claim gnf4.cold-engine.phase0-premise-refuted); read docs/INDEX.md before quoting any cold-engine number
This entry is the primitive layer (arena, reader, row tier, low-level residency); which bytes live where and the end-to-end run of a MoE larger than VRAM are experts4bit-qlora's pages run-moe-larger-than-vram (decision/router) and offload-moe-experts-to-cpu-or-nvme (model-level integration)

verify-checkpoint-provenance · models, requirements and limits

Model families: any safetensors checkpoint; MXFP4 expert tensors for gpt-oss / DeepSeek-V4 / Kimi lineage
OS / accelerator: Linux · none: pure torch (a CUDA GPU is optional)
Python: >=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)

A quantize-at-bake arena (nvme_bake_nf4) is a documented two-hop transform: its manifest's bake_mode says it is bit-identical to the quantiser's output, not to a bf16 release
Making the verifier pass on non-matching bytes is the highest-severity class in SECURITY.md; report privately

rendered from experts4bit-qlora 0.58.1 · source: pjordanandrsn/experts4bit-qlora@605d21e67958 · checked 2026-10-25

rendered from grouped-nf4-gemm 0.51.0 · source: pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f · checked 2026-10-25

Compatibility

Package compatibility

Historical records

experts4bit-qlora — docs/capabilities.json at 605d21e67958

grouped-nf4-gemm — docs/capabilities.json at 30adb1bfa76f

`experts4bit-qlora` — docs/capabilities.json at 605d21e67958

`grouped-nf4-gemm` — docs/capabilities.json at 30adb1bfa76f