CERIN AMROTH · ML systems

Compatibility

The package-to-package floor comes from the system manifest; the per-capability tables are generated from each repository's docs/capabilities.json at the pinned commit — the same file the repositories validate in CI. The status column is each row's own register value, shown per row and never merged (the words are defined in the evidence vocabulary). A family or platform not listed here is not supported; the repositories fail fast rather than fall back.

Package compatibility

From the compatibility records of docs/system-manifest.json. The current record is the one whose consumer_versions range contains the rendered experts4bit-qlora version (0.35.3); its floor is validated against pyproject.toml at the pin, never trusted from the manifest, and the rendered grouped-nf4-gemm (0.30.2) satisfies it.

experts4bit-qlora >=0.35.0 → grouped-nf4-gemm >=0.30.0 (extra: fast)

Why: combine_rows (grouped-nf4-gemm#335) with the swiglu_rows / reduce_partials decode glue that experts4bit-qlora#385 integrates, and gptq_pack for the calibrated int4 expert store (experts4bit-qlora#384)

Consumer CI pin: the consumer's CI installs the kernel package from the git commit of the v0.30.1 release · since 2026-09-04.

Historical records

Earlier floors, kept as the record; not current.

experts4bit-qlora — docs/capabilities.json at 0c2a256dcdc2

CapabilityModesFormatsModel familiesOS / acceleratorPythonStatusLimitations
quantize-fused-moe-experts
experts4bit-qlora		quantization, inference, trainingnf4, fp4, int8, fp8, bf16, fp16olmoe; qwen3_moe; qwen3_5_moe; gemma4_text; granitemoe; gpt_oss; deepseek_v4 (Flash, Pro); qwen3_next; deepseek_v2; mixtralLinux · NVIDIA CUDA GPU (bitsandbytes 4-bit)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • Unsupported model_type raises NotImplementedError; identity-expert families (LongCat-Flash) are refused by name
  • 4-bit is a memory trade when the model already fits, and cost energy on the measured comparator (one card, one bitsandbytes development build): claim e4b.train.energy-honest.scoped-a2000
  • Dense (non-MoE) models are not the target; use bitsandbytes directly
qlora-fused-moe-experts
experts4bit-qlora		trainingnf4olmoe; qwen3_moe; gemma4_text; mixtral; granitemoeLinux · NVIDIA CUDA GPU (bitsandbytes 4-bit)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • enable_fast_train returns 0 (does not raise) when the kernel package is absent: assert the return value
  • dgrad=True needs grouped-nf4-gemm>=0.7.0 (below the current [fast] floor in pyproject.toml, so any [fast] install has it); on older cuts it is downgraded with a RuntimeWarning
  • Loss parity is measured on the registered criteria in docs/METHODOLOGY.md, not asserted for every dataset
  • model_families is evidence-gated: a family is listed when its fast_train path -- the headline path, enable_fast_train -- PASSES the registered loss-parity criteria on real weights with a receipt in this repository (training_support is the per-path record; bench/train-parity-20260905/tp1/ is lane tp1, all six serving families). olmoe is confirmed on real weights and a registered text; qwen3_moe and gemma4_text carry tp1 PASSes beside their flagship matrices; mixtral entered on its tp1 fused PASS under offload; granitemoe entered on its corrected-counter re-run's PASS (its first attempt was a harness error, kept as its own row); gpt_oss is refused by every training enabler and stays out
  • gpt_oss is not a training family here: the loader builds its experts bare (no ExpertsLoRA), and both enable_fast_train and enable_batched_train return zero on it -- REFUSED rows on tp1 (e4b.train.parity.tp1.gptoss.fused.2026-09-05, e4b.train.parity.tp1.gptoss.batched.2026-09-05). Attention-only QLoRA over its frozen experts trains (e4b.train.parity.tp1.gptoss.attn_only.2026-09-05); the only route that trains its experts is grouped-nf4-gemm's experimental mxfp4_qlora.ExpertsMxfp4LoRA under the mxfp4-moe-training-and-residency capability -- experimental, never licensed
  • enable_batched_train falls back to the reference forward per call above _PAD_WASTE_LIMIT (engines/batched.py): the patch count stays positive while some or all layers run the per-expert loop on some steps. In the shipped code tp1 measured this fallback was uncounted, and three arms read VOID on exactly that -- OLMoE, Qwen3 and Gemma-4 (e4b.train.parity.tp1.olmoe.batched.2026-09-05, e4b.train.parity.tp1.qwen3.batched.2026-09-05, e4b.train.parity.tp1.gemma4.batched.2026-09-05); it engaged everywhere only on Mixtral's eight-expert and Granite's forty-expert shapes. As of the release carrying that bundle the fallback is countable: batched_fallback_stats(model) reports it (#402; those VOID rows are the reason it exists). Assert kernel engagement, not the return value alone
  • enable_fast, enable_fast_train and enable_batched_train refuse a wrapper whose base violates the stock-epilogue contract (EpilogueContractError, #402) instead of silently computing act(gate)*up over a base that applies biases, clamps or another gating rule -- a refusal is loud, a wrong epilogue is not; gpt-oss is the family this protects, and DeepSeek-V4 supplies its epilogue through _apply_gate and is admitted
  • Head-to-head against Unsloth's 4-bit MoE QLoRA path on one identical training problem (lane p38, 2026-09-05, one rented RTX 5090, Qwen3-30B-A3B, the clinical fixture tokenised once, seq 512, r 8 / alpha 16, identical targets, optimizer, batch, steps, precision and eval; bench/h2h-20260905/p38/): the shipped fused dgrad path with NF4 attention is faster per step, lower in peak VRAM and lower in energy per step than Unsloth on that box, and the two are comparable on held-out loss at sixty steps (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05, ...quality-n60) -- BUT the two-hundred-step curve favours Unsloth: its held-out loss is lower at the end of the curve while e4b's flattens from step sixty (e4b.train.h2h.unsloth.qwen3.5090.2026-09-05.curve-n200). Quote the position only beside that row. Candidate causes (the eval schedule, the checkpointing mode, the transformers/peft versions of the two stacks, the expert adapter's precision -- bf16 on this side, the loader passes the model dtype to ExpertsLoRA -- against fp32 on Unsloth's) are not established. One workload, one box, one family; nothing here licenses a path (tp1 does) and no general speed claim is made
  • tp2/P40 (2026-09-06, one rented RTX 5090, receipts in bench/h2h-20260906/tp2/) extends the Unsloth head-to-head to all six families on one box and one fixture, and its rows are statuses, not one number: positions exist only for Qwen3-30B-A3B (s/step ratio Unsloth/e4b 1.457, e4b faster per step, quality COMPARABLE; it agrees with P38's 1.413 inside the pre-registered anchor band, e4b.train.h2h.unsloth.qwen3.5090.2026-09-06) and Mixtral-8x7B, whose row leads with the footprint trade: e4b trained its experts under CPU offload at roughly one-ninth of Unsloth's resident allocator peak (the registered design; the footprint row e4b.train.h2h.unsloth.mixtral.5090.2026-09-06.footprint -- a trainable-on-smaller-cards capability, quality COMPARABLE) while Unsloth ran resident, its only mode, and what that VRAM buys it is speed per step (s/step ratio 0.361, e4b.train.h2h.unsloth.mixtral.5090.2026-09-06 -- a footprint-vs-speed trade, not a kernel deficit). No position on the other four, and the coverage rows are results, not empty cells: on Granite the comparator attached LoRA to the attention only, a fraction of e4b's trainable count with ExpertsLoRA on every MoE layer (e4b.train.h2h.unsloth.granite.5090.2026-09-06.coverage), and on OLMoE it died at MoE-LoRA engage while both e4b arms trained (e4b.train.h2h.unsloth.olmoe.5090.2026-09-06.coverage); gpt-oss refuses on both sides (P5 held), and Gemma-4's e4b attention-4-bit arms died on the projection-count check (#412; the bf16-attention fast_train path stays as tp1 left it) while Unsloth's arm trained. P38's two-hundred-step curve row (Unsloth lower at the end of the curve) still stands beside any Qwen3 position; per-arm rows are e4b.train.h2h.unsloth.<family>.5090.2026-09-06.arm.*
run-moe-larger-than-vram
experts4bit-qlora		offload, inference, training, servingnf4olmoe; qwen3_moe; qwen3_5_moe; gemma4_text; granitemoe; gpt_oss; deepseek_v4 (Flash, Pro); qwen3_next; deepseek_v2; mixtralLinux · NVIDIA CUDA GPU (bitsandbytes 4-bit)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • enable_pipelined_residency patches an ExpertsLoRA wrapper's base and only runs when the wrapper delegates (eval mode, no_grad, an adapter that contributes nothing): assert the count and check the served path
  • enable_hot_residency is deprecated since 0.6.2
  • Speed absolutes depend on the host link; only ratios travel between hosts (claim e4b.host.ratios-travel-absolutes-do-not)
offload-moe-experts-to-cpu-or-nvme
experts4bit-qlora		offload, serving, training, streamingnf4, mxfp4qwen3_moe; olmoe; deepseek_v4 (Flash, Pro); gpt_ossLinux · NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • No shipped tool bakes the training arena from a bf16 checkpoint yet (docs/STATUS.md, open)
  • enable_nvme_train_residency refuses hot_rows below the expert count and requires gradient checkpointing
  • NVMe residency is a batch-throughput path; cold-expert latency is bounded by the device link
serve-moe-on-consumer-gpu
experts4bit-qlora		serving, inferencenf4, int4, fp8qwen3_moe; olmoe; granitemoe; gpt_oss; gemma4_text; mixtralLinux · NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • Not a general-purpose vLLM replacement: on the same box with identical prompt ids vLLM 0.28.0 is ahead of this package's NF4 control at both batch sizes (claim e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05); the ratio against the licensed stack is not quoted on that lane because its licensed arms were void on that box under the pre-registered pack-fingerprint rule (the streamed calibration did not reproduce the licensed pack's expert-matrix counts), the registered K8 gate run on that box's pack failed its second text (claim e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05.gate), so the recipe's speed there stays unlicensed and the streamed calibration's host-dependence is an open item; the 2026-09-03 comparison is superseded
  • Gemma-4 has no quality instrument at this resolution (claim e4b.parity.gemma4.no-reference); gpt-oss raw-text perplexity cannot rank arms
  • Granite's int4-expert rows fail the registered K8 budget (claim IDs carry the numbers); its licensed stack keeps NF4 experts with the folds and epilogue (e4b.serve.census.bo7.granite.b1.5090.2026-09-05, licensed by e4b.serve.buildout.granite.b1.5090.2026-09-04); the per-family positions the census quotes are in docs/STATUS.md -- Qwen3's licensed stack is the streamed calibrated pack (e4b.serve.buildout.bo6c.qwen3.all-calibexp-streamed-64k.k8.2026-09-05), OLMoE's and Mixtral's position is NF4, Gemma-4's is the exact round-1 fold on NF4 with no quality instrument, gpt-oss's is its NF4 reference arm
  • Single-stream speed absolutes are measured-private where the register says so; ratios travel, absolutes do not
  • The streamed 64k calibration recipe does not reproduce its licence across hosts: re-derived on another RTX 5090 host with the same checkpoint, text and knobs it produced a different pack (a handful of expert matrices classified differently at the min_rows threshold) that fails the registered gate on C4 validation (claim e4b.serve.h2h.vllm-0.28.0.qwen3.5090.2026-09-05.gate); the licence of e4b.serve.buildout.bo6c.qwen3.all-calibexp-streamed-64k.k8.2026-09-05 stands on its box as measured, and a pack derived elsewhere carries no licence until gated on that box -- tracked as issue #405
mxfp4-moe-training-and-residency
experts4bit-qlora		inference, training, offload, servingmxfp4, nf4gpt_oss; deepseek_v4 (Flash, Pro)Linux · NVIDIA CUDA GPU, sm_80 or newer (Triton kernels via grouped-nf4-gemm)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)experimental
  • A uniform int4 grid cannot hold e2m1 levels: re-quantising MXFP4 experts to int4-b32 fails the quality gate (docs/STATUS.md)
  • gpt-oss raw-text perplexity is not an instrument for ranking an exact arm against a noisy one; the native store's speed is quoted with its quality gate open (docs/SERVING-THROUGHPUT.md)
  • MXFP4 arena training needs grouped-nf4-gemm>=0.12.0 (F8_E8M0 scales), below the current [fast] floor in pyproject.toml
  • gpt-oss expert training is EXPERIMENTAL and unlicensed. Lane tp1 (bench/train-parity-20260905/tp1/) ran grouped-nf4-gemm's run_mxfp4_20b_qlora (mxfp4_qlora.ExpertsMxfp4LoRA over the released e2m1/e8m0 bytes, faithful clamped GLU and biases) on its own text: the step-0 canary against transformers' dequant path passed, provenance pre == post held over every native expert tensor, and the loss fell -- the row is e4b.train.parity.tp1.gptoss.mxfp4.2026-09-05. No parity pair exists, the run is never compared to the e4b arms, and nothing licenses it; the e4b enablers refuse gpt-oss (zero patched), and the arena_train=True wrap of gpt-oss computes a generic epilogue without the per-expert biases and clamp -- do not train gpt-oss experts through it
  • enable_mxfp4_nvme_residency REFUSES bias-carrying (gpt-oss-shaped) modules (#402): it passed no biases and defaulted to the DeepSeek-V4 epilogue, so binding a gpt-oss stack to a native-byte arena computed an unfaithful forward (#397); the native MXFP4 store for gpt-oss serving is the paged engine's route (enable_serve_experts_int4 on gpt-oss), and gpt-oss training against an arena has no route -- training_support.gpt_oss.nvme_train is refused, with that code reference

grouped-nf4-gemm — docs/capabilities.json at e2af4cfb91b2

CapabilityModesFormatsModel familiesOS / acceleratorPythonStatusLimitations
grouped-nf4-gemm
grouped-nf4-gemm		inference, training, benchmarknf4any fused-expert MoE whose experts are packed in the bitsandbytes NF4 gemm_4bit layout (Qwen3-MoE, OLMoE, Mixtral, GraniteMoe, Gemma-4 text tower via experts4bit-qlora)Linux · NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • Loses to a CUDA-graphed per-expert baseline at some decode shapes (claim gnf4.kernel.graphed-baseline-decode-loses); shapes below the routing threshold go back to the reference path (docs/STATUS.md)
  • CUDA + Triton only; calling the kernel on CPU raises and names dequant_ref
  • The fidelity claim is measured against fp32 accumulation on NVIDIA tensor cores; other backends must re-measure (docs/PORTABILITY.md)
  • Every speed figure is per card -- the census is sm_86, the head-to-head and the real-finetune cells are RTX 4090 / H100, the torch._grouped_mm cell is RTX 5090 -- and none is an architecture-wide statement; the registered comparator is this repository's own per-expert dequantize-then-GEMM loop, and no registered cell times bitsandbytes' own packed 2-D inference forward (0.50.0 and later)
native-mxfp4-moe-inference
grouped-nf4-gemm		inference, training, servingmxfp4gpt-oss (20b, 120b); DeepSeek-V4 lineage; Kimi K3 lineage (reference decode)Linux · NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • A uniform int4 grid cannot represent e2m1 levels: do not quantise-bake an already-MXFP4 checkpoint to NF4 or int4 (README)
  • gemv_mxfp4_b32 re-streams weights per row: it is a decode lever for a handful of rows; batched rows use the grouped GEMM or the consumer's NF4 path
  • No ROCm/XPU
  • The claims here are fidelity and provenance results from single runs per card (a rented H200 pod for serving, an L40S and an A2000 for training, the K3 oracle on CPU); the MXFP4 GEMM has no speed census of its own and gemv_mxfp4_b32 is inside no registered claim
int4-decode-gemv
grouped-nf4-gemm		inference, serving, calibration, packingint4any expert or projection weight packed with pack_int4_b32 / gptq_pack_int4_b32 (driven by experts4bit-qlora's serve lanes)Linux · NVIDIA CUDA GPU, sm_80 or newer (sm_120 is the primary serving target)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)measured-private
  • The int4-b32 GEMV and the calibrated packer are measured-private (claims gnf4.serve.int4-b32-gemv, gnf4.serve.gptq-pack-int4-b32): the receipts live in the consumer's private audit tree
  • Split-K on the decode GEMV was refuted once and ships dormant (claim gnf4.retired.splitk-gemv records the refutation); the shipped plan is the measured one
  • Whether int4 experts pass a quality gate is the consumer's per-family question (experts4bit-qlora docs/STATUS.md): small-active-parameter models fail it with round-to-nearest packing
  • The single-stream decode anchor (claim gnf4.serve.decode-anchor-5090) is a consumer-measured anchor of the serving class on a knob-off basis, not a measurement of this lane, and is not cited here as its evidence
fp8-paged-attention-fp8-compute
grouped-nf4-gemm		serving, inferencefp8, bf16driven by experts4bit-qlora's paged engine for Qwen3-MoE, OLMoE, GraniteMoe, gpt-oss, Gemma-4, MixtralLinux · NVIDIA CUDA GPU, sm_89 or newer (the fp8 tensor-core dot); measured on the RTX 5090 (sm_120) only -- sm_89 and sm_90 meet the requirement but no registered cell was run there>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • One card class measured: both claims were run on the RTX 5090 (sm_120). The sm_89+ requirement is the kernel's precondition, not a measured statement about Ada or Hopper parts
  • Adds one e4m3 rounding on q and one on p, so its test bound is wider than the f32 path's (kernel/test_fp8_paged_attn.py)
  • pack_heads=True falls back to the split fp8 kernel with a one-time RuntimeWarning where the packed tile exceeds the card's shared memory
  • Served-path parity (Granite, gpt-oss, Gemma-4) is measured in experts4bit-qlora's private receipts; the evidence here is kernel-level parity against paged_attn_ref
  • KV budgets in docs/context-budgets.md are rung-one (A2000) numbers; pending rows must not be promoted
fp8-paged-attention-f32-compute
grouped-nf4-gemm		serving, inferencefp8, bf16driven by experts4bit-qlora's paged engine for Qwen3-MoE, OLMoE, GraniteMoe, gpt-oss, Gemma-4, MixtralLinux · NVIDIA CUDA GPU, sm_80 or newer; this is the path sm_80-sm_88 take by default. Its reference tests were run on an RTX 5090 (sm_120) with the mode forced; no registered cell exists on sm_80-sm_88>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)unsupported
  • Open, #319 (claim gnf4.open.f32-compute-modes-triton34, status open, so it backs nothing): on torch 2.8.0+cu128 / triton 3.4.0, on unmodified main, the split and packed f32 modes miss their fp32 reference beyond the registered tolerance in a subset of the paged-attention tests; the claim carries the numbers and the issue. A kernel that imports and launches is not numerically supported; this entry stays unsupported until the issue closes and the register carries a passing cell
  • A lane on this path cannot be gated with -x on those torch/Triton pairs; gate the fp8 modes with -k "f8dot or pf8" (README, docs/STATUS.md)
  • Explicit GNF4_ATTN_COMPUTE=f32 or compute='f32' selects this path on any card, sm_89+ included: the explicit request is honoured, never downgraded to fp8
decode-glue-kernels
grouped-nf4-gemm		serving, inferencebf16driven by experts4bit-qlora's paged engine for Qwen3-MoE, OLMoE, GraniteMoe, gpt-oss, Gemma-4, MixtralLinux · NVIDIA CUDA GPU, sm_80 or newer (the composition was measured on the RTX 5090, sm_120)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)measured-private
  • The composition timing and the paired perplexity deltas are measured-private (claim gnf4.serve.decode-glue-kernels): real runs through the consumer on the RTX 5090 whose receipts live in a private audit tree; what is checkable here is each kernel's parity test
  • int4_b32 imports triton at module level and is not importable without it
  • reduce_partials (0.29.0) and combine_rows (0.30.0) are inside no registered claim: capability without a published measurement
stream-experts-from-host-or-nvme
grouped-nf4-gemm		offload, streaming, serving, trainingnf4, mxfp4Qwen3-235B-A22B (host-RAM flagship); gpt-oss / DeepSeek-V4 / Kimi lineage MXFP4 arenas; any checkpoint whose experts the bake can enumerateLinux · NVIDIA CUDA GPU for the serving tiers and for the default NF4 quantise bake (nvme_bake_nf4.bake_nf4 with quantize_fn=None); the relocation bake and verify (nvme_arena.bake / bake_expert_tensors / verify) need no GPU>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • NVMe is a batch tier: per-token latency is bounded by the device link (claim gnf4.nvme.tier-batch-only)
  • Expert prefetch closed negative over four arcs (claim gnf4.flagship.prefetch-closed-negative)
  • The cold-engine 'free floor' premise was refuted (claim gnf4.cold-engine.phase0-premise-refuted); read docs/INDEX.md before quoting any cold-engine number
  • This entry is the primitive layer (arena, reader, row tier, low-level residency); which bytes live where and the end-to-end run of a MoE larger than VRAM are experts4bit-qlora's pages run-moe-larger-than-vram (decision/router) and offload-moe-experts-to-cpu-or-nvme (model-level integration)
verify-checkpoint-provenance
grouped-nf4-gemm		verificationmxfp4, nf4, bf16any safetensors checkpoint; MXFP4 expert tensors for gpt-oss / DeepSeek-V4 / Kimi lineageLinux · none: pure torch (a CUDA GPU is optional)>=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)supported
  • A quantize-at-bake arena (nvme_bake_nf4) is a documented two-hop transform: its manifest's bake_mode says it is bit-identical to the quantiser's output, not to a bf16 release
  • Making the verifier pass on non-matching bytes is the highest-severity class in SECURITY.md; report privately

CURRENT · source: pjordanandrsn/experts4bit-qlora@0c2a256dcdc2 · rendered package: 0.35.3 · latest published package: 0.35.3

CURRENT · source: pjordanandrsn/grouped-nf4-gemm@e2af4cfb91b2 · rendered package: 0.30.2 · latest published package: 0.30.2