# MoE Training on Consumer GPUs — Loggetta & Cerin Amroth

Source: https://cerinamroth.com/ml/

[Home](https://cerinamroth.com/) › ML systems 

# Large models. Smaller machines.

 

Fine-tune Mixture-of-Experts models on a single GPU. Start with Loggetta: check the hardware, plan the run, and keep a report of what happened.

 

```
pip install loggetta
loggetta inspect
loggetta plan Qwen/Qwen3-30B-A3B --seq 2048
```

[Get started on GitHub](https://github.com/pjordanandrsn/loggetta) · [PyPI 0.7.0](https://pypi.org/project/loggetta/0.7.0/)

 

## A result you can check

 

Unsloth spends 1.92× e4b's GPU time per training step on Qwen3-30B-A3B / RTX 5090, and 2.80× its wall-clock time on an AMD EPYC 7713 host. Matched adapters, initialization, tokens, and software; comparable held-out loss. Unsloth peaked lower (24.27 vs 26.16 GB).

[Read the benchmark](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/bench/h2h-2026-10-02/tc1/RESULTS-tc1-pos69.md) · [Latest results and earlier readings](https://cerinamroth.com/research/#update-2026-10-08)

 

GPU time is the host-independent figure: Unsloth spends more of each step on the CPU, so its wall-clock ratio grows on a slower host. Measured on torch 2.12.1+cu130 / transformers 5.5.0.

 

## What you can use today

 

Loggetta 0.7.0 plans and runs supported single-GPU MoE training on your own data, and keeps reusable adapters ([guide](https://github.com/pjordanandrsn/loggetta/blob/v0.7.0/docs/TRAINING.md)). It also plans serving placement; server launch uses the runtime separately.

 

### Check your model

 

Start with the [supported and tested model matrix](https://github.com/pjordanandrsn/loggetta/blob/v0.7.0/PYPI.md#models). Memory estimates are checked against real runs; they are estimates, not a guarantee that every workload fits.

 

## One install, three projects

 

 | Project | Job 
 | [Loggetta](https://github.com/pjordanandrsn/loggetta) | Inspect, plan, run, and save reports. 
 | [experts4bit-qlora](https://cerinamroth.com/ml/experts4bit-qlora/) | Load, fine-tune, serve, and offload model weights. 
 | [grouped-nf4-gemm](https://cerinamroth.com/ml/grouped-nf4-gemm/) | Compute on packed weights and move expert data. 

 

## Contributions upstream

 

Unsloth Zoo merged [PR #852](https://github.com/unslothai/unsloth-zoo/pull/852) on 15 October 2026. It adds opt-in `mask_out_tokens` support to response-only training: chosen token sequences, such as `</think>`, are excluded from loss without altering default behavior. The change covers list/tensor data, vision collators, and MLX delegation, with regression tests. This records an upstream merge, not a packaged-release claim.

 

## Choose your next step

 
- [Plan and run training](https://github.com/pjordanandrsn/loggetta)
- [Solve a memory, training, or kernel problem](https://cerinamroth.com/ml/solutions/)
- [Check model and hardware requirements](https://cerinamroth.com/ml/compatibility/)
- [Discuss an integration or research project](https://jordananderson.work/#contact)

 

Versions, evidence, and source details

 

`experts4bit-qlora` [v0.58.1 · current status →](https://cerinamroth.com/ml/status/#experts4bit-qlora) — claims tiered in the public register: 465 measured · 8 measured-private · 3 open · 19 superseded · 9 retired. `grouped-nf4-gemm` [v0.51.0 · current status →](https://cerinamroth.com/ml/status/#grouped-nf4-gemm) — claims tiered in the public register: 11 confirmed · 36 measured · 3 measured-private · 1 projected · 2 open · 1 superseded · 4 retired.

### Evidence vocabulary

confirmeda public receipt plus an independent confirmation cell (kernel register)verifieda public receipt plus an independent check (runtime register)measuredthe run happened and the receipt is public in the repositorymeasured-privatethe run happened and the receipt lives outside the repository; NOT publicly reproducible from the repository aloneprojectedarithmetic from measured inputs, not a runopena statement the documents make that has no evidence either way yet; it backs no capabilitysupersededreplaced by a later claim that names it; kept as the record, never currentretiredwithdrawn; kept as the record, never current

 

Runtime/kernel compatibility: [`experts4bit-qlora >=0.55.0 → grouped-nf4-gemm >=0.43.0 (extra: fast)`](https://cerinamroth.com/ml/compatibility/#package-compatibility).

 

### Task guides

 | Problem | Package
 | [Why does `load_in_4bit` still OOM on a Mixture-of-Experts model?](https://cerinamroth.com/ml/solutions/bitsandbytes-moe-load-in-4bit-still-ooms/) | `experts4bit-qlora`
 | [How do I train and serve MoE models released in MXFP4 (gpt-oss, DeepSeek-V4)?](https://cerinamroth.com/ml/solutions/mxfp4-moe-training-and-residency/) | `experts4bit-qlora`
 | [How do I offload MoE experts to host RAM, or serve and train them from an NVMe arena?](https://cerinamroth.com/ml/solutions/offload-moe-experts-to-cpu-or-nvme/) | `experts4bit-qlora`
 | [How do I QLoRA-train the fused experts of a MoE (per-expert LoRA on 4-bit experts)?](https://cerinamroth.com/ml/solutions/qlora-fused-moe-experts/) | `experts4bit-qlora`
 | [How can I run a Mixture-of-Experts model larger than my GPU's VRAM?](https://cerinamroth.com/ml/solutions/run-moe-larger-than-vram/) | `experts4bit-qlora`
 | [How do I serve a large MoE on a consumer GPU (RTX 5090 class)?](https://cerinamroth.com/ml/solutions/serve-large-moe-on-a-consumer-gpu/) | `experts4bit-qlora`
 | [How do I run paged decode attention over an FP8 KV cache for a quantized MoE serving path, with sliding windows, attention sinks and a custom scale?](https://cerinamroth.com/ml/solutions/fp8-paged-attention-for-moe-serving/) | `grouped-nf4-gemm`
 | [How do I run a single-token INT4 decode GEMV over routed MoE experts, and pack calibrated (GPTQ) weights for it?](https://cerinamroth.com/ml/solutions/int4-decode-gemv/) | `grouped-nf4-gemm`
 | [How do I run MoE expert inference natively on MXFP4 (e2m1 + e8m0) weights, straight from the released checkpoint bytes?](https://cerinamroth.com/ml/solutions/native-mxfp4-moe-inference/) | `grouped-nf4-gemm`
 | [How do I run a grouped GEMM directly on NF4 packed MoE expert weights, without dequantizing to bf16 first?](https://cerinamroth.com/ml/solutions/nf4-grouped-gemm-without-bf16-materialization/) | `grouped-nf4-gemm`
 | [How do I run a MoE whose experts do not fit in VRAM, streaming them from pinned host RAM or an NVMe arena?](https://cerinamroth.com/ml/solutions/stream-moe-experts-from-host-or-nvme/) | `grouped-nf4-gemm`
 | [How do I verify that the quantized expert bytes I serve or train on are the released checkpoint bytes?](https://cerinamroth.com/ml/solutions/verify-quantized-checkpoint-provenance/) | `grouped-nf4-gemm`

 

## Data and documentation

 

[Site index](https://cerinamroth.com/site.json) · [Package data](https://cerinamroth.com/ml/packages.json) · [Compatibility manifest](https://cerinamroth.com/ml/system-manifest.json) · [Source record](https://cerinamroth.com/ml/system-manifest.provenance.json) · [Latest research data](https://cerinamroth.com/research/latest.json) · [AI reading guide](https://cerinamroth.com/ml/llms.txt)

 

[OpenVINO upstream fixes](https://cerinamroth.com/ml/openvino/) · [Research notes](https://cerinamroth.com/research/) · [Hugging Face](https://huggingface.co/pjordanandrsn)

 

rendered from experts4bit-qlora 0.58.1 · source: [pjordanandrsn/experts4bit-qlora@605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) · checked 2026-10-25

rendered from grouped-nf4-gemm 0.51.0 · source: [pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) · checked 2026-10-25

Sources and publication details

The generated ML documentation surfaces on this site (`/ml/` and its sub-pages, `/ml/packages.json`, both `llms.txt` files) are derived from the repositories below at pinned commits; each repository's `docs/claims.json` at that commit is the only source for numbers. Pages with their own source are listed separately.

- [pjordanandrsn/experts4bit-qlora@605d21e67958](https://github.com/pjordanandrsn/experts4bit-qlora/commit/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) (2026-10-24) · package `experts4bit-qlora` 0.58.1 · PyPI 0.58.1 (published) · documents: [`docs/capabilities.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/capabilities.json), [`docs/claims.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/claims.json), [`docs/solutions/bitsandbytes-moe-load-in-4bit-still-ooms.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/bitsandbytes-moe-load-in-4bit-still-ooms.md), [`docs/solutions/mxfp4-moe-training-and-residency.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/mxfp4-moe-training-and-residency.md), [`docs/solutions/offload-moe-experts-to-cpu-or-nvme.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/offload-moe-experts-to-cpu-or-nvme.md), [`docs/solutions/qlora-fused-moe-experts.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/qlora-fused-moe-experts.md), [`docs/solutions/run-moe-larger-than-vram.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/run-moe-larger-than-vram.md), [`docs/solutions/serve-large-moe-on-a-consumer-gpu.md`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/solutions/serve-large-moe-on-a-consumer-gpu.md), [`docs/system-manifest.json`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/docs/system-manifest.json), [`pyproject.toml`](https://github.com/pjordanandrsn/experts4bit-qlora/blob/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d/pyproject.toml) · registers: capabilities, claims · synced 2026-10-25T00:58:32Z · [files at this commit](https://github.com/pjordanandrsn/experts4bit-qlora/tree/605d21e67958fa94d5e73fc3a95ffcb5c8eae02d) · [latest on main](https://github.com/pjordanandrsn/experts4bit-qlora) (unpinned; not the source of any fact rendered here)
- [pjordanandrsn/grouped-nf4-gemm@30adb1bfa76f](https://github.com/pjordanandrsn/grouped-nf4-gemm/commit/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) (2026-10-24) · package `grouped-nf4-gemm` 0.51.0 · PyPI 0.51.0 (published) · documents: [`docs/capabilities.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/capabilities.json), [`docs/claims.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/claims.json), [`docs/solutions/fp8-paged-attention-for-moe-serving.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/fp8-paged-attention-for-moe-serving.md), [`docs/solutions/int4-decode-gemv.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/int4-decode-gemv.md), [`docs/solutions/native-mxfp4-moe-inference.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/native-mxfp4-moe-inference.md), [`docs/solutions/nf4-grouped-gemm-without-bf16-materialization.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/nf4-grouped-gemm-without-bf16-materialization.md), [`docs/solutions/stream-moe-experts-from-host-or-nvme.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/stream-moe-experts-from-host-or-nvme.md), [`docs/solutions/verify-quantized-checkpoint-provenance.md`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/solutions/verify-quantized-checkpoint-provenance.md), [`docs/system-manifest.json`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/docs/system-manifest.json), [`pyproject.toml`](https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/30adb1bfa76f95f9ce4a4c29734b8f36de907eca/pyproject.toml) · registers: capabilities, claims · synced 2026-10-24T14:51:49Z · [files at this commit](https://github.com/pjordanandrsn/grouped-nf4-gemm/tree/30adb1bfa76f95f9ce4a4c29734b8f36de907eca) · [latest on main](https://github.com/pjordanandrsn/grouped-nf4-gemm) (unpinned; not the source of any fact rendered here)
- External versioned source: [Loggetta 0.7.0](https://pypi.org/project/loggetta/0.7.0/) · sections: `loggetta-front-door`, `models-supported-vs-tested` — the install and model matrix are quoted from the PyPI 0.7.0 release (tag v0.7.0 @ 3bbf7ab0d5ac); performance ratios below come from the pinned e4b claim register.
- OpenVINO page ([/ml/openvino/](https://cerinamroth.com/ml/openvino/), separate source): [pjordanandrsn/ov-impact-bench@810ec5f49a9d](https://github.com/pjordanandrsn/ov-impact-bench/commit/810ec5f49a9d6e0f80010999dd29fa25dab93104) (2026-05-24) · documents: `results.jsonl`, `README.md` · fetched 2026-09-04T15:18:26Z · generated by `scripts/build_openvino.py`.
- Site-authored (`scripts/build_ml_site.py` in Cerin-Amroth/cerinamroth-site, private): `start-with-the-need`, `released-and-research`, `upstream-work` — site-authored navigation and summary; the merged Unsloth Zoo contribution is linked directly to its upstream pull request.