ML systems
Fitting and serving large Mixture-of-Experts LLMs on accessible GPUs — and proving the performance claims that come with them.
MoE won the open-weights frontier and outgrew the hardware most people own: the models activate few parameters per token but must hold every expert resident, so a 30B-class MoE wants 20+ GB for weights alone against the 8–16 GB that ships on accessible NVIDIA cards. This practice attacks that gap from two sides — making the weights fit for fine-tuning and serving, and making the 4-bit compute path fast — and holds both to the same evidence standard used on the security side: claims registered and cryptographically timestamped before the data exists, failures published at the same volume as wins. The same goal — real inference on the hardware a person already owns — extends to Intel GPUs through upstream work on the OpenVINO toolkit.
Projects
- experts-4bit
4-bit quantization of fused MoE expert weights, composable with the bitsandbytes stack. Recompute-in-backward keeps fine-tuning memory flat as expert count grows, so a single 12 GB card can both adapt and serve a 30B-class MoE. Upstream-first as bitsandbytes PR #1965.
- grouped-nf4-gemm
The expert GEMM run directly on the 4-bit weights — one launch, decode inside the mainloop, no dequantize-to-bf16 round trip. Blind-confirmed under preregistered, OpenTimestamps-stamped protocols; the methodology writeup — six confirmatories, five published failures — is the receipts-driven-engineering case study.
- OpenVINO
The same goal aimed at Intel hardware: upstream fixes to Intel's inference toolkit that keep Intel-GPU LLM inference off the silent CPU-fallback path on modern drivers. Three merged pull requests and one in review; impact measured with a purpose-built harness.