{
  "title": "How do I run paged decode attention over an FP8 KV cache for a quantized MoE serving path, with sliding windows, attention sinks and a custom scale?",
  "summary": "fp8_paged_decode_attention runs paged flash-decode over an E4M3 KV cache with windows, sinks and a custom scale; the fp8 compute path is supported on sm_89+, the f32 path is open under #319.",
  "summary_origin": "comment",
  "project": "grouped-nf4-gemm",
  "canonical_package": "grouped-nf4-gemm",
  "canonical_url": "https://cerinamroth.com/ml/solutions/fp8-paged-attention-for-moe-serving/",
  "install": {
    "primary": {
      "purpose": "kernel package",
      "command": "pip install grouped-nf4-gemm",
      "capabilities": [
        "fp8-paged-attention-fp8-compute",
        "fp8-paged-attention-f32-compute",
        "decode-glue-kernels"
      ]
    },
    "alternatives": [
      {
        "purpose": "through the model consumer",
        "command": "pip install \"experts4bit-qlora[fast]\"",
        "capabilities": [
          "fp8-paged-attention-fp8-compute",
          "fp8-paged-attention-f32-compute",
          "decode-glue-kernels"
        ]
      }
    ],
    "environment": {
      "os": [
        "Linux"
      ],
      "python": [
        ">=3.11 tested in CI (pyproject says >=3.9; 3.9/3.10 are not tested)"
      ],
      "accelerator": [
        "NVIDIA CUDA GPU, sm_80 or newer (the composition was measured on the RTX 5090, sm_120)",
        "NVIDIA CUDA GPU, sm_80 or newer; this is the path sm_80-sm_88 take by default. Its reference tests were run on an RTX 5090 (sm_120) with the mode forced; no registered cell exists on sm_80-sm_88",
        "NVIDIA CUDA GPU, sm_89 or newer (the fp8 tensor-core dot); measured on the RTX 5090 (sm_120) only -- sm_89 and sm_90 meet the requirement but no registered cell was run there"
      ],
      "special_requirements": [
        "shape constraints, all from fp8_paged_attn.fp8_compute_unsupported (the single predicate the default selector and the path's own asserts share): v_groups == 1; k_groups in (1, 2, 4, 8, 16); head_dim // k_groups >= 32; q in bf16 or fp16; split branch: ktile >= 32 when the caller supplies it; packed branch (pack_heads=True): block_tokens * n_kv_heads >= 32. A call that fails any of these with GNF4_ATTN_COMPUTE unset takes the f32 path (its own entry)",
        "the open item is on torch 2.8.0+cu128 / triton 3.4.0 -- the package floor -- so no torch/Triton pair inside the supported range is registered as passing this path's reference tests",
        "torch>=2.8 (pre-releases accepted)",
        "triton>=3.4 (Linux-only distribution)"
      ]
    }
  },
  "source": {
    "repository": "https://github.com/pjordanandrsn/grouped-nf4-gemm",
    "commit": "e2af4cfb91b2c29968989c5254bff55aa7da3347",
    "path": "docs/solutions/fp8-paged-attention-for-moe-serving.md",
    "sha256": "375b44c1e48a65fc101d207306cda4a4551a18d3ee3e23bcee6a40593c775f38",
    "generation_schema": "1.2"
  },
  "pinned_url": "https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/solutions/fp8-paged-attention-for-moe-serving.md",
  "current_main_url": "https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/main/docs/solutions/fp8-paged-attention-for-moe-serving.md",
  "current_main_label": "latest on main — unpinned; not the source of any fact rendered here",
  "freshness": {
    "source": {
      "repository": "pjordanandrsn/grouped-nf4-gemm",
      "commit": "e2af4cfb91b2c29968989c5254bff55aa7da3347",
      "version": "0.30.2"
    },
    "publication": {
      "latest_pypi_version": "0.30.2",
      "fresh": true,
      "state": "CURRENT",
      "checked_at": "2026-09-06T04:07:55Z"
    },
    "generated_at": "2026-09-06T04:07:55Z"
  },
  "markdown": "https://cerinamroth.com/ml/solutions/fp8-paged-attention-for-moe-serving/index.md",
  "sources": [
    {
      "repository": "https://github.com/pjordanandrsn/grouped-nf4-gemm",
      "commit": "e2af4cfb91b2c29968989c5254bff55aa7da3347",
      "commit_date": "2026-09-05",
      "package": "grouped-nf4-gemm",
      "package_version": "0.30.2",
      "published_version": "0.30.2",
      "publication": "published",
      "documents_used": [
        "docs/capabilities.json",
        "docs/solutions/fp8-paged-attention-for-moe-serving.md"
      ],
      "registers_used": [
        "capabilities"
      ],
      "synced_at": "2026-09-06T00:07:26Z",
      "pinned_tree": "https://github.com/pjordanandrsn/grouped-nf4-gemm/tree/e2af4cfb91b2c29968989c5254bff55aa7da3347",
      "current_main": "https://github.com/pjordanandrsn/grouped-nf4-gemm",
      "current_main_label": "latest on main — unpinned; not the source of any fact rendered here",
      "system_manifest": {
        "url": "https://cerinamroth.com/ml/system-manifest.json",
        "provenance_url": "https://cerinamroth.com/ml/system-manifest.provenance.json",
        "sha256": "196cb55a41e890842f258305f307b873d55639cc617cd4ae61c771a063204918",
        "pinned": "https://github.com/pjordanandrsn/grouped-nf4-gemm/blob/e2af4cfb91b2c29968989c5254bff55aa7da3347/docs/system-manifest.json"
      }
    },
    {
      "repository": "https://github.com/Cerin-Amroth/cerinamroth-site",
      "commit": null,
      "package_version": null,
      "documents_used": [
        "scripts/build_ml_site.py"
      ],
      "registers_used": [],
      "synced_at": null,
      "sections_owned": [
        "common-wrong-approaches"
      ],
      "kind": "site",
      "note": "site-authored prose; cites the registers above and copies no numbers"
    }
  ]
}