# openbitsys/bitdecoding-fp4 architecture

> Architecture diagram of the GitHub repository openbitsys/bitdecoding-fp4, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openbitsys/bitdecoding-fp4
- Repository: https://github.com/openbitsys/bitdecoding-fp4

## Overview

BitDecoding-FP4 is an NVFP4 KV-cache quantization and native FP4 attention system integrated as a custom vLLM backend. In serving, vLLM builds the engine and backend; per-step metadata, shared runtime state, and cache writers manage the hybrid cache, while the decode interface invokes the attention kernel. Calibration can provide key smoothing and probability settings. Separately, the evaluation runner selects cache modes, runs benchmark adapters, and scores results. Connections below follow README claims and the supplied source index or excerpts; unsampled implementation details are not inferred.

Main entry points: [`src/nvfp4_vllm/engine.py`](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/engine.py) (Engine setup), [`src/nvfp4_vllm/runtime.py`](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/runtime.py) (Shared GPU runtime), [`evaluation/run_eval.py`](https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/run_eval.py) (Evaluation runner), [`src/nvfp4_vllm/builder.py`](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/builder.py) (Metadata builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["vLLM Serving"]
  node_engine["Engine setup<br/>[engine.py]"]
  node_backend["Custom backend<br/>[backend.py]"]
  node_builder["Metadata builder<br/>[builder.py]"]
  node_metadata["Step metadata<br/>[metadata.py]"]
  node_control["Tail control plane<br/>[control.py]"]
  node_impl["Attention implementation<br/>[impl.py]"]
  node_runtime["Shared GPU runtime<br/>[runtime.py]"]
  node_writer["KV cache writer<br/>[write.py]"]
end

subgraph group_kernel["FP4 Decode Kernels"]
  node_decode_api["Decode interface<br/>[interface.py]"]
  node_decode_kernel["Paged attention kernel"]
  node_kv_quantizer["KV quantization"]
  node_block_scale["Block scaling<br/>[_block_scale.py]"]
  node_scale_modes["Scale modes<br/>[scale_mode.py]"]
end

subgraph group_calibration["Calibration"]
  node_calibrator["Key calibration<br/>[calibrate.py]"]
  node_prob_cal["Probability calibration"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_runner["Evaluation runner<br/>[run_eval.py]"]
  node_eval_config["Mode configuration<br/>[config.py]"]
  node_runner["vLLM evaluation wrapper<br/>[vllm_runner.py]"]
  node_benchmarks["Benchmark adapters"]
  node_scoring["Scoring metrics<br/>[scoring.py]"]
end

node_user(("Serving user"))
node_vllm["vLLM engine"]

node_user -->|"configures"| node_engine
node_engine -->|"builds engine"| node_vllm
node_engine -->|"registers"| node_backend
node_engine -->|"calibrates keys"| node_calibrator
node_engine -->|"sets slot limits"| node_control
node_engine -->|"configures runtime"| node_runtime
node_backend -.->|"creates metadata"| node_builder
node_builder -->|"uses"| node_control
node_builder -->|"uses"| node_metadata
node_builder -->|"uses"| node_writer
node_impl -->|"calls"| node_decode_api
node_impl -->|"uses"| node_runtime
node_impl -->|"uses"| node_writer
node_runtime -->|"uses"| node_control
node_runtime -->|"uses"| node_kv_quantizer
node_writer -->|"quantizes pages"| node_kv_quantizer
node_writer -->|"uses"| node_control
node_decode_api -->|"dispatches"| node_decode_kernel
node_kv_quantizer -->|"uses"| node_block_scale
node_kv_quantizer -->|"specializes scales"| node_scale_modes
node_decode_kernel -->|"uses"| node_block_scale
node_decode_kernel -->|"uses"| node_scale_modes
node_calibrator -->|"installs collector"| node_prob_cal
node_calibrator -->|"updates runtime"| node_runtime
node_eval_runner -->|"selects modes"| node_eval_config
node_eval_runner -->|"runs models"| node_runner
node_eval_runner -->|"dispatches benchmarks"| node_benchmarks
node_eval_runner -->|"scores results"| node_scoring
node_runner -->|"inspects mode"| node_decode_api
node_runner -->|"uses extension"| node_calibrator
node_benchmarks -->|"scores tasks"| node_scoring
node_benchmarks -->|"loads settings"| node_eval_config

click node_engine "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/engine.py"
click node_backend "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/backend.py"
click node_builder "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/builder.py"
click node_metadata "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/metadata.py"
click node_control "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/control.py"
click node_impl "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/impl.py"
click node_runtime "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/runtime.py"
click node_writer "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/write.py"
click node_decode_api "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/interface.py"
click node_decode_kernel "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/fp4_decode_kernel.py"
click node_kv_quantizer "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/quantize_kv_kernel.py"
click node_block_scale "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/_block_scale.py"
click node_scale_modes "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/scale_mode.py"
click node_calibrator "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/calibrate.py"
click node_prob_cal "https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/probability_calibration.py"
click node_eval_runner "https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/run_eval.py"
click node_eval_config "https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/config.py"
click node_runner "https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/vllm_runner.py"
click node_benchmarks "https://github.com/openbitsys/bitdecoding-fp4/tree/main/evaluation"
click node_scoring "https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/scoring.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_engine,node_backend,node_builder,node_metadata,node_control,node_impl,node_runtime,node_writer,node_user toneBlue
class node_decode_api,node_decode_kernel,node_kv_quantizer,node_block_scale,node_scale_modes toneAmber
class node_calibrator,node_prob_cal toneMint
class node_eval_runner,node_eval_config,node_runner,node_benchmarks,node_scoring toneRose
class node_vllm toneIndigo
```

## Components

### vLLM Serving

- [Engine setup](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/engine.py): `src/nvfp4_vllm/engine.py`
- [Custom backend](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/backend.py): `src/nvfp4_vllm/backend.py`
- [Metadata builder](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/builder.py): `src/nvfp4_vllm/builder.py`
- [Step metadata](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/metadata.py): `src/nvfp4_vllm/metadata.py`
- [Tail control plane](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/control.py): `src/nvfp4_vllm/control.py`
- [Attention implementation](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/impl.py): `src/nvfp4_vllm/impl.py`
- [Shared GPU runtime](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/runtime.py): `src/nvfp4_vllm/runtime.py`
- [KV cache writer](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/write.py): `src/nvfp4_vllm/write.py`

### FP4 Decode Kernels

- [Decode interface](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/interface.py): `src/nvfp4_decode_kernel/interface.py`
- [Paged attention kernel](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/fp4_decode_kernel.py): `src/nvfp4_decode_kernel/fp4_decode_kernel.py`
- [KV quantization](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/quantize_kv_kernel.py): `src/nvfp4_decode_kernel/quantize_kv_kernel.py`
- [Block scaling](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/_block_scale.py): `src/nvfp4_decode_kernel/_block_scale.py`
- [Scale modes](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_decode_kernel/scale_mode.py): `src/nvfp4_decode_kernel/scale_mode.py`

### Calibration

- [Key calibration](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/calibrate.py): `src/nvfp4_vllm/calibrate.py`
- [Probability calibration](https://github.com/openbitsys/bitdecoding-fp4/blob/main/src/nvfp4_vllm/probability_calibration.py): `src/nvfp4_vllm/probability_calibration.py`

### Evaluation

- [Evaluation runner](https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/run_eval.py): `evaluation/run_eval.py`
- [Mode configuration](https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/config.py): `evaluation/config.py`
- [vLLM evaluation wrapper](https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/vllm_runner.py): `evaluation/vllm_runner.py`
- [Benchmark adapters](https://github.com/openbitsys/bitdecoding-fp4/tree/main/evaluation): `evaluation`
- [Scoring metrics](https://github.com/openbitsys/bitdecoding-fp4/blob/main/evaluation/scoring.py): `evaluation/scoring.py`

### Other components

- **Serving user**
- **vLLM engine**

## Connections

- Serving user → Engine setup: configures (evidence: `README.md`)
- Engine setup → vLLM engine: builds engine (evidence: `src/nvfp4_vllm/engine.py`)
- Engine setup → Custom backend: registers (evidence: `src/nvfp4_vllm/engine.py`)
- Engine setup → Key calibration: calibrates keys (evidence: `src/nvfp4_vllm/engine.py`)
- Engine setup → Tail control plane: sets slot limits (evidence: `src/nvfp4_vllm/engine.py`)
- Engine setup → Shared GPU runtime: configures runtime (evidence: `src/nvfp4_vllm/engine.py`)
- Custom backend → Metadata builder: creates metadata
- Metadata builder → Tail control plane: uses (evidence: `src/nvfp4_vllm/builder.py`)
- Metadata builder → Step metadata: uses (evidence: `src/nvfp4_vllm/builder.py`)
- Metadata builder → KV cache writer: uses (evidence: `src/nvfp4_vllm/builder.py`)
- Attention implementation → Decode interface: calls (evidence: `src/nvfp4_vllm/impl.py`)
- Attention implementation → Shared GPU runtime: uses (evidence: `src/nvfp4_vllm/impl.py`)
- Attention implementation → KV cache writer: uses (evidence: `src/nvfp4_vllm/impl.py`)
- Shared GPU runtime → Tail control plane: uses (evidence: `src/nvfp4_vllm/runtime.py`)
- Shared GPU runtime → KV quantization: uses (evidence: `src/nvfp4_vllm/runtime.py`)
- KV cache writer → KV quantization: quantizes pages (evidence: `src/nvfp4_vllm/write.py`)
- KV cache writer → Tail control plane: uses (evidence: `src/nvfp4_vllm/write.py`)
- Decode interface → Paged attention kernel: dispatches (evidence: `src/nvfp4_decode_kernel/interface.py`)
- KV quantization → Block scaling: uses (evidence: `src/nvfp4_decode_kernel/quantize_kv_kernel.py`)
- KV quantization → Scale modes: specializes scales (evidence: `src/nvfp4_decode_kernel/quantize_kv_kernel.py`)
- Paged attention kernel → Block scaling: uses (evidence: `src/nvfp4_decode_kernel/fp4_decode_kernel.py`)
- Paged attention kernel → Scale modes: uses (evidence: `src/nvfp4_decode_kernel/fp4_decode_kernel.py`)
- Key calibration → Probability calibration: installs collector (evidence: `src/nvfp4_vllm/calibrate.py`)
- Key calibration → Shared GPU runtime: updates runtime (evidence: `src/nvfp4_vllm/calibrate.py`)
- Evaluation runner → Mode configuration: selects modes (evidence: `evaluation/run_eval.py`)
- Evaluation runner → vLLM evaluation wrapper: runs models (evidence: `evaluation/run_eval.py`)
- Evaluation runner → Benchmark adapters: dispatches benchmarks (evidence: `evaluation/run_eval.py`)
- Evaluation runner → Scoring metrics: scores results (evidence: `evaluation/run_eval.py`)
- vLLM evaluation wrapper → Decode interface: inspects mode (evidence: `evaluation/vllm_runner.py`)
- vLLM evaluation wrapper → Key calibration: uses extension (evidence: `evaluation/vllm_runner.py`)
- Benchmark adapters → Scoring metrics: scores tasks (evidence: `evaluation/lcb.py`)
- Benchmark adapters → Mode configuration: loads settings (evidence: `evaluation/lcb.py`)