openbitsys/bitdecoding-fp4

How openbitsys/bitdecoding-fp4 works

BitDecoding-FP4 is an NVFP4 KV-cache quantization and native FP4 attention system integrated as a custom vLLM backend. In serving, vLLM builds the engine and backend; per-step metadata, shared runtime state, and cache writers manage the hybrid cache, while the decode interface invokes the attention kernel. Calibration can provide key smoothing and probability settings. Separately, the evaluation runner selects cache modes, runs benchmark adapters, and scores results. Connections below follow README claims and the supplied source index or excerpts; unsampled implementation details are not inferred.

Main entry points: engine.py Engine setup · runtime.py Shared GPU runtime · run_eval.py Evaluation runner · builder.py Metadata builder

Components 22

vLLM Serving

Engine setup src/nvfp4_vllm/engine.py
Custom backend src/nvfp4_vllm/backend.py
Metadata builder src/nvfp4_vllm/builder.py
Step metadata src/nvfp4_vllm/metadata.py
Tail control plane src/nvfp4_vllm/control.py
Attention implementation src/nvfp4_vllm/impl.py
Shared GPU runtime src/nvfp4_vllm/runtime.py
KV cache writer src/nvfp4_vllm/write.py

FP4 Decode Kernels

Decode interface src/nvfp4_decode_kernel/interface.py
Paged attention kernel src/nvfp4_decode_kernel/fp4_decode_kernel.py
KV quantization src/nvfp4_decode_kernel/quantize_kv_kernel.py
Block scaling src/nvfp4_decode_kernel/_block_scale.py
Scale modes src/nvfp4_decode_kernel/scale_mode.py

Calibration

Key calibration src/nvfp4_vllm/calibrate.py
Probability calibration src/nvfp4_vllm/probability_calibration.py

Evaluation

Evaluation runner evaluation/run_eval.py
Mode configuration evaluation/config.py
vLLM evaluation wrapper evaluation/vllm_runner.py
Benchmark adapters evaluation
Scoring metrics evaluation/scoring.py

Other components

Serving user
vLLM engine

Connections 32

Serving user → Engine setup: configures README.md
Engine setup → vLLM engine: builds engine src/nvfp4_vllm/engine.py
Engine setup → Custom backend: registers src/nvfp4_vllm/engine.py
Engine setup → Key calibration: calibrates keys src/nvfp4_vllm/engine.py
Engine setup → Tail control plane: sets slot limits src/nvfp4_vllm/engine.py
Engine setup → Shared GPU runtime: configures runtime src/nvfp4_vllm/engine.py
Custom backend → Metadata builder: creates metadata
Metadata builder → Tail control plane: uses src/nvfp4_vllm/builder.py
Metadata builder → Step metadata: uses src/nvfp4_vllm/builder.py
Metadata builder → KV cache writer: uses src/nvfp4_vllm/builder.py
Attention implementation → Decode interface: calls src/nvfp4_vllm/impl.py
Attention implementation → Shared GPU runtime: uses src/nvfp4_vllm/impl.py
Attention implementation → KV cache writer: uses src/nvfp4_vllm/impl.py
Shared GPU runtime → Tail control plane: uses src/nvfp4_vllm/runtime.py
Shared GPU runtime → KV quantization: uses src/nvfp4_vllm/runtime.py
KV cache writer → KV quantization: quantizes pages src/nvfp4_vllm/write.py
KV cache writer → Tail control plane: uses src/nvfp4_vllm/write.py
Decode interface → Paged attention kernel: dispatches src/nvfp4_decode_kernel/interface.py
KV quantization → Block scaling: uses src/nvfp4_decode_kernel/quantize_kv_kernel.py
KV quantization → Scale modes: specializes scales src/nvfp4_decode_kernel/quantize_kv_kernel.py
Paged attention kernel → Block scaling: uses src/nvfp4_decode_kernel/fp4_decode_kernel.py
Paged attention kernel → Scale modes: uses src/nvfp4_decode_kernel/fp4_decode_kernel.py
Key calibration → Probability calibration: installs collector src/nvfp4_vllm/calibrate.py
Key calibration → Shared GPU runtime: updates runtime src/nvfp4_vllm/calibrate.py
Evaluation runner → Mode configuration: selects modes evaluation/run_eval.py
Evaluation runner → vLLM evaluation wrapper: runs models evaluation/run_eval.py
Evaluation runner → Benchmark adapters: dispatches benchmarks evaluation/run_eval.py
Evaluation runner → Scoring metrics: scores results evaluation/run_eval.py
vLLM evaluation wrapper → Decode interface: inspects mode evaluation/vllm_runner.py
vLLM evaluation wrapper → Key calibration: uses extension evaluation/vllm_runner.py
Benchmark adapters → Scoring metrics: scores tasks evaluation/lcb.py
Benchmark adapters → Mode configuration: loads settings evaluation/lcb.py

Drawn by GitDiagram on November 12, 2026 from the repository's file tree, README and source files. Read as Markdown ·