# leanmodels/dfloat11 architecture

> Architecture diagram of the GitHub repository leanmodels/dfloat11, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leanmodels/dfloat11
- Repository: https://github.com/leanmodels/dfloat11

## Overview

##### Project type and purpose
- This is a Python library + CLI tool for lossless compression of LLM weights (“DFloat11”) and fast GPU inference.
- It integrates with Hugging Face Transformers and Accelerate to load a standard BFloat16 model and a corresponding compressed DFloat11 model, then runs causal-LM generation on CUDA devices.

##### File structure
- Top level:
- `setup.py` – package installation
- `inference.py` – CLI entry point for inference
- `dfloat11/` – the core library
– `modeling_dfloat11_llama.py` – defines `DFloat11ModelForCausalLM`, weight-loading logic, and custom CUDA kernel hooks
- No microservices or separate frontend; this is a standalone tool/library.
- Dependencies (in README): PyTorch, Hugging Face Transformers & Hub, Accelerate, CUDA.

##### README for additional insights
- Describes a single-process pipeline: download model weights, instantiate a Hugging Face model wrapped by DFloat11 loader, dispatch to GPU(s) via Accelerate, run generate loops.
- Highlights the compression ratio (~30%), bit-identical outputs, and custom GPU kernel for decompression & forward pass.

## Diagram (Mermaid)

```mermaid
flowchart TB
  subgraph "User Process"
    CLI["inference.py<br/>CLI"]:::cli
  end

  subgraph "External Services"
    HF["Hugging Face Hub"]:::external
  end

  subgraph "Local System"
    Cache["Local File Cache"]:::external
    Loader["modeling_dfloat11_llama.py<br/>DFloat11 Loader"]:::library
    HFModel["BFloat16 Model<br/>(Transformers)"]:::library
    DFWeights["Compressed DFloat11 Weights"]:::library
    CUDAKernel["Custom DFloat11 CUDA Kernel"]:::kernel
    PyTorchEngine["PyTorch + Accelerate"]:::library
    GPU["GPU / CUDA"]:::hardware
    Metrics["Output & Metrics"]:::library
  end

  subgraph "Project Files"
    Setup["setup.py"]:::config
    README["README.md"]:::config
    License["LICENSE"]:::config
    Gitignore[".gitignore"]:::config
    Init["__init__.py"]:::library
  end

  CLI -->|"model download"| HF
  HF -->|"weights"| Cache
  CLI -->|"load"| Loader
  Loader --> HFModel
  Loader --> DFWeights
  Loader --> CUDAKernel
  HFModel --> PyTorchEngine
  DFWeights --> CUDAKernel
  CUDAKernel --> GPU
  PyTorchEngine --> GPU
  GPU --> PyTorchEngine
  PyTorchEngine --> Metrics
  Metrics --> CLI

  click CLI "https://github.com/leanmodels/dfloat11/blob/master/inference.py"
  click Loader "https://github.com/leanmodels/dfloat11/blob/master/dfloat11/modeling_dfloat11_llama.py"
  click Init "https://github.com/leanmodels/dfloat11/blob/master/dfloat11/__init__.py"
  click Setup "https://github.com/leanmodels/dfloat11/blob/master/setup.py"
  click README "https://github.com/leanmodels/dfloat11/blob/master/README.md"
  click License "https://github.com/leanmodels/dfloat11/tree/master/LICENSE"
  click Gitignore "https://github.com/leanmodels/dfloat11/blob/master/.gitignore"

  classDef external fill:#D0E8FF,stroke:#0066CC;
  classDef library fill:#E8FFE8,stroke:#009900;
  classDef hardware fill:#FFE8D0,stroke:#CC6600;
  classDef kernel fill:#FFF0E8,stroke:#CC4400;
  classDef cli fill:#E0E0E0,stroke:#888888;
  classDef config fill:#F0F0F0,stroke:#BBBBBB;
```
