# mit-han-lab/hart architecture

> Architecture diagram of the GitHub repository mit-han-lab/hart, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mit-han-lab/hart
- Repository: https://github.com/mit-han-lab/hart

## Overview

##### Main components
- Application Entry Points: Highlight the main executable files like app.py (for launching the Gradio demo), sample.py (for command-line inference with single or multiple prompts), and latency_profile.py for benchmarking.
- Core Model Modules: In the “hart/modules” directory, separate out the major submodules:
– Autoencoder: Found under “modules/models/autoencoder” with its configuration and quantization logic.
– Transformer: In “modules/models/transformer”, which includes the autoregressive text-to-image generation logic.
– Diffusion Module: Under “modules/diffusion” which implements the lightweight residual diffusion to refine continuous components.
– Losses & Networks: Including VQGAN-style losses (“modules/losses/vqgan”), and supporting network architectures (“modules/networks”).
- Kernels Layer: Under “hart/kernels”, this layer contains CUDA/C++ implementations (e.g., for layer normalization and fused rope operations) and Python bindings (pybind.cpp) that accelerate various computations.
- Utility Components: For miscellaneous functionalities including safety checks, random utilities, and constants (spread over “hart/utils” and “modules/utils”).

##### Relationships and interactions
- The Application Layer (app.py, sample.py, latency_profile.py) accepts inputs (prompts) and interacts with the model components.
- Input Prompts are processed through the text model (external dependency) and then passed into the hybrid transformer architecture.
- The Autoencoder provides a latent representation where the hybrid tokenizer decomposes the input into discrete (big picture) and continuous (residual) tokens.
- The Discrete Part is modeled with the autoregressive Transformer (from “modules/models/transformer”) while the Continuous Part is refined via a diffusion module.
- The high-performance kernels (CUDA implementations in “hart/kernels”) underpin the efficient execution of key operations like layer normalization and token handling, ensuring low MACs and high throughput.
- Utility modules support overall system coordination, safety checking, and additional logic useful in pre- or post-processing.

##### Architectural patterns and design principles
- Modular Design: The codebase is clearly segmented into layers – inference/application, core model logic, and compute kernels – which enables clear abstraction layers.
- Hybrid Modeling: The key research innovation is in the hybrid tokenizer and dual-path model (AR for discrete components, diffusion for continuous).
- Acceleration: The separation of CUDA-based low-level operations (housed in the “kernels” directory) from the high-level model logic shows a design that emphasizes performance.
- Extensibility: With separate modules for diffusion, losses, networks, and utility functions, the architecture supports further experimentation and extension.

##### Additional detailed points
- How the system ensures scalability by decoupling critical processes (model components and kernel operations).
- Optionally, mark external dependencies that must be configured during setup (e.g., paths to models, safety check module).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and Application Layer
    U("User Input"):::external

    subgraph "Application Entry Points"
        A1("Gradio Demo (app.py)"):::app
        A2("Command-Line Inference (sample.py)"):::app
        A3("Benchmarking (latency_profile.py)"):::app
    end

    U --> A1
    U --> A2
    U --> A3

    %% External Text Model Dependency
    TM("Text Model (Qwen2-VL-1.5B-Instruct)"):::external
    A1 --> TM
    A2 --> TM
    A3 --> TM

    %% Core Model Modules Subgraph
    subgraph "Core Model Modules"
        AE("Autoencoder"):::core
        VN("VQGAN-style Losses"):::core
        SN("Supporting Networks"):::core

        %% Branching into Discrete and Continuous Pipelines
        subgraph "Hybrid Tokenization"
            TR("Discrete Path: Transformer Module"):::core
            DF("Continuous Path: Diffusion Module"):::core
        end
        
        %% CUDA Kernels for Acceleration
        CK("CUDA Kernels"):::kernel
        CO("Combine Outputs"):::core
    end

    TM --> AE
    AE --> TR
    AE --> DF

    %% Support modules linked to Autoencoder
    AE --- VN
    AE --- SN

    %% Pipeline acceleration using CUDA Kernels
    TR --> CK
    DF --> CK
    CK --> CO

    %% Utility Components supporting operations
    subgraph "Utility Components"
        U1("General Utils (hart/utils)"):::utility
        U2("Module-specific Utils (modules/utils)"):::utility
    end
    A1 --- U1
    A2 --- U1
    A3 --- U1
    AE --- U2
    TR --- U2
    DF --- U2

    %% External Safety Filter dependency
    SF("Safety Filter (ShieldGemma-2B)"):::external
    CO --> SF

    %% Final Output
    O("High-Resolution Image Output"):::external
    SF --> O

    %% Click Events for Application Entry Points
    click A1 "https://github.com/mit-han-lab/hart/blob/main/app.py"
    click A2 "https://github.com/mit-han-lab/hart/blob/main/sample.py"
    click A3 "https://github.com/mit-han-lab/hart/blob/main/latency_profile.py"

    %% Click Events for Core Model Modules
    click AE "https://github.com/mit-han-lab/hart/tree/main/hart/modules/models/autoencoder"
    click TR "https://github.com/mit-han-lab/hart/tree/main/hart/modules/models/transformer"
    click DF "https://github.com/mit-han-lab/hart/tree/main/hart/modules/diffusion"
    click VN "https://github.com/mit-han-lab/hart/tree/main/hart/modules/losses/vqgan"
    click SN "https://github.com/mit-han-lab/hart/tree/main/hart/modules/networks"

    %% Click Event for Kernels Layer
    click CK "https://github.com/mit-han-lab/hart/tree/main/hart/kernels"

    %% Click Events for Utility Components
    click U1 "https://github.com/mit-han-lab/hart/tree/main/hart/utils"
    click U2 "https://github.com/mit-han-lab/hart/tree/main/hart/modules/utils"

    %% Styling classes
    classDef app fill:#a2d9ce,stroke:#2ecc71,stroke-width:2px;
    classDef core fill:#f7dc6f,stroke:#f1c40f,stroke-width:2px;
    classDef kernel fill:#e6b0aa,stroke:#c0392b,stroke-width:2px;
    classDef external fill:#d6eaf8,stroke:#3498db,stroke-width:2px;
    classDef utility fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
```
