How mit-han-lab/hart works
Main components
• Application Entry Points: Highlight the main executable files like app.py (for launching the Gradio demo), sample.py (for command-line inference with single or multiple prompts), and latency_profile.py for benchmarking.
• Core Model Modules: In the “hart/modules” directory, separate out the major submodules:
– Autoencoder: Found under “modules/models/autoencoder” with its configuration and quantization logic.
– Transformer: In “modules/models/transformer”, which includes the autoregressive text-to-image generation logic.
– Diffusion Module: Under “modules/diffusion” which implements the lightweight residual diffusion to refine continuous components.
– Losses & Networks: Including VQGAN-style losses (“modules/losses/vqgan”), and supporting network architectures (“modules/networks”).
• Kernels Layer: Under “hart/kernels”, this layer contains CUDA/C++ implementations (e.g., for layer normalization and fused rope operations) and Python bindings (pybind.cpp) that accelerate various computations.
• Utility Components: For miscellaneous functionalities including safety checks, random utilities, and constants (spread over “hart/utils” and “modules/utils”).
Relationships and interactions
• The Application Layer (app.py, sample.py, latency_profile.py) accepts inputs (prompts) and interacts with the model components.
• Input Prompts are processed through the text model (external dependency) and then passed into the hybrid transformer architecture.
• The Autoencoder provides a latent representation where the hybrid tokenizer decomposes the input into discrete (big picture) and continuous (residual) tokens.
• The Discrete Part is modeled with the autoregressive Transformer (from “modules/models/transformer”) while the Continuous Part is refined via a diffusion module.
• The high-performance kernels (CUDA implementations in “hart/kernels”) underpin the efficient execution of key operations like layer normalization and token handling, ensuring low MACs and high throughput.
• Utility modules support overall system coordination, safety checking, and additional logic useful in pre- or post-processing.
Architectural patterns and design principles
• Modular Design: The codebase is clearly segmented into layers – inference/application, core model logic, and compute kernels – which enables clear abstraction layers.
• Hybrid Modeling: The key research innovation is in the hybrid tokenizer and dual-path model (AR for discrete components, diffusion for continuous).
• Acceleration: The separation of CUDA-based low-level operations (housed in the “kernels” directory) from the high-level model logic shows a design that emphasizes performance.
• Extensibility: With separate modules for diffusion, losses, networks, and utility functions, the architecture supports further experimentation and extension.
Additional detailed points
• How the system ensures scalability by decoupling critical processes (model components and kernel operations).
• Optionally, mark external dependencies that must be configured during setup (e.g., paths to models, safety check module).