# nvlabs/sana architecture

> Architecture diagram of the GitHub repository nvlabs/sana, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/sana
- Repository: https://github.com/nvlabs/sana

## Overview

##### Main Components
- Data & Configuration:
– Data ingestion modules: Look at the “diffusion/data” directories (datasets, transforms, and wids) along with example data in “asset/example_data”.
– Config directories: The “configs”, “configs/sana_config”, “configs/sana_controlnet_config”, and others provide parameter files for different model variations and training modes.
- Core Model Modules:
– Diffusion models and submodules: The “diffusion/model” folder contains the core modules, including DC-AE components, Linear DiT (linear attention transformer) modules, various utilities in “nets” (like sana, sana_U_shape, and multi-scale variants), and scheduler implementations under “diffusion/scheduler”.
– The text encoder and VAE modules (implicit in the pipelines) are key components as well.
- Pipeline Abstractions for Inference:
– In “app” we see multiple pipelines (e.g., app_sana.py, app_sana_4bit.py, sana_pipeline.py) that encapsulate inference logic.
– Integration with external frameworks: The README explains how Sana integrates with the Hugging Face diffusers API (e.g., SanaPipeline, SanaPAGPipeline) to support diverse inference scenarios.
- Training & Evaluation:
– Training scripts are organized under “train_scripts” and “CIs” (automation scripts & FSDP/ DDP configurations).
– Evaluation and metric tools: “tools/metrics” hold various modules (CLIP-score, geneval, DPG-Bench, etc.) for assessing model performance.
- Supplementary Tools & CLI:
– Directories such as “sana/cli” and “tools” provide helper scripts for downloading models, converting datasets (e.g., convert_ImgDataset_to_WebDatasetMS_format.py), and other utility actions.
- Infrastructure & CI/CD:
– The ".github" folder and Dockerfile illustrate deployment and testing (Dockerized inference) environments.

##### Relationships and Data Flow
- Data Flow:
– Raw image-text pair data and webdataset conversions feed into data loaders under “diffusion/data” which then supply training pipelines.
– Configuration files are read by both training and inference modules to set up the appropriate hyperparameters and model weights.
- Model Pipeline:
– The diffusers and custom pipelines (“SanaPipeline”, “SanaPAGPipeline”) take as input processed data and textual prompts then run through the encoder (decoder-only text encoder) and the diffusion model (with DC-AE and Linear DiT) to generate image outputs.
– The schedulers (Flow-DPM-Solver, SCM, etc.) adjust the sampling process, reducing inference steps and accelerating generation.
- Training and Evaluation Flow:
– Training scripts (in “train_scripts”) integrate with data modules and configuration files to run distributed training (supporting FSDP/DDP).
– Post-training, the evaluation is conducted using the metric toolkit and various specialized tools in the “tools/metrics” directory.

##### Important Architectural Patterns and Design Principles
- Modularity: Each major function (data, model architecture, training, inference, evaluation) is encapsulated in different directories.
- Configuration-Driven: The use of extensive YAML configuration files (configs/) allows the same codebase to be adapted for different model variants (e.g., standard, ControlNet, quantized, sprint variants).
- Pipeline Abstraction: Separation of inference pipelines (native Python pipelines, diffusers pipelines, Docker-based inference) helps isolate concerns.
- Distributed and Accelerated Computing: Support for FSDP and DDP in training scripts emphasizes efficiency, which is further promoted by optimized scheduling algorithms (e.g., Flow-DPM-Solver).

##### Relevant Technologies and Libraries
- Python (>=3.10) and PyTorch (>=2.0.1+cu12.1) underpin the project.
- Hugging Face diffusers integration is key (for pipelines like SanaPipeline and SanaPAGPipeline).
- Docker for containerized deployment.
- CI/CD practices via GitHub Actions (in .github/workflows).
- Metric evaluation tools (CLIP, geneval) and conversion utilities.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Data Processing
    subgraph "Data Processing"
        DI["Data Ingestion"]:::data
        ED["Example Data"]:::data
    end

    %% Configuration Files
    subgraph "Configuration Files"
        CFG1["configs"]:::config
        CFG2["Sana Config"]:::config
        CFG3["Sana ControlNet Config"]:::config
        CFG4["Sana Sprint Config"]:::config
        CFG5["Sana App Config"]:::config
    end

    %% Core Diffusion Engine
    subgraph "Core Diffusion Engine"
        DM["Diffusion Model"]:::model
        DCAE["DC-AE Modules"]:::model
        NETS["Linear DiT & Nets"]:::model
    end

    %% Inference Pipelines
    subgraph "Inference Pipelines"
        SP["Sana Pipeline"]:::inference
        AS["App Sana"]:::inference
        AS4["App Sana 4bit"]:::inference
    end

    %% Training Module
    subgraph "Training Module"
        TS["Training Scripts"]:::training
        CI["CI Configurations"]:::training
    end

    %% Evaluation & Metrics
    EM["Evaluation & Metrics"]:::evaluation

    %% Supplementary Tools & CLI
    subgraph "Supplementary Tools & CLI"
        CLI["CLI Tools"]:::tools
        CONV["Conversion Utilities"]:::tools
    end

    %% Infrastructure & CI/CD
    subgraph "Infrastructure & CI/CD"
        GH[".github Workflows"]:::infrastructure
        DOCK["Docker"]:::infrastructure
    end

    %% External Integration
    EX["HuggingFace Diffusers"]:::external

    %% Data Flow Relationships
    DI -->|"feeds"| TS
    DI -->|"feeds"| SP
    ED -->|"provides sample"| DI

    %% Configurations injection
    CFG1 --- TS
    CFG2 --- TS
    CFG3 --- TS
    CFG4 --- TS
    CFG5 --- TS

    CFG1 --- SP
    CFG2 --- SP
    CFG3 --- SP
    CFG4 --- SP
    CFG5 --- SP

    CFG1 --- EM
    CFG2 --- EM
    CFG3 --- EM
    CFG4 --- EM
    CFG5 --- EM

    %% Training and Model interactions
    TS -->|"trains"| DM
    CI --> DM
    DCAE --- DM
    NETS --- DM

    %% Model to Inference flow
    DM -->|"informs"| SP
    DM -->|"informs"| AS
    DM -->|"informs"| AS4

    %% External integration via diffusers
    EX -->|"integrates"| SP
    EX -->|"integrates"| AS
    EX -->|"integrates"| AS4

    %% Supplementary tools support data conversion
    CONV -->|"converts"| DI
    CLI -->|"supports"| TS

    %% Infrastructure supports training and inference
    GH --> TS
    GH --> SP
    GH --> AS
    GH --> AS4
    DOCK --> TS
    DOCK --> SP
    DOCK --> AS
    DOCK --> AS4

    %% Training outputs evaluated by metrics
    TS -->|"generates models"| EM

    %% Click Events for Data Processing
    click DI "https://github.com/nvlabs/sana/tree/main/diffusion/data"
    click ED "https://github.com/nvlabs/sana/tree/main/asset/example_data"

    %% Click Events for Configuration Files
    click CFG1 "https://github.com/nvlabs/sana/tree/main/configs"
    click CFG2 "https://github.com/nvlabs/sana/tree/main/configs/sana_config"
    click CFG3 "https://github.com/nvlabs/sana/tree/main/configs/sana_controlnet_config"
    click CFG4 "https://github.com/nvlabs/sana/tree/main/configs/sana_sprint_config"
    click CFG5 "https://github.com/nvlabs/sana/tree/main/configs/sana_app_config"

    %% Click Events for Core Diffusion Engine
    click DM "https://github.com/nvlabs/sana/tree/main/diffusion/model"
    click DCAE "https://github.com/nvlabs/sana/tree/main/diffusion/model/dc_ae"
    click NETS "https://github.com/nvlabs/sana/tree/main/diffusion/model/nets"

    %% Click Events for Inference Pipelines
    click SP "https://github.com/nvlabs/sana/blob/main/app/sana_pipeline.py"
    click AS "https://github.com/nvlabs/sana/blob/main/app/app_sana.py"
    click AS4 "https://github.com/nvlabs/sana/blob/main/app/app_sana_4bit.py"

    %% Click Events for Training Module
    click TS "https://github.com/nvlabs/sana/tree/main/train_scripts"
    click CI "https://github.com/nvlabs/sana/tree/main/CIs"

    %% Click Event for Evaluation & Metrics
    click EM "https://github.com/nvlabs/sana/tree/main/tools/metrics"

    %% Click Events for Supplementary Tools & CLI
    click CLI "https://github.com/nvlabs/sana/tree/main/sana/cli"
    click CONV "https://github.com/nvlabs/sana/blob/main/tools/convert_ImgDataset_to_WebDatasetMS_format.py"

    %% Click Events for Infrastructure & CI/CD
    click GH "https://github.com/nvlabs/sana/blob/main/.github"
    click DOCK "https://github.com/nvlabs/sana/tree/main/Dockerfile"

    %% Styles
    classDef data fill:#90EE90,stroke:#006400,stroke-width:2px;
    classDef config fill:#ADD8E6,stroke:#00008B,stroke-width:2px;
    classDef model fill:#FFB6C1,stroke:#8B0000,stroke-width:2px;
    classDef inference fill:#FFD700,stroke:#B8860B,stroke-width:2px;
    classDef training fill:#FFA07A,stroke:#CD5C5C,stroke-width:2px;
    classDef evaluation fill:#DDA0DD,stroke:#800080,stroke-width:2px;
    classDef tools fill:#E6E6FA,stroke:#00008B,stroke-width:2px;
    classDef infrastructure fill:#F0E68C,stroke:#BDB76B,stroke-width:2px;
    classDef external fill:#C0C0C0,stroke:#696969,stroke-width:2px;
```
