# snowfallingplum/shmt architecture

> Architecture diagram of the GitHub repository snowfallingplum/shmt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/snowfallingplum/shmt
- Repository: https://github.com/snowfallingplum/shmt

## Overview

##### Project Type and Purpose
- This is a research codebase built for a NeurIPS 2024 paper focused on makeup transfer using latent diffusion techniques.
- The project includes training, inference, and data processing components centered around latent diffusion models (LDM) augmented for makeup transfer.
- It builds on existing projects like LDM and Paint-by-Example, and uses PyTorch as the primary deep learning framework.

##### Analysis of the File Structure
- Top-Level Files:
- README.md and LICENSE: Provide project overview, usage instructions, and licensing.
- environment.yaml: Conda environment setup for dependencies.
- main.py, makeup_inference_h0.py, makeup_inference_h4.py, makeup_inference_model2.py: These files serve as entry points for training and inference.

- Directories and their significance:
- configs: Holds YAML configuration files (e.g., shmt_h0.yaml and shmt_h4.yaml), which define model hyperparameters and training/inference settings.
- examples: Contains sample data directories for images, segmentation (seg2), and 3D data, illustrating the expected input structure.
- ldm: This package contains the core modules:
- ldm/data: Data loading and preprocessing components, including specialized dataset classes.
- ldm/models: The autoencoder and diffusion models, including various diffusion implementations (ddim, ddpm, plms, classifier-based, etc.).
- ldm/modules: Contains additional modules that support attention, encoder operations, image degradation (for data augmentation), and custom losses.
- ldm/util.py: Includes utility functions used across the project.
- taming: Another major directory hosting alternate data modules, models (like vqgan), diffusion modules, and specialized losses used for conditional image generation.

##### Key Components
- Data Preparation / Ingestion:
- Pipeline for processing input images, segmentation maps, and 3D images from the examples directory.
- How the datasets (possibly combined from external projects like BeautyGAN, Wild-MT, LADN) are prepared as “train” and “inference” data.

- Configuration Management:
- A component for the configuration files in the configs/latent-diffusion directory, indicating that these YAMLs drive the training and inference scripts.

- Core Model Components:
- Autoencoder: Shown under ldm/models/autoencoder.py and its pre-trained status (e.g., VQ-f4 from CompVis/LDM).
- Diffusion Modules: Represent various diffusion strategies (ddim, ddpm, plms, classifier) under ldm/models/diffusion and taming/modules/diffusionmodules.
- Encoders and Attention: Captured from the ldm/modules/encoders and ldm/modules/attention, highlighting any self-supervision or hierarchical components.
- Specialized Loss Functions: Components that implement perceptual losses or feature-based losses under ldm/modules/losses and taming/modules/losses.

- Inference Pipelines:
- Separate components for the inference scripts (makeup_inference_h0.py, makeup_inference_h4.py, makeup_inference_model2.py). These will show the flow from input data to processed outputs using the defined model architecture.

- Training Workflow:
- That main.py is the training entry point, which initializes the model by reading configuration settings and datasets, and then triggers training using GPUs.

- External Dependencies & Pre-trained Models:
- External services/components for pre-trained autoencoding models (downloaded from LDM) and mention integration of third-party libraries (for face parsing and 3D processing via 3DDFA_V2, etc.).

##### Architectural Patterns and Interactions
- Modularization: Split the system into clearly defined modules for data processing, model building, training, and inference.
- Re-usability: Emphasize the reuse of modules (e.g., latent diffusion techniques from LDM and VQGAN inspired components from the taming directory) and how they interact.

##### Final Assembly
1. “User/Interface” or “Data Preparation” block.
2. Flow into the “Configuration Manager” that reads the YAML settings.
3. Feed into two parallel pipelines: one for training (with training scripts and model components) and one for inference (with dedicated inference scripts).
4. Both pipelines make use of the shared core libraries in ldm and taming.
5. Output results and tie back to the “Asset” components (e.g., sample outputs shown in README).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Block
    subgraph "Data Preparation"
        D1["Input Data (Images,Segmentation,3D)"]:::dataprep
        D2["Data Loader (ldm/data/makeup_dataset.py)"]:::dataprep
    end

    %% Configuration Manager Block
    subgraph "Configuration Manager"
        C1["Configuration Files (configs/latent-diffusion)"]:::config
    end

    %% Training Pipeline Block
    subgraph "Training Pipeline"
        T1["Training Entry (main.py)"]:::training
    end

    %% Inference Pipeline Block
    subgraph "Inference Pipeline"
        I1["Makeup Inference H0 (makeup_inference_h0.py)"]:::inference
        I2["Makeup Inference H4 (makeup_inference_h4.py)"]:::inference
        I3["Makeup Inference Model2 (makeup_inference_model2.py)"]:::inference
    end

    %% Core Model Components Block
    subgraph "Core Model Components"
        M1["Autoencoder (ldm/models/autoencoder.py)"]:::core
        M2["Diffusion Modules (LDM)"]:::core
        M3["Diffusion Modules (Taming)"]:::core
        M4["Encoders (ldm/modules/encoders)"]:::core
        M5["Attention (ldm/modules/attention.py)"]:::core
        M6["Losses (ldm/modules/losses)"]:::core
        M7["Losses (Taming)"]:::core
        M8["External Model (taming/models/vqgan.py)"]:::external
    end

    %% Assets Block
    subgraph "Output"
        A1["Assets/Output Images (asset)"]:::assets
    end

    %% Connections from Data Preparation and Configurations to Pipelines
    D1 --> D2
    D2 --> T1
    D2 --> I1
    D2 --> I2
    D2 --> I3
    C1 --> T1
    C1 --> I1
    C1 --> I2
    C1 --> I3

    %% Training Pipeline to Core Model Components
    T1 --> M1
    T1 --> M2
    T1 --> M3
    T1 --> M4
    T1 --> M5
    T1 --> M6
    T1 --> M7
    T1 --> M8

    %% Inference Pipeline to Core Model Components
    I1 --> M1
    I1 --> M2
    I1 --> M3
    I1 --> M4
    I1 --> M5
    I1 --> M6
    I1 --> M7
    I1 --> M8

    I2 --> M1
    I2 --> M2
    I2 --> M3
    I2 --> M4
    I2 --> M5
    I2 --> M6
    I2 --> M7
    I2 --> M8

    I3 --> M1
    I3 --> M2
    I3 --> M3
    I3 --> M4
    I3 --> M5
    I3 --> M6
    I3 --> M7
    I3 --> M8

    %% Core Model Components to Output (Inference Results)
    M1 --- A1
    M2 --- A1
    M3 --- A1
    M4 --- A1
    M5 --- A1
    M6 --- A1
    M7 --- A1
    M8 --- A1

    %% Click Events
    click D1 "https://github.com/snowfallingplum/shmt/tree/main/examples/"
    click D2 "https://github.com/snowfallingplum/shmt/blob/main/ldm/data/makeup_dataset.py"
    click C1 "https://github.com/snowfallingplum/shmt/tree/main/configs/latent-diffusion"
    click T1 "https://github.com/snowfallingplum/shmt/blob/main/main.py"
    click M1 "https://github.com/snowfallingplum/shmt/blob/main/ldm/models/autoencoder.py"
    click M2 "https://github.com/snowfallingplum/shmt/tree/main/ldm/models/diffusion"
    click M3 "https://github.com/snowfallingplum/shmt/blob/main/taming/modules/diffusionmodules/model.py"
    click M4 "https://github.com/snowfallingplum/shmt/blob/main/ldm/modules/encoders/modules.py"
    click M5 "https://github.com/snowfallingplum/shmt/blob/main/ldm/modules/attention.py"
    click M6 "https://github.com/snowfallingplum/shmt/blob/main/ldm/modules/losses/contperceptual.py"
    click M7 "https://github.com/snowfallingplum/shmt/blob/main/taming/modules/losses/lpips.py"
    click M8 "https://github.com/snowfallingplum/shmt/blob/main/taming/models/vqgan.py"
    click I1 "https://github.com/snowfallingplum/shmt/blob/main/makeup_inference_h0.py"
    click I2 "https://github.com/snowfallingplum/shmt/blob/main/makeup_inference_h4.py"
    click I3 "https://github.com/snowfallingplum/shmt/blob/main/makeup_inference_model2.py"

    %% Styles
    classDef dataprep fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef config fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef training fill:#A9DFBF,stroke:#229954,stroke-width:2px;
    classDef inference fill:#F5B7B1,stroke:#CB4335,stroke-width:2px;
    classDef core fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef external fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef assets fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
```
