# neu-vi/smoodi architecture

> Architecture diagram of the GitHub repository neu-vi/smoodi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neu-vi/smoodi
- Repository: https://github.com/neu-vi/smoodi

## Overview

##### Project type and purpose
- This is an open-source research toolkit and full-pipeline ML application in Python: “SMooDi,” a stylized motion diffusion model.
- It ingests motion-capture datasets (e.g., HumanML3D, 100STYLE), trains a diffusion model, evaluates it, and renders synthesized motions via Blender/FFmpeg.

##### File structure
- Top-level directories and scripts:
- configs/ – YAML configuration files (global + module-level)
- datasets/ – raw and processed data files and metadata
- mld/ – main Python package subdivided into:
· data/ – dataset definitions, loaders, sampling utilities
· transforms/ – motion representation and conversion utilities
· models/ – architecture definitions, model‐factory, losses, metrics, operators
· render/ – motion visualization (Blender integration, video output)
· tools/, utils/, callback/ – logging, run-ID management, training callbacks
- prepare/ – shell scripts to download dependencies, pretrained weights, SMPL models
- scripts/, demo.sh, train.sh, test.sh – user‐facing entry points for demo, training, evaluation
- train.py – Python entry point for training
- Configuration-driven: everything parameterized via YAML under configs/.

##### README for additional insights
- Describes data processing steps, environment setup (conda, ffmpeg, Blender), dataset preparation, model inference (“demo.sh”), training (“train.sh”), evaluation (“test.sh”).
- Notes external dependencies: PyTorch, SMPL/SMPL-X, PyTorch3D, OpenAI CLIP, Blender, ffmpeg.

##### Main components and their interactions
a. Configuration Manager
- Reads YAMLs from configs/, builds unified config object.
- Injects settings (datasets paths, model hyperparameters, training flags).
b. Data Pipeline
- mld/data/ loader modules (HumanML3D, 100STYLE, KIT, UESTC)
- Sampling modules produce batches via mld/data/sampling.
- Transform modules convert SMPL poses to network features.
c. Model Factory & Architectures
- mld/models/get_model.py uses config to instantiate specific architecture (VAE, diffusion, style encoder, text encoder).
- Loss modules compute reconstruction, KL, style/sampling losses.
- Metrics modules evaluate generation quality.
d. Training & Evaluation
- train.py orchestrates training loop:
– Initialize model, optimizer, dataloaders
– Register callbacks (progress logging, checkpointing)
– Compute losses and backprop
- test.sh uses test_cmld.py to run evaluation metrics with or without style guidance.
e. Inference & Demo
- demo.sh + demo_cmld.py: read text prompt and style motion, run diffusion inference, output motion files under demo/.
f. Rendering
- mld/render/ interfaces with Blender via mld/launch/blender.py
- Renders sequences into videos, applies materials, cameras, joints.
- scripts/visualize_motion*.sh wrap rendering for batch jobs.
g. External Dependencies
- Pretrained weights (MLD base model, CLIP, BERT, evaluators)
- SMPL/SMPL-X, PyTorch3D, Blender, ffmpeg

##### Architectural patterns and principles
- Modular package structure by concern (data, model, render, utils).
- Config-driven: decouples code from hyperparameters.
- Factory pattern for model instantiation.
- Callback pattern for training monitoring and checkpointing.
- Clear separation of training, evaluation, inference, rendering stages.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Config Layer
    subgraph "Configuration Manager"
        ConfigFiles["configs/"] 
        ConfigPy["mld/config.py"]
    end

    %% Data Pipeline
    subgraph "Data Pipeline" 
        Datasets["datasets/"]
        DataDefs["mld/data/"]
        Sampling["mld/data/sampling/"]
    end

    %% Transform Modules
    subgraph "Transform Modules"
        Transforms["mld/transforms/"]
        J2jfeats["mld/transforms/joints2jfeats/"]
        J2rots["mld/transforms/joints2rots/"]
    end

    %% Model Factory & Architectures
    subgraph "Model Pipeline"
        Factory["mld/models/get_model.py"]
        Archs["mld/models/architectures/"]
        ModelTypes["mld/models/modeltype/"]
        Losses["mld/models/losses/"]
        Metrics["mld/models/metrics/"]
    end

    %% Training Engine & Callbacks
    subgraph "Training Engine"
        TrainPy["train.py"]
        ProgressCB["mld/callback/progress.py"]
        RunID["mld/tools/runid.py"]
        Checkpoints["Checkpoint Store"]
    end

    %% Evaluation
    subgraph "Evaluation"
        TestSh["test.sh"]
        TestPy["test_cmld.py"]
    end

    %% Inference & Demo
    subgraph "Inference & Demo"
        DemoSh["demo.sh"]
        DemoPy["demo_cmld.py"]
        DemoOut["demo/"]
    end

    %% Rendering Subsystem
    subgraph "Rendering Subsystem"
        RenderPkg["mld/render/"]
        BlenderInt["mld/render/blender/"]
        VisScript["scripts/visualize_motion.sh"]
        VideoOutputs["Video Outputs"]
    end

    %% External Dependencies
    Pretrained["prepare/"]:::external
    SMPL["SMPL/SMPL-X"]:::external
    Py3D["PyTorch3D"]:::external
    CLIP["OpenAI CLIP"]:::external
    BlenderFF["Blender/ffmpeg"]:::external

    %% Connections
    ConfigFiles -->|"settings"| Datasets
    ConfigFiles -->|"settings"| DataDefs
    ConfigFiles -->|"settings"| Sampling
    ConfigFiles -->|"settings"| Transforms
    ConfigFiles -->|"settings"| Factory
    ConfigPy -->|"settings"| Factory

    Datasets -->|"raw data"| DataDefs
    DataDefs -->|"datasets"| Sampling
    Sampling -->|"batches"| Factory
    Transforms -->|"preprocessed features"| Factory

    Factory -->|"instantiates"| Archs
    Factory -->|"uses"| ModelTypes
    Factory -->|"defines"| Losses
    Factory -->|"defines"| Metrics

    Factory -->|"forward/backward"| TrainPy
    TrainPy -->|"registers"| ProgressCB
    TrainPy -->|"registers"| RunID
    TrainPy -->|"saves"| Checkpoints
    TrainPy -->|"computes"| Losses
    TrainPy -->|"computes"| Metrics
    TrainPy -->|"uses"| DataDefs

    Checkpoints -->|"loaded by"| TestSh
    TestSh -->|"runs"| TestPy
    TestPy -->|"reads"| DataDefs
    TestPy -->|"computes"| Metrics

    Checkpoints -->|"loaded by"| DemoSh
    DemoSh -->|"runs"| DemoPy
    DemoPy -->|"outputs motions"| DemoOut
    DemoOut -->|"to render"| RenderPkg

    RenderPkg -->|"calls"| BlenderInt
    RenderPkg -->|"invokes"| VisScript
    VisScript -->|"generates"| VideoOutputs

    Pretrained -->|"provides weights"| Factory
    Pretrained -->|"provides assets"| SMPL
    SMPL -->|"used by"| DataDefs
    Py3D -->|"used by"| RenderPkg
    CLIP -->|"used by"| Factory
    BlenderFF -->|"used by"| RenderPkg

    %% Click Events
    click ConfigFiles "https://github.com/neu-vi/smoodi/tree/main/configs/"
    click ConfigPy "https://github.com/neu-vi/smoodi/blob/main/mld/config.py"
    click Datasets "https://github.com/neu-vi/smoodi/tree/main/datasets/"
    click DataDefs "https://github.com/neu-vi/smoodi/tree/main/mld/data/"
    click Sampling "https://github.com/neu-vi/smoodi/tree/main/mld/data/sampling/"
    click Transforms "https://github.com/neu-vi/smoodi/tree/main/mld/transforms/"
    click J2jfeats "https://github.com/neu-vi/smoodi/tree/main/mld/transforms/joints2jfeats/"
    click J2rots "https://github.com/neu-vi/smoodi/tree/main/mld/transforms/joints2rots/"
    click Factory "https://github.com/neu-vi/smoodi/blob/main/mld/models/get_model.py"
    click Archs "https://github.com/neu-vi/smoodi/tree/main/mld/models/architectures/"
    click ModelTypes "https://github.com/neu-vi/smoodi/tree/main/mld/models/modeltype/"
    click Losses "https://github.com/neu-vi/smoodi/tree/main/mld/models/losses/"
    click Metrics "https://github.com/neu-vi/smoodi/tree/main/mld/models/metrics/"
    click TrainPy "https://github.com/neu-vi/smoodi/blob/main/train.py"
    click ProgressCB "https://github.com/neu-vi/smoodi/blob/main/mld/callback/progress.py"
    click RunID "https://github.com/neu-vi/smoodi/blob/main/mld/tools/runid.py"
    click TestSh "https://github.com/neu-vi/smoodi/blob/main/test.sh"
    click TestPy "https://github.com/neu-vi/smoodi/blob/main/test_cmld.py"
    click DemoSh "https://github.com/neu-vi/smoodi/blob/main/demo.sh"
    click DemoPy "https://github.com/neu-vi/smoodi/blob/main/demo_cmld.py"
    click DemoOut "https://github.com/neu-vi/smoodi/tree/main/demo/"
    click RenderPkg "https://github.com/neu-vi/smoodi/tree/main/mld/render/"
    click BlenderInt "https://github.com/neu-vi/smoodi/tree/main/mld/render/blender/"
    click VisScript "https://github.com/neu-vi/smoodi/blob/main/scripts/visualize_motion.sh"
    click Pretrained "https://github.com/neu-vi/smoodi/tree/main/prepare/"

    %% Styles
    classDef config fill:#ccf,stroke:#333
    classDef data fill:#cce,stroke:#333
    classDef transforms fill:#cce,stroke:#333
    classDef model fill:#fdd,stroke:#333
    classDef training fill:#dfd,stroke:#333
    classDef scripts fill:#dfd,stroke:#333
    classDef external fill:#fcc,stroke:#900
    classDef rendering fill:#ffe,stroke:#333

    class ConfigFiles,ConfigPy config
    class Datasets,DataDefs,Sampling data
    class Transforms,J2jfeats,J2rots transforms
    class Factory,Archs,ModelTypes,Losses,Metrics model
    class TrainPy,ProgressCB,RunID,Checkpoints training
    class TestSh,TestPy scripts
    class DemoSh,DemoPy,DemoOut scripts
    class RenderPkg,BlenderInt,VisScript,VideoOutputs rendering
```
