# guytevet/motion-diffusion-model architecture

> Architecture diagram of the GitHub repository guytevet/motion-diffusion-model, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/guytevet/motion-diffusion-model
- Repository: https://github.com/guytevet/motion-diffusion-model

## Overview

##### Project type and purpose
- This is a research‑oriented full‑stack ML codebase implementing the “Human Motion Diffusion Model” (MDM).
- It supports three core pipelines:
a. Text‑to‑Motion (HumanML3D, KIT)
b. Action‑to‑Motion (UESTC, HumanAct12)
c. Unconstrained motion generation
- It offers data preparation, model definitions, training, sampling (inference), evaluation, and visualization.

##### File structure
- Top‑level directories:
- data_loaders — adapters for each dataset
- diffusion — core diffusion scheduler, losses, utilities
- model — network architectures (MDM, BERT encoder, SMPL utilities)
- train — training entry points and loops
- sample — inference scripts (generate, edit, predict)
- eval — evaluation scripts and metrics for each task
- prepare — data/model/download helpers
- utils — shared functions (config, logging, distributed, rotations)
- Configuration files: environment.yml, cog.yaml, DiP.md (DiP extension)
- Storage: assets/, dataset/, body_models/, save/ (user‑provided at runtime)

##### Key README insights
- Core dependencies: PyTorch, CUDA, CLIP, SMPL, SMPL‑X, PyTorch3D
- Multi‑stage modeling: diffusion process with configurable steps, text encoder choices (CLIP or BERT), optional EMA
- Three usage modes: synthesis, editing, training, evaluation, rendering

##### Main components
- A. Data Layer
- Raw datasets (HumanML3D, KIT, UESTC, HumanAct12)
- Scripts to download and preprocess (prepare/*)
- B. Data Loader Layer
- data_loaders/*: transforms raw files → tensors, skeleton conversions, quaternions
- C. Model Layer
- model/… (mdm diffusion nets, BERT encoder, SMPL utilities)
- diffusion/* (noise schedule, sampling, losses)
- D. Training Pipeline
- train/train_mdm.py, training_loop.py, train_platforms.py
- Inputs: data loader, model, optimizer, config
- Outputs: checkpoints in save/
- E. Inference Pipeline
- sample/generate.py, edit.py, predict.py
- Loads checkpoint → runs reverse diffusion → outputs results.npy, stick‑figure MP4s
- F. Evaluation Pipeline
- eval/*: metric calculators (FID, diversity, accuracy, recognition)
- Loads generated data + ground truth → computes and logs metrics
- G. Visualization Pipeline
- H. External Dependencies
- CLIP model (one call cached)
- SMPL/SMPL‑X anthropometric models
- GPU/CUDA environment
- Optional WandB/TensorBoard for logging

##### Relationships & interactions
- Data downloads (prepare) → Dataset storage → Data Loaders → Training / Inference
- Training writes checkpoints → Sampling/Editing loads checkpoints → Generates raw motion
- Sampling → Visualization for mesh rendering
- Generated motions + GT → Evaluation metrics
- Utils (config, logging, distributed) used by all pipelines

##### Architectural patterns & design principles
- Config‑driven: single utils/config.py driving CLI across modules
- Reusable components: BERT/CLIP encoders pluggable via --text_encoder_type
- Extensible pipelines: each task has its own eval and data_loader submodule
- Lightweight shell scripts for dependency and data management (prepare/)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data & Downloads Layer
    subgraph "Data Layer" 
        direction TB
        prepare["prepare/"]:::data
        dataset["dataset/"]:::data
    end

    %% Data Loader Layer
    subgraph "Data Loader Layer"
        direction TB
        dl_dir["data_loaders/"]:::data
        a2m["a2m/"]:::data
        humanml["humanml/"]:::data
        get_data["get_data.py"]:::data
        tensors["tensors.py"]:::data
    end

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        model_dir["model/"]:::model
        mdm["mdm.py"]:::model
        bert["BERT_encoder.py"]:::model
        smpl["smpl.py"]:::model
        rot2xyz["rotation2xyz.py"]:::model
        diffusion_dir["diffusion/"]:::model
        gaussian["gaussian_diffusion.py"]:::model
        losses["losses.py"]:::model
        resample["resample.py"]:::model
        respace["respace.py"]:::model
        nn_mod["nn.py"]:::model
        logger_mod["logger.py"]:::model
        fp16["fp16_util.py"]:::model
    end

    %% Training Pipeline
    subgraph "Training Pipeline" 
        direction TB
        train_dir["train/"]:::core
        train_entry["train_mdm.py"]:::core
        loop["training_loop.py"]:::core
        platforms["train_platforms.py"]:::core
        save["save/"]:::output
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        direction TB
        sample_dir["sample/"]:::core
        gen["generate.py"]:::core
        edit["edit.py"]:::core
        predict["predict.py"]:::core
    end

    %% Evaluation Pipeline
    subgraph "Evaluation Pipeline"
        direction TB
        eval_dir["eval/"]:::output
        eval_a2m["a2m/"]:::output
        eval_humanml["eval_humanml.py"]:::output
        eval_humanact["eval_humanact12_uestc.py"]:::output
        eval_uncon["unconstrained/"]:::output
    end

    %% Visualization Pipeline
    subgraph "Visualization Pipeline"
        direction TB
        vis_dir["visualize/"]:::output
        render["render_mesh.py"]:::output
        joints["joints2smpl/"]:::output
        hik["motions2hik.py"]:::output
        simp["simplify_loc2rot.py"]:::output
        vis_utils["vis_utils.py"]:::output
    end

    %% Config & External
    subgraph "Configuration & External"
        direction TB
        config["utils/config.py"]:::external
        env["environment.yml"]:::external
        cog["cog.yaml"]:::external
        dip["DiP.md"]:::external
        gpu["GPU/CUDA"]:::external
        clip["CLIP Model"]:::external
        smpl_ext["SMPL/SMPL-X"]:::external
        wandb["WandB/TensorBoard"]:::external
    end

    %% Connections
    prepare -->|downloads| dataset
    dataset --> dl_dir
    dl_dir --> a2m
    dl_dir --> humanml
    dl_dir --> get_data
    dl_dir --> tensors

    dl_dir --> train_entry
    model_dir --> train_entry
    train_entry --> loop
    loop --> platforms
    loop --> save

    save --> gen
    gen --> edit
    gen --> predict

    gen --> render
    edit --> render

    gen --> eval_dir
    eval_humanml --> eval_dir
    eval_a2m --> eval_dir
    eval_humanact --> eval_dir
    eval_uncon --> eval_dir

    gen --> vis_dir
    render --> vis_dir

    config --> train_entry
    config --> gen
    config --> eval_dir
    config --> render

    clip -.-> gen
    clip -.-> bert
    smpl_ext -.-> smpl
    gpu -.-> train_entry
    gpu -.-> gen

    wandb -.-> train_entry

    %% Click Events
    click dataset "https://github.com/guytevet/motion-diffusion-model/tree/main/dataset/"
    click prepare "https://github.com/guytevet/motion-diffusion-model/tree/main/prepare/"
    click dl_dir "https://github.com/guytevet/motion-diffusion-model/tree/main/data_loaders/"
    click a2m "https://github.com/guytevet/motion-diffusion-model/tree/main/data_loaders/a2m/"
    click humanml "https://github.com/guytevet/motion-diffusion-model/tree/main/data_loaders/humanml/"
    click get_data "https://github.com/guytevet/motion-diffusion-model/blob/main/data_loaders/get_data.py"
    click tensors "https://github.com/guytevet/motion-diffusion-model/blob/main/data_loaders/tensors.py"
    click model_dir "https://github.com/guytevet/motion-diffusion-model/tree/main/model/"
    click mdm "https://github.com/guytevet/motion-diffusion-model/blob/main/model/mdm.py"
    click bert "https://github.com/guytevet/motion-diffusion-model/blob/main/model/BERT/BERT_encoder.py"
    click smpl "https://github.com/guytevet/motion-diffusion-model/blob/main/model/smpl.py"
    click rot2xyz "https://github.com/guytevet/motion-diffusion-model/blob/main/model/rotation2xyz.py"
    click diffusion_dir "https://github.com/guytevet/motion-diffusion-model/tree/main/diffusion/"
    click gaussian "https://github.com/guytevet/motion-diffusion-model/blob/main/diffusion/gaussian_diffusion.py"
    click losses "https://github.com/guytevet/motion-diffusion-model/blob/main/diffusion/losses.py"
    click resample "https://github.com/guytevet/motion-diffusion-model/blob/main/diffusion/resample.py"
    click respace "https://github.com/guytevet/motion-diffusion-model/blob/main/diffusion/respace.py"
    click nn_mod "https://github.com/guytevet/motion-diffusion-model/blob/main/diffusion/nn.py"
    click logger_mod "https://github.com/guytevet/motion-diffusion-model/blob/main/diffusion/logger.py"
    click fp16 "https://github.com/guytevet/motion-diffusion-model/blob/main/diffusion/fp16_util.py"
    click train_dir "https://github.com/guytevet/motion-diffusion-model/tree/main/train/"
    click train_entry "https://github.com/guytevet/motion-diffusion-model/blob/main/train/train_mdm.py"
    click loop "https://github.com/guytevet/motion-diffusion-model/blob/main/train/training_loop.py"
    click platforms "https://github.com/guytevet/motion-diffusion-model/blob/main/train/train_platforms.py"
    click sample_dir "https://github.com/guytevet/motion-diffusion-model/tree/main/sample/"
    click gen "https://github.com/guytevet/motion-diffusion-model/blob/main/sample/generate.py"
    click edit "https://github.com/guytevet/motion-diffusion-model/blob/main/sample/edit.py"
    click predict "https://github.com/guytevet/motion-diffusion-model/blob/main/sample/predict.py"
    click eval_dir "https://github.com/guytevet/motion-diffusion-model/tree/main/eval/"
    click eval_a2m "https://github.com/guytevet/motion-diffusion-model/tree/main/eval/a2m/"
    click eval_humanml "https://github.com/guytevet/motion-diffusion-model/blob/main/eval/eval_humanml.py"
    click eval_humanact "https://github.com/guytevet/motion-diffusion-model/blob/main/eval/eval_humanact12_uestc.py"
    click eval_uncon "https://github.com/guytevet/motion-diffusion-model/tree/main/eval/unconstrained/"
    click vis_dir "https://github.com/guytevet/motion-diffusion-model/tree/main/visualize/"
    click render "https://github.com/guytevet/motion-diffusion-model/blob/main/visualize/render_mesh.py"
    click joints "https://github.com/guytevet/motion-diffusion-model/tree/main/visualize/joints2smpl/"
    click hik "https://github.com/guytevet/motion-diffusion-model/blob/main/visualize/motions2hik.py"
    click simp "https://github.com/guytevet/motion-diffusion-model/blob/main/visualize/simplify_loc2rot.py"
    click vis_utils "https://github.com/guytevet/motion-diffusion-model/blob/main/visualize/vis_utils.py"
    click config "https://github.com/guytevet/motion-diffusion-model/blob/main/utils/config.py"

    %% Styles
    classDef data fill:#bfdeff,stroke:#336699
    classDef model fill:#ffd699,stroke:#cc9900
    classDef core fill:#bfffbf,stroke:#33aa33
    classDef output fill:#e0b3ff,stroke:#663399
    classDef external fill:#f2f2f2,stroke-dasharray: 5 5,stroke:#999999
```
