How guytevet/motion-diffusion-model works
Project type and purpose
• This is a research‑oriented full‑stack ML codebase implementing the “Human Motion Diffusion Model” (MDM).
• It supports three core pipelines:
a. Text‑to‑Motion (HumanML3D, KIT)
b. Action‑to‑Motion (UESTC, HumanAct12)
c. Unconstrained motion generation
• It offers data preparation, model definitions, training, sampling (inference), evaluation, and visualization.
File structure
• Top‑level directories:
• data_loaders — adapters for each dataset
• diffusion — core diffusion scheduler, losses, utilities
• model — network architectures (MDM, BERT encoder, SMPL utilities)
• train — training entry points and loops
• sample — inference scripts (generate, edit, predict)
• eval — evaluation scripts and metrics for each task
• prepare — data/model/download helpers
• utils — shared functions (config, logging, distributed, rotations)
• Configuration files: environment.yml, cog.yaml, DiP.md (DiP extension)
• Storage: assets/, dataset/, body_models/, save/ (user‑provided at runtime)
Key README insights
• Core dependencies: PyTorch, CUDA, CLIP, SMPL, SMPL‑X, PyTorch3D
• Multi‑stage modeling: diffusion process with configurable steps, text encoder choices (CLIP or BERT), optional EMA
• Three usage modes: synthesis, editing, training, evaluation, rendering
Main components
• A. Data Layer
• Raw datasets (HumanML3D, KIT, UESTC, HumanAct12)
• Scripts to download and preprocess (prepare/*)
• B. Data Loader Layer
• data_loaders/*: transforms raw files → tensors, skeleton conversions, quaternions
• C. Model Layer
• model/… (mdm diffusion nets, BERT encoder, SMPL utilities)
• diffusion/* (noise schedule, sampling, losses)
• D. Training Pipeline
• train/train_mdm.py, training_loop.py, train_platforms.py
• Inputs: data loader, model, optimizer, config
• Outputs: checkpoints in save/
• E. Inference Pipeline
• sample/generate.py, edit.py, predict.py
• Loads checkpoint → runs reverse diffusion → outputs results.npy, stick‑figure MP4s
• F. Evaluation Pipeline
• eval/*: metric calculators (FID, diversity, accuracy, recognition)
• Loads generated data + ground truth → computes and logs metrics
• G. Visualization Pipeline
• H. External Dependencies
• CLIP model (one call cached)
• SMPL/SMPL‑X anthropometric models
• GPU/CUDA environment
• Optional WandB/TensorBoard for logging
Relationships & interactions
• Data downloads (prepare) → Dataset storage → Data Loaders → Training / Inference
• Training writes checkpoints → Sampling/Editing loads checkpoints → Generates raw motion
• Sampling → Visualization for mesh rendering
• Generated motions + GT → Evaluation metrics
• Utils (config, logging, distributed) used by all pipelines
Architectural patterns & design principles
• Config‑driven: single utils/config.py driving CLI across modules
• Reusable components: BERT/CLIP encoders pluggable via --text_encoder_type
• Extensible pipelines: each task has its own eval and data_loader submodule
• Lightweight shell scripts for dependency and data management (prepare/)