How neu-vi/smoodi works
Project type and purpose
• This is an open-source research toolkit and full-pipeline ML application in Python: “SMooDi,” a stylized motion diffusion model.
• It ingests motion-capture datasets (e.g., HumanML3D, 100STYLE), trains a diffusion model, evaluates it, and renders synthesized motions via Blender/FFmpeg.
File structure
• Top-level directories and scripts:
• configs/ – YAML configuration files (global + module-level)
• datasets/ – raw and processed data files and metadata
• mld/ – main Python package subdivided into:
· data/ – dataset definitions, loaders, sampling utilities
· transforms/ – motion representation and conversion utilities
· models/ – architecture definitions, model‐factory, losses, metrics, operators
· render/ – motion visualization (Blender integration, video output)
· tools/, utils/, callback/ – logging, run-ID management, training callbacks
• prepare/ – shell scripts to download dependencies, pretrained weights, SMPL models
• scripts/, demo.sh, train.sh, test.sh – user‐facing entry points for demo, training, evaluation
• train.py – Python entry point for training
• Configuration-driven: everything parameterized via YAML under configs/.
README for additional insights
• Describes data processing steps, environment setup (conda, ffmpeg, Blender), dataset preparation, model inference (“demo.sh”), training (“train.sh”), evaluation (“test.sh”).
• Notes external dependencies: PyTorch, SMPL/SMPL-X, PyTorch3D, OpenAI CLIP, Blender, ffmpeg.
Main components and their interactions
a. Configuration Manager
• Reads YAMLs from configs/, builds unified config object.
• Injects settings (datasets paths, model hyperparameters, training flags).
b. Data Pipeline
• mld/data/ loader modules (HumanML3D, 100STYLE, KIT, UESTC)
• Sampling modules produce batches via mld/data/sampling.
• Transform modules convert SMPL poses to network features.
c. Model Factory & Architectures
• mld/models/get_model.py uses config to instantiate specific architecture (VAE, diffusion, style encoder, text encoder).
• Loss modules compute reconstruction, KL, style/sampling losses.
• Metrics modules evaluate generation quality.
d. Training & Evaluation
• train.py orchestrates training loop:
– Initialize model, optimizer, dataloaders
– Register callbacks (progress logging, checkpointing)
– Compute losses and backprop
• test.sh uses test_cmld.py to run evaluation metrics with or without style guidance.
e. Inference & Demo
• demo.sh + demo_cmld.py: read text prompt and style motion, run diffusion inference, output motion files under demo/.
f. Rendering
• mld/render/ interfaces with Blender via mld/launch/blender.py
• Renders sequences into videos, applies materials, cameras, joints.
• scripts/visualize_motion*.sh wrap rendering for batch jobs.
g. External Dependencies
• Pretrained weights (MLD base model, CLIP, BERT, evaluators)
• SMPL/SMPL-X, PyTorch3D, Blender, ffmpeg
Architectural patterns and principles
• Modular package structure by concern (data, model, render, utils).
• Config-driven: decouples code from hyperparameters.
• Factory pattern for model instantiation.
• Callback pattern for training monitoring and checkpointing.
• Clear separation of training, evaluation, inference, rendering stages.