How haimsaw/lora-mdm works
Project type and purpose
• This is a deep-learning research codebase implementing “Dance Like a Chicken: Low-Rank Stylization for Human Motion Diffusion” (LoRA-MDM).
• It is not a typical web full-stack app but a modular machine-learning framework for text-to-motion synthesis, style adaptation (LoRA), training, sampling, evaluation, and visualization of human motion data.
File structure
• Top-level directories:
• assets: sample prompts, precomputed action names
• data_loaders: dataset abstractions for A2M, HumanML3D, style data
• diffusion: core diffusion algorithms, losses, utilities
• model: model definitions (MDM, BERT encoder, SMPL)
• lora_pytorch: LoRA adapter implementation
• train: training entrypoints, loops, platform integrations
• sample: motion-generation scripts (generate, edit, predict)
• eval: evaluation scripts (style classifier, motion metrics)
• prepare: download-and-setup scripts
• utils: common utilities (config, distribution, seed, rotation conversions)
• Config files and environment:
• environment.yml, cog.yaml, .gitignore, LICENSE, README.md
• Patterns:
Insights from the README
• Explains dependencies (PyTorch, SMPL, CLIP, datasets)
• Describes workflows for setup, training MDM, fine-tuning LoRA, sampling, rendering.
• Mentions external services/datasets: SMooDi (100STYLE), HumanML3D, pretrained checkpoints, SMPLify.
a. Main components
1. Data Ingestion Layer
• assets/ (example prompts/texts)
• data_loaders/ (A2M, HumanML3D, style)
2. Model Layer
• model/ (MDM core networks, BERT encoder, SMPL utils)
• lora_pytorch/ (LoRA adapters)
3. Diffusion Engine
• diffusion/ (Gaussian diffusion, scheduler, losses)
4. Orchestration / CLI
• train/ (training_loop, train_mdm, train_platforms)
• sample/ (generate, ddim_invert, edit)
• eval/ (eval_humanml, eval_lora_mdm, style_classifier)
5. Utilities & Configuration
• utils/ (config parsing, distributed util, seed fixing)
6. External Dependencies
• SMPL, CLIP, PyTorch3D, spaCy, ffmpeg
b. Relationships and interactions
• Data flows from assets & external datasets → data_loaders → batched input
• Batches pass to model + diffusion engine
• During training: train script invokes diffusion steps, uses LoRA adapters if fine-tuning, writes checkpoints to save/
• During sampling: sample script loads model checkpoints, runs reverse diffusion, outputs .npy + .mp4
• Evaluation: eval scripts load generated data + models → compute metrics → output scores
• Visualization: render_mesh (SMPLify) reads stick-figure .mp4 → outputs SMPL .obj per frame
• Utilities manage config, seed, distributed training, logging
c. Architectural patterns & design principles
• Modular design: each concern (data, model, training, sampling, evaluation, visualization) isolated
• Plugin/extensibility: new dataset or style adapter can be added under data_loaders or lora_pytorch
• CLI-driven workflow: each major stage is a separate command module for reproducibility
• Low-rank adapter (LoRA) pattern: dynamically augmenting base MDM model without full retraining
d. Key technologies & frameworks
• PyTorch (core ML)
• spaCy (text preprocessing)
• OpenAI CLIP (optional text-vision features)
• SMPL/SMPLify (3D human mesh fitting)
• PyTorch3D (3D processing)
• ffmpeg (video encoding)
Detail guidance
• Diffusion scheduler and loss module as part of the Diffusion Engine
• Configuration & seed utilities as a sidebar, indicating their injection into every stage