# haimsaw/lora-mdm architecture

> Architecture diagram of the GitHub repository haimsaw/lora-mdm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haimsaw/lora-mdm
- Repository: https://github.com/haimsaw/lora-mdm

## Overview

##### Project type and purpose
- This is a deep-learning research codebase implementing “Dance Like a Chicken: Low-Rank Stylization for Human Motion Diffusion” (LoRA-MDM).
- It is not a typical web full-stack app but a modular machine-learning framework for text-to-motion synthesis, style adaptation (LoRA), training, sampling, evaluation, and visualization of human motion data.

##### File structure
- Top-level directories:
- assets: sample prompts, precomputed action names
- data_loaders: dataset abstractions for A2M, HumanML3D, style data
- diffusion: core diffusion algorithms, losses, utilities
- model: model definitions (MDM, BERT encoder, SMPL)
- lora_pytorch: LoRA adapter implementation
- train: training entrypoints, loops, platform integrations
- sample: motion-generation scripts (generate, edit, predict)
- eval: evaluation scripts (style classifier, motion metrics)
- prepare: download-and-setup scripts
- utils: common utilities (config, distribution, seed, rotation conversions)
- Config files and environment:
- environment.yml, cog.yaml, .gitignore, LICENSE, README.md
- Patterns:

##### Insights from the README
- Explains dependencies (PyTorch, SMPL, CLIP, datasets)
- Describes workflows for setup, training MDM, fine-tuning LoRA, sampling, rendering.
- Mentions external services/datasets: SMooDi (100STYLE), HumanML3D, pretrained checkpoints, SMPLify.

a. Main components
1. Data Ingestion Layer
- assets/ (example prompts/texts)
- data_loaders/ (A2M, HumanML3D, style)
2. Model Layer
- model/ (MDM core networks, BERT encoder, SMPL utils)
- lora_pytorch/ (LoRA adapters)
3. Diffusion Engine
- diffusion/ (Gaussian diffusion, scheduler, losses)
4. Orchestration / CLI
- train/ (training_loop, train_mdm, train_platforms)
- sample/ (generate, ddim_invert, edit)
- eval/ (eval_humanml, eval_lora_mdm, style_classifier)
5. Utilities & Configuration
- utils/ (config parsing, distributed util, seed fixing)
6. External Dependencies
- SMPL, CLIP, PyTorch3D, spaCy, ffmpeg
b. Relationships and interactions
- Data flows from assets & external datasets → data_loaders → batched input
- Batches pass to model + diffusion engine
- During training: train script invokes diffusion steps, uses LoRA adapters if fine-tuning, writes checkpoints to save/
- During sampling: sample script loads model checkpoints, runs reverse diffusion, outputs .npy + .mp4
- Evaluation: eval scripts load generated data + models → compute metrics → output scores
- Visualization: render_mesh (SMPLify) reads stick-figure .mp4 → outputs SMPL .obj per frame
- Utilities manage config, seed, distributed training, logging
c. Architectural patterns & design principles
- Modular design: each concern (data, model, training, sampling, evaluation, visualization) isolated
- Plugin/extensibility: new dataset or style adapter can be added under data_loaders or lora_pytorch
- CLI-driven workflow: each major stage is a separate command module for reproducibility
- Low-rank adapter (LoRA) pattern: dynamically augmenting base MDM model without full retraining
d. Key technologies & frameworks
- PyTorch (core ML)
- spaCy (text preprocessing)
- OpenAI CLIP (optional text-vision features)
- SMPL/SMPLify (3D human mesh fitting)
- PyTorch3D (3D processing)
- ffmpeg (video encoding)

##### Detail guidance
- Diffusion scheduler and loss module as part of the Diffusion Engine
- Configuration & seed utilities as a sidebar, indicating their injection into every stage

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        Assets["Assets"]:::data
        DataLoaders["Data Loaders"]:::data
        Preparation["Dataset Preparation Scripts"]:::data
    end
    click Assets "https://github.com/haimsaw/lora-mdm/tree/main/assets/"
    click DataLoaders "https://github.com/haimsaw/lora-mdm/tree/main/data_loaders/"
    click Preparation "https://github.com/haimsaw/lora-mdm/tree/main/prepare/"

    %% Model Layer
    subgraph "Model Layer"
        CoreModel["Core Networks"]:::model
        LoRAAdapter["LoRA Adapter"]:::model
    end
    click CoreModel "https://github.com/haimsaw/lora-mdm/tree/main/model/"
    click LoRAAdapter "https://github.com/haimsaw/lora-mdm/tree/main/lora_pytorch/"

    %% Diffusion Engine
    Diffusion["Diffusion Engine"]:::engine
    click Diffusion "https://github.com/haimsaw/lora-mdm/tree/main/diffusion/"

    %% Orchestration / CLI
    subgraph "Orchestration / CLI"
        Trainer["Training CLI"]:::cli
        Sampler["Sampling CLI"]:::cli
        Evaluator["Evaluation CLI"]:::cli
        VisualizerCLI["Visualization CLI"]:::cli
    end
    click Trainer "https://github.com/haimsaw/lora-mdm/tree/main/train/"
    click Sampler "https://github.com/haimsaw/lora-mdm/tree/main/sample/"
    click Evaluator "https://github.com/haimsaw/lora-mdm/tree/main/eval/"
    click VisualizerCLI "https://github.com/haimsaw/lora-mdm/tree/main/visualize/"

    %% Utilities & External
    Utilities["Utilities & Config"]:::utils
    click Utilities "https://github.com/haimsaw/lora-mdm/tree/main/utils/"
    External["External Dependencies"]:::external

    %% Checkpoints and Outputs
    Checkpoints["Model Checkpoints"]:::storage
    Outputs["Generated Outputs"]:::storage

    %% Data Flow
    Assets --> DataLoaders
    Preparation --> DataLoaders
    DataLoaders --> CoreModel
    DataLoaders --> Diffusion
    CoreModel --> Diffusion
    LoRAAdapter --> Diffusion
    CoreModel <--> LoRAAdapter

    %% Training Flow
    Diffusion --> Trainer
    Trainer --> Checkpoints

    %% Post-Training Flow
    Checkpoints --> Sampler
    Sampler --> Outputs
    Outputs --> Evaluator
    Outputs --> VisualizerCLI

    %% Utilities & External Injection
    Utilities --> Trainer
    Utilities --> Sampler
    Utilities --> Evaluator
    Utilities --> VisualizerCLI
    External -.-> DataLoaders
    External -.-> CoreModel
    External -.-> Diffusion
    External -.-> Trainer
    External -.-> Sampler
    External -.-> Evaluator
    External -.-> VisualizerCLI

    %% Styles
    classDef data fill:#E3F2FD,stroke:#90CAF9,color:#0D47A1;
    classDef model fill:#E8F5E9,stroke:#66BB6A,color:#1B5E20;
    classDef engine fill:#F1F8E9,stroke:#AED581,color:#33691E;
    classDef cli fill:#FFF3E0,stroke:#FFB74D,color:#E65100;
    classDef utils fill:#ECEFF1,stroke:#90A4AE,color:#37474F;
    classDef external stroke-dasharray: 5 5,stroke:#BDBDBD,color:#424242;
    classDef storage fill:#F3E5F5,stroke:#BA68C8,color:#4A148C;
```
