# pananananas/mvd architecture

> Architecture diagram of the GitHub repository pananananas/mvd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pananananas/mvd
- Repository: https://github.com/pananananas/mvd

## Overview

##### Main Components
- Data Acquisition & Processing
- In the src/data folder you have modules for cleaning, downloading (including multiple downloaders for Objaverse, Omniobject, etc.), and rendering data.
- File “objaverse_dataset.py” which likely orchestrates dataset loading/preparation.
- Model Definition & Architecture
- In the src/models directory you see components such as:
▪ image_encoder.py (extracts image features),
▪ camera_encoder.py,
▪ attention.py,
▪ mvd_unet.py,
▪ pipeline.py (which might tie the different modules together).
- These modules indicate that the model uses multiple components (like UNET architecture, multi-view attention layers, etc.) for both image conditioning and view conditioning.
- Training & Validation
- The training code is in src/training (losses.py and training.py) along with train.py and train.sh.
- Validation scripts are present (validation.py, val.sh, validate_cross_attention.py), which suggests an end-to-end pipeline from dataset ingestion and training to model verification.
- Configuration & Experimentation
- The config directory with train_config.yaml, plus integration scripts (train.sh, pyproject.toml) and the README discussion about WandB, Hydra, and Lightning integration for multi-GPU training, indicate that configuration management and experiment logging play a role.

##### Relationships and Data Flow
- Processed data is consolidated into a dataset (handled by objaverse_dataset.py) that feeds into the training pipeline.
- The model components (e.g., image encoder, camera encoder, attention layers) are used within a pipeline (in pipeline.py) to perform image conditioning and multi-view attention-based generation.
- Training logic (src/training/training.py) consumes the dataset and computes losses (SSIM, PSNR, perceptual, geometric) as detailed in the TODO list.
- Validation scripts check the behavior of attention mechanisms and match features between generated and source images.
- Configuration files and scripts (e.g., config/train_config.yaml, train.sh) ensure that the training and evaluation experiments run consistently.

##### Architectural Patterns and Technical Stack
- The project follows a modular design separating data handling, model definitions, and training routines—a pattern common in machine learning projects.
- There is a separation of concerns: data processing (downloaders, cleaning, rendering), model creation (various encoders and custom attention layers), and training/validation.
- It also hints at integration with external systems (e.g., WandB for experiment tracking, Hydra for configuration management, and possibly PyTorch Lightning for multi-GPU training).

##### Final Detailed Steps
- Module from the file tree to a block (data modules, model modules, training/validation modules).
- Don’t forget to include configuration files (like config/train_config.yaml) and scripts (train.sh, val.sh) as auxiliary components that control how the main pipeline executes.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    e0("External Datasets"):::data
    subgraph "Data Layer"
        a1("Data Downloaders"):::data
        a2("Data Cleaning"):::data
        a3("Data Rendering"):::data
        a4("Dataset Preparation"):::data
    end

    %% Model Layer
    subgraph "Model Layer"
        m1("Image Encoder"):::model
        m2("Camera Encoder"):::model
        m3("Attention Mechanisms"):::model
        m4("Multi-View UNET"):::model
        m5("Pipeline Integration"):::model
    end

    %% Training Layer
    subgraph "Training Layer"
        t1("Main Training Script"):::training
        t2("Training Logic"):::training
        t3("Loss Calculation"):::training
        t5("Validation Shell Script"):::training
        t4("Validation Script"):::training
        t6("Cross-Attention Validation"):::training
    end

    %% Configuration & Experimentation
    subgraph "Configuration & Experimentation"
        c1("Train Config"):::config
        c2("PyProject"):::config
    end

    %% Relationships - Data Layer Flow
    e0 --> a1
    a1 --> a2
    a2 --> a3
    a3 --> a4

    %% Data to Model
    a4 --> m5

    %% Model Components Integration
    m1 --> m5
    m2 --> m5
    m3 --> m5
    m4 --> m5

    %% Model to Training
    m5 --> t1

    %% Configuration to Model and Training
    c1 --> m5
    c2 --> m5
    c1 --> t1
    c2 --> t1

    %% Training Layer Flow
    t1 --> t2
    t2 --> t3
    t2 --> t4
    t2 --> t6
    t1 --> t5

    %% Styles
    classDef data fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef model fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef training fill:#A9DFBF,stroke:#27AE60,stroke-width:2px;
    classDef config fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;

    %% Click Events - Data Layer
    click a1 "https://github.com/pananananas/mvd/tree/main/src/data/downloaders/"
    click a2 "https://github.com/pananananas/mvd/blob/main/src/data/cleaning/main.py"
    click a3 "https://github.com/pananananas/mvd/tree/main/src/data/rendering/"
    click a4 "https://github.com/pananananas/mvd/blob/main/src/data/objaverse_dataset.py"

    %% Click Events - Model Layer
    click m1 "https://github.com/pananananas/mvd/blob/main/src/models/image_encoder.py"
    click m2 "https://github.com/pananananas/mvd/blob/main/src/models/camera_encoder.py"
    click m3 "https://github.com/pananananas/mvd/blob/main/src/models/attention.py"
    click m4 "https://github.com/pananananas/mvd/blob/main/src/models/mvd_unet.py"
    click m5 "https://github.com/pananananas/mvd/blob/main/src/models/pipeline.py"

    %% Click Events - Training Layer
    click t1 "https://github.com/pananananas/mvd/blob/main/train.py"
    click t2 "https://github.com/pananananas/mvd/blob/main/src/training/training.py"
    click t3 "https://github.com/pananananas/mvd/blob/main/src/training/losses.py"
    click t4 "https://github.com/pananananas/mvd/blob/main/src/validation.py"
    click t5 "https://github.com/pananananas/mvd/blob/main/val.sh"
    click t6 "https://github.com/pananananas/mvd/blob/main/validate_cross_attention.py"

    %% Click Events - Configuration & Experimentation
    click c1 "https://github.com/pananananas/mvd/blob/main/config/train_config.yaml"
    click c2 "https://github.com/pananananas/mvd/blob/main/pyproject.toml"
```
