# haian-jin/lvsm architecture

> Architecture diagram of the GitHub repository haian-jin/lvsm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haian-jin/lvsm
- Repository: https://github.com/haian-jin/lvsm

## Overview

##### Project Type and Purpose
- This project is a machine learning research implementation for "LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias." It focuses on synthesizing novel views of scenes using deep learning.
- The repository includes both training and inference pipelines, supports distributed training (via torchrun), and processes real-world 3D scenes (with minimal 3D inductive bias), as described in the README.

##### Analyzing the File Structure
- Top-Level Files: Core files such as README.md, LICENSE.md, setup.py, and requirements.txt indicate the standard packaging and documentation setup.
- configs Directory: Contains YAML configuration files (e.g., LVSM_scene_decoder_only.yaml, LVSM_scene_encoder_decoder.yaml) that configure model parameters, training settings, API keys (Wandb integration), and other hyperparameters.
- data Directory: Holds scripts and assets for dataset loading, including scene-specific dataset python files and evaluation JSON indexes.
- model Directory: Contains the implementation of the model architectures, including two variants (decoder-only and encoder-decoder), the transformer backbone, and loss functions.
- Training and Inference Scripts: train.py and inference.py drive the respective stages, while process_data.py and generate_html.py are responsible for data preprocessing and result visualization.
- utils Directory: Offers helper modules for camera operations, data manipulation, metrics calculation, and training utilities. These encapsulate common functions across the pipeline.
- Other auxiliary configuration files (e.g., .pyrightconfig.json) help with development environment setup.

a. Main Components:
- Data Ingestion and Preprocessing: Represented by the "data" folder and process_data.py script. This component manages the transformation of raw RealEstate10K dataset into the required format.
- Training Pipeline: Captured by train.py; show how training inputs and configuration (from the configs YAML files) flow into this component. Emphasize the distributed training elements (torchrun, multi-GPU/multi-node setup) and batch size settings.
- Inference Pipeline and Evaluation: Shown via inference.py and generate_html.py; this module uses processed/pre-trained model checkpoints to perform scene synthesis, compute metrics, and produce visual HTML/video outputs.
- Utility Modules: Consolidate modules from the utils directory, link them to both training and inference pipelines. They support functions like camera calibration, data augmentation, and metric calculations.
- Configuration Management: Include a component for the config files (found in the configs directory) that feed parameters to training, inference, and logging (e.g., Wandb API keys).

b. Relationships and Data Flow:
- That the Model Architecture is central to both pipelines, with training feeding price parameters (via configuration) into the model components.
- That the Utils modules support the overall flow by connecting to both training and inference.

c. Architectural Patterns and Design Principles:
- Modularity by breaking down the project into data handling, model definition, training/inference pipelines, and utility functions.
- Use of distributed training principles where train.py leverages multiple GPUs and nodes.
- That the design supports two model variants (decoder-only vs. encoder-decoder), which may share many common components (e.g., transformer and loss functions), reflecting a flexible architecture.

d. Technologies and Frameworks:
- Python is the core language with dependencies managed via pip (requirements.txt) and conda.
- PyTorch is implied for model training and inference (using torchrun for distributed training).
- Configuration is managed through YAML files.
- Wandb is used for experiment tracking and logging (as described in the README).
- The overall design supports scalability (distributed training) and modularity (separate modules for data, methods, and utilities).

##### Elements
- Clear Labels: Each component (e.g., Data Preprocessing, Model – Decoder-Only/Encoder-Decoder, Training, Inference, Utilities, and Config Management) should be explicitly labeled.
- Core pipelines (training/inference)
- Auxiliary modules (data, utils)
- Configuration and external dependencies (configs, Wandb, distributed system)
- Even consider adding sub-components for distributed training (show multiple GPUs/nodes if needed) to indicate the scale of training operations.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        DS["Dataset Scene (data/dataset_scene.py)"]:::data
        EI["Evaluation Index (data/evaluation_index_re10k.json)"]:::data
        PD["Process Data (process_data.py)"]:::data
    end

    %% Configuration Management
    subgraph "Configuration Management"
        CFG["Configs Directory (configs)"]:::config
    end

    %% Model Architecture
    subgraph "Model Architecture"
        M1["LVSM Decoder Only (model/LVSM_scene_decoder_only.py)"]:::model
        M2["LVSM Encoder-Decoder (model/LVSM_scene_encoder_decoder.py)"]:::model
        M3["Transformer (model/transformer.py)"]:::model
        M4["Loss Calculation (model/loss.py)"]:::model
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        T2["Config Decoder Only (configs/LVSM_scene_decoder_only.yaml)"]:::training
        T3["Config Encoder-Decoder (configs/LVSM_scene_encoder_decoder.yaml)"]:::training
        T4["API Keys Config (configs/api_keys_example.yaml)"]:::training
        T1["Train Script (train.py)"]:::training
    end

    %% Inference Pipeline & Evaluation
    subgraph "Inference Pipeline & Evaluation"
        I1["Inference Script (inference.py)"]:::inference
        I2["Generate HTML (generate_html.py)"]:::inference
    end

    %% Utility Modules
    subgraph "Utility Modules"
        U1["Camera Utils (utils/camera_utils.py)"]:::utils
        U2["Data Utils (utils/data_utils.py)"]:::utils
        U3["Metric Utils (utils/metric_utils.py)"]:::utils
        U4["Training Utils (utils/training_utils.py)"]:::utils
    end

    %% External Integrations
    subgraph "External Integrations"
        DT["Distributed Training (torchrun)"]:::external
        WB["Wandb Logging"]:::external
    end

    %% Data Ingestion internal flow
    DS --> PD
    EI --> PD

    %% Configuration flows to pipelines
    CFG --> T2
    CFG --> T3
    CFG --> T4
    CFG --> T1
    CFG --> I1

    %% Data Ingestion feeds pipelines
    PD --> T1
    PD --> I1

    %% Training Pipeline internal flow
    T2 --> T1
    T3 --> T1
    T4 --> T1

    %% Pipelines use Model Architecture
    T1 --> M1
    T1 --> M2
    T1 --> M3
    T1 --> M4
    I1 --> M1
    I1 --> M2
    I1 --> M3
    I1 --> M4

    %% Utility Modules support pipelines
    U1 --> T1
    U2 --> T1
    U3 --> T1
    U4 --> T1
    U1 --> I1
    U2 --> I1
    U3 --> I1
    U4 --> I1

    %% External integrations for distributed training and logging (dashed arrows)
    T1 -.-> DT
    T1 -.-> WB

    %% Click Events
    click DS "https://github.com/haian-jin/lvsm/blob/main/data/dataset_scene.py"
    click EI "https://github.com/haian-jin/lvsm/blob/main/data/evaluation_index_re10k.json"
    click PD "https://github.com/haian-jin/lvsm/blob/main/process_data.py"
    click CFG "https://github.com/haian-jin/lvsm/tree/main/configs"
    click M1 "https://github.com/haian-jin/lvsm/blob/main/model/LVSM_scene_decoder_only.py"
    click M2 "https://github.com/haian-jin/lvsm/blob/main/model/LVSM_scene_encoder_decoder.py"
    click M3 "https://github.com/haian-jin/lvsm/blob/main/model/transformer.py"
    click M4 "https://github.com/haian-jin/lvsm/blob/main/model/loss.py"
    click T1 "https://github.com/haian-jin/lvsm/blob/main/train.py"
    click T2 "https://github.com/haian-jin/lvsm/blob/main/configs/LVSM_scene_decoder_only.yaml"
    click T3 "https://github.com/haian-jin/lvsm/blob/main/configs/LVSM_scene_encoder_decoder.yaml"
    click T4 "https://github.com/haian-jin/lvsm/blob/main/configs/api_keys_example.yaml"
    click I1 "https://github.com/haian-jin/lvsm/blob/main/inference.py"
    click I2 "https://github.com/haian-jin/lvsm/blob/main/generate_html.py"
    click U1 "https://github.com/haian-jin/lvsm/blob/main/utils/camera_utils.py"
    click U2 "https://github.com/haian-jin/lvsm/blob/main/utils/data_utils.py"
    click U3 "https://github.com/haian-jin/lvsm/blob/main/utils/metric_utils.py"
    click U4 "https://github.com/haian-jin/lvsm/blob/main/utils/training_utils.py"

    %% Styles
    classDef data fill:#D6EAF8,stroke:#3498DB,stroke-width:2px;
    classDef model fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
    classDef training fill:#FADBD8,stroke:#E74C3C,stroke-width:2px;
    classDef inference fill:#EBDEF0,stroke:#8E44AD,stroke-width:2px;
    classDef utils fill:#FCF3CF,stroke:#F1C40F,stroke-width:2px;
    classDef config fill:#D7BDE2,stroke:#8E44AD,stroke-width:2px;
    classDef external fill:#F0F3F4,stroke:#7F8C8D,stroke-dasharray:5 5,stroke-width:2px;
```
