# esdolo/freevs architecture

> Architecture diagram of the GitHub repository esdolo/freevs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/esdolo/freevs
- Repository: https://github.com/esdolo/freevs

## Overview

##### Main Components
- Data Ingestion & Pre-processing:
– Components handling Waymo raw data inputs (e.g., TFRecord files).
– Scripts for extracting ground truth images (extract_gt_images.py) and generating pseudo-images (lidarproj_halfreso_multiframe.py and its mismatch-augmentation variant).
– Data processing scripts (e.g., waymo_data_generation_subsegbycampos_multiframe.py) that combine GT and pseudo images into pickle files for downstream consumption.
- Model & Pipeline Components:
– The core diffusion model components (e.g., models, pipelines, schedulers within diffusers/src/diffusers).
– Specific pipelines for stable video diffusion (pipeline_stable_video_diffusion) and additional free view synthesis extensions (inside examples/freevs and diffusers/examples/freevs).
– The “FreeVS” components including encoder modules (convnext, swin_transformer, unified_encoder) and custom UNet variants (unet_spatial_temporal_condition_custom) in the examples module.
- Training & Inference Modules:
– Training scripts (scripts/run_train.sh, run_train_onlyunet.sh) which show both joint training (encoder and diffusion) and training with a frozen pseudo-image encoder.
– Inference scripts (inference_svd.py) that load the pretrained or newly trained model to synthesize views.
- External Dependencies & Asset Management:
– Pretrained checkpoints (from Hugging Face hub) and local pretrained directories (pretrained/layout_encoder, pretrained for SVD).
– Integration with Hugging Face Accelerate and Hub modules for model/version management.
– External data sources like the Waymo Open Dataset.

##### Relationships & Data Flow
– Start with external inputs: Waymo dataset enters via data ingestion.
– The output (pickle files containing processed scene information) flows into the Training module.
– Training interacts with model pipelines: diffusion model architectures and scheduler modules (from diffusers/src/diffusers/models and schedulers).
– Finally, the trained model is used by the Inference module, which outputs synthesized views (rendered images/videos).
– Add connections to external services: Hugging Face hub for downloading pretrained models and executing CLI commands.

##### Architectural Patterns & Key Technologies
– Emphasize the modular design of the diffusers library: separation between model definitions (autoencoders, UNets, transformers) and pipelines.
– Show usage of a microservices-like or component-based approach; data processing, training, and inference are separate but interact via clear interfaces (e.g., pickle files, configuration YAMLs).
– Note extensions for video synthesis, where stable diffusion models are adapted for temporal (video) tasks.
– Underline that the project uses shell scripts, Python entry points (CLI commands), and configuration files (e.g., fsdp.yaml) to manage distributed training.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global External Input
    A["Waymo Raw Data"]:::ext

    %% Data Ingestion & Pre-processing Subgraph
    subgraph "Data Ingestion & Pre-processing"
        B["Extract GT Images (extract_gt_images.py)"]:::data
        C["Pseudo-image Generation (lidarproj_halfreso_multiframe.py)"]:::data
        D["Pseudo-image Generation (Mismatch Augmentation) (lidarproj_halfreso_multiframe_mismatchframeaug.py)"]:::data
        E["Data Generation (waymo_data_generation_subsegbycampos_multiframe.py)"]:::data
        F["Processed Dataset (Pickle Files)"]:::data
    end

    %% Model & Pipeline Components Subgraph
    subgraph "Model & Pipeline Components"
        G1["ConvNeXT Encoder (convnext.py)"]:::model
        G2["Swin Transformer Encoder (swin_transformer.py)"]:::model
        G3["Unified Encoder (unified_encoder.py)"]:::model
        H["Custom UNet (unet_spatial_temporal_condition_custom.py)"]:::model
        I["Stable Video Diffusion Pipeline (pipeline_stable_video_diffusion_custom.py)"]:::model
    end

    %% Training Modules Subgraph
    subgraph "Training Modules"
        J1["Training Script (run_train.sh)"]:::train
        J2["Training Script (run_train_onlyunet.sh)"]:::train
        J3["Training Script (train_svd.py)"]:::train
        J4["Training Script (train_svd_onlytrainunet.py)"]:::train
        K["Trained Model Checkpoints"]:::train
    end

    %% Inference Module
    L["Inference Pipeline – SVD (inference_svd.py)"]:::train

    %% External Dependencies & Asset Management Subgraph
    subgraph "External Dependencies & Asset Management"
        M["Pretrained Checkpoints (layout_encoder)"]:::ext
        N["Hugging Face Accelerate Config (default_config.yaml)"]:::ext
        O["Hugging Face Hub (hub/)"]:::ext
        P["Distributed Training Config (fsdp.yaml)"]:::ext
    end

    %% Data Flow Connections
    A --> B
    A --> E
    B --> F
    C --> F
    D --> F
    E --> F

    F --> J1
    F --> J2
    F --> J3
    F --> J4

    %% Model & Pipeline components used in training
    G1 --- J1
    G2 --- J1
    G3 --- J1
    H --- J1
    I --- J1

    %% Training Scripts converge into Checkpoints
    J1 --> K
    J2 --> K
    J3 --> K
    J4 --> K

    K --> L

    %% External Dependencies integration
    M --- K
    N --- K
    O --- K
    P --- K
    N --- L
    O --- L
    P --- L

    %% Styles
    classDef data fill:#f9e79f,stroke:#c0392b,stroke-width:2px;
    classDef model fill:#d5f5e3,stroke:#27ae60,stroke-width:2px;
    classDef train fill:#d6eaf8,stroke:#2980b9,stroke-width:2px;
    classDef ext fill:#fbeee6,stroke:#e67e22,stroke-width:2px;

    %% Click Events for Data Ingestion & Pre-processing
    click B "https://github.com/esdolo/freevs/blob/master/diffusers/waymo_process/extract_gt_images.py"
    click C "https://github.com/esdolo/freevs/blob/master/diffusers/waymo_process/lidarproj_halfreso_multiframe.py"
    click D "https://github.com/esdolo/freevs/blob/master/diffusers/waymo_process/lidarproj_halfreso_multiframe_mismatchframeaug.py"
    click E "https://github.com/esdolo/freevs/blob/master/diffusers/data_process/waymo_data_generation_subsegbycampos_multiframe.py"

    %% Click Events for Model & Pipeline Components
    click G1 "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/src/encoder/convnext.py"
    click G2 "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/src/encoder/swin_transformer.py"
    click G3 "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/src/encoder/unified_encoder.py"
    click H "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/src/model/unet_spatial_temporal_condition_custom.py"
    click I "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/src/pipelines/pipeline_stable_video_diffusion_custom.py"

    %% Click Events for Training Modules
    click J1 "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/scripts/run_train.sh"
    click J2 "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/scripts/run_train_onlyunet.sh"
    click J3 "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/train_svd.py"
    click J4 "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/train_svd_onlytrainunet.py"

    %% Click Event for Inference Module
    click L "https://github.com/esdolo/freevs/blob/master/diffusers/examples/freevs/inference_svd.py"

    %% Click Events for External Dependencies
    click M "https://github.com/esdolo/freevs/tree/master/diffusers/pretrained/layout_encoder/"
    click N "https://github.com/esdolo/freevs/blob/master/diffusers/huggingface/accelerate/default_config.yaml"
    click O "https://github.com/esdolo/freevs/tree/master/diffusers/huggingface/hub/"
    click P "https://github.com/esdolo/freevs/blob/master/diffusers/fsdp.yaml"
```
