# aleafy/relightvid architecture

> Architecture diagram of the GitHub repository aleafy/relightvid, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aleafy/relightvid
- Repository: https://github.com/aleafy/relightvid

## Overview

##### Project type and purpose
- This is an open-source, CLI-driven Python framework for **temporally consistent video relighting** using diffusion models. It is not a web or full-stack app but a research-oriented tool that assembles pre-trained models into a pipeline.

##### File structure
- Root
- inference.py ← CLI entry point, orchestrates everything
- configs/inference_fbc.yaml ← runtime parameters
- requirements.txt
- assets/ ← sample videos, masks, backgrounds
- pipelines/
- diffusion.py ← diffusion scheduler & sampler
- relightvid.py ← relighting workflow
- inference.py ← pipeline driver called by CLI
- modules/
- kl_autoencoder/autoencoder.py
- vqvae/autoencoder.py & model.py
- openclip/modules.py ← embedding/text conditioning
- video_unet_temporal/… ← temporal U-Net, attention & motion modules
- utils/
- video_utils.py, image_utils.py ← I/O & preprocessing
- load_utils.py, model_utils.py ← weight loading & model construction

##### README for additional insights
- Inputs: original video, per-frame masks, background video (or text/HDR in the future)
- Pre-trained models: RealisticVision-v51 (diffusion base), IC-Light, RelightVid weights (motion modules)
- Key emphasis on **frame-to-frame consistency** via temporal modules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI & Config Layer
    subgraph "CLI & Config" 
        direction TB
        CLI["inference.py"]:::cli
        Config["configs/inference_fbc.yaml"]:::cli
    end

    %% Core Orchestration Layer
    subgraph "Orchestration (pipelines)" 
        direction TB
        PipelineDriver["pipelines/inference.py"]:::model
        RelightWorkflow["pipelines/relightvid.py"]:::model
    end

    %% Data I/O Layer
    subgraph "Data Ingestion & Preprocessing" 
        direction TB
        DataIO["Video/Mask/BG Reader"]:::io
        Preprocessor["Frame/Image Preprocessor"]:::io
    end

    %% Model Loading Layer
    subgraph "Model Loader" 
        direction TB
        LoadUtils["utils/load_utils.py"]:::model
        ModelUtils["utils/model_utils.py"]:::model
        ExternalRepo(("HuggingFace / models/")):::ext
    end

    %% Core Computation Pipeline
    subgraph "Core Pipeline" 
        direction TB
        DiffusionScheduler["pipelines/diffusion.py"]:::model
        subgraph "Temporal U-Net" 
            direction TB
            UNet["unet.py"]:::model
            UNetBlocks["unet_blocks.py"]:::model
            Attention["attention.py"]:::model
            MotionModule["motion_module.py"]:::model
            ResNetBackbone["resnet.py"]:::model
        end
        subgraph "Autoencoders" 
            direction TB
            KLVAE["kl_autoencoder/autoencoder.py"]:::model
            VQVAEEnc["vqvae/autoencoder.py"]:::model
            VQVAEModel["vqvae/model.py"]:::model
        end
        TextConditioning["openclip/modules.py"]:::model
    end

    %% Postprocessing Layer
    subgraph "Postprocessing & Output" 
        direction TB
        Postprocessor["utils/video_utils.py"]:::io
        ImageSaver["utils/image_utils.py"]:::io
        VideoWriter["utils/video_utils.py"]:::io
    end

    %% External Dependencies
    ExternalRepo---|provides weights|LoadUtils

    %% Flows
    CLI -->|loads| Config
    CLI -->|invokes| PipelineDriver
    Config -->|params| PipelineDriver

    PipelineDriver -->|reads| DataIO
    DataIO -->|frames| Preprocessor

    Preprocessor -->|preprocessed frames| DiffusionScheduler
    Preprocessor -->|preprocessed frames| UNet

    LoadUtils -->|loads weights| DiffusionScheduler
    LoadUtils -->|loads weights| UNet
    LoadUtils -->|loads weights| KLVAE
    LoadUtils -->|loads weights| VQVAEEnc
    LoadUtils -->|loads weights| TextConditioning

    ModelUtils -->|builds models| DiffusionScheduler
    ModelUtils -->|builds models| UNet
    ModelUtils -->|builds models| Attention
    ModelUtils -->|builds models| MotionModule
    ModelUtils -->|builds models| KLVAE
    ModelUtils -->|builds models| VQVAEModel
    ModelUtils -->|builds models| TextConditioning

    DiffusionScheduler -->|samples| UNet
    UNet -->|latent features| KLVAE
    KLVAE -->|decoded frames| Postprocessor
    VQVAEEnc -->|encoded| VQVAEModel
    TextConditioning -->|conditioning| UNet

    Postprocessor -->|processed frames| VideoWriter

    PipelineDriver -->|orchestrates| RelightWorkflow
    RelightWorkflow -->|invokes| DiffusionScheduler
    RelightWorkflow -->|invokes| UNet
    RelightWorkflow -->|invokes| KLVAE

    %% Click Events
    click CLI "https://github.com/aleafy/relightvid/blob/main/inference.py"
    click Config "https://github.com/aleafy/relightvid/blob/main/configs/inference_fbc.yaml"
    click PipelineDriver "https://github.com/aleafy/relightvid/blob/main/pipelines/inference.py"
    click RelightWorkflow "https://github.com/aleafy/relightvid/blob/main/pipelines/relightvid.py"
    click DataIO "https://github.com/aleafy/relightvid/blob/main/utils/video_utils.py"
    click Preprocessor "https://github.com/aleafy/relightvid/blob/main/utils/image_utils.py"
    click LoadUtils "https://github.com/aleafy/relightvid/blob/main/utils/load_utils.py"
    click ModelUtils "https://github.com/aleafy/relightvid/blob/main/utils/model_utils.py"
    click DiffusionScheduler "https://github.com/aleafy/relightvid/blob/main/pipelines/diffusion.py"
    click UNet "https://github.com/aleafy/relightvid/blob/main/modules/video_unet_temporal/unet.py"
    click UNetBlocks "https://github.com/aleafy/relightvid/blob/main/modules/video_unet_temporal/unet_blocks.py"
    click Attention "https://github.com/aleafy/relightvid/blob/main/modules/video_unet_temporal/attention.py"
    click MotionModule "https://github.com/aleafy/relightvid/blob/main/modules/video_unet_temporal/motion_module.py"
    click ResNetBackbone "https://github.com/aleafy/relightvid/blob/main/modules/video_unet_temporal/resnet.py"
    click KLVAE "https://github.com/aleafy/relightvid/blob/main/modules/kl_autoencoder/autoencoder.py"
    click VQVAEEnc "https://github.com/aleafy/relightvid/blob/main/modules/vqvae/autoencoder.py"
    click VQVAEModel "https://github.com/aleafy/relightvid/blob/main/modules/vqvae/model.py"
    click TextConditioning "https://github.com/aleafy/relightvid/blob/main/modules/openclip/modules.py"
    click VideoWriter "https://github.com/aleafy/relightvid/blob/main/utils/video_utils.py"

    %% Styles
    classDef cli fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef io fill:#FFE5B4,stroke:#EA7F00,color:#A45B00;
    classDef model fill:#D4F4DD,stroke:#17A74B,color:#0F5132;
    classDef ext stroke-dasharray: 5 5,fill:none,stroke:#888888,color:#888888;
    class CLI,Config cli
    class DataIO,Preprocessor,Postprocessor,ImageSaver,VideoWriter io
    class PipelineDriver,RelightWorkflow,LoadUtils,ModelUtils,DiffusionScheduler,UNet,UNetBlocks,Attention,MotionModule,ResNetBackbone,KLVAE,VQVAEEnc,VQVAEModel,TextConditioning model
    class ExternalRepo ext
```
