# bcmi/light-a-video architecture

> Architecture diagram of the GitHub repository bcmi/light-a-video, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bcmi/light-a-video
- Repository: https://github.com/bcmi/light-a-video

## Overview

##### Project Type and Purpose
- This is an open-source, training-free video relighting framework (“Light-A-Video”).
- It provides zero-shot illumination control on arbitrary videos by orchestrating image-relighting models (IC-Light) and video-diffusion models (AnimateDiff, CogVideoX, Wan2.1), plus postprocessing (Progressive Light Fusion).

##### File Structure
- Top-level scripts (lav_relight.py, lav_paint.py, lav_cog_relight.py, lav_wan_relight.py, sam2.py) correspond to different workflows:
- `lav_relight.py`: frame-by-frame relighting with AnimateDiff
- `lav_paint.py`: segmentation + inpaint relighting
- `lav_cog_relight.py`: CogVideoX-based relighting
- `lav_wan_relight.py`: Wan2.1-based relighting
- `sam2.py`: generates foreground masks via SAM2
- `configs/`: YAML configs for each pipeline and each video example
- `input_*`, `masks_animatediff/`: data directories for raw videos and precomputed masks
- `models/`: placeholder for automatic download of pretrained checkpoints
- `src/`: implementation of modular “pipeline” classes—one per video-diffusion backbone and for the IC-Light relight model
- `utils/tools.py`: helper functions (video I/O, blending, attention manipulations)

##### README Insights
- Two key methodological blocks:
a. Consistent Light Attention (CLA) – cross-frame attention tweaks in the image relight model
b. Progressive Light Fusion (PLF) – linear blending strategy for temporal smoothness
- Supports three video diffusion backbones: AnimateDiff, CogVideoX, Wan2.1
- Automatic model downloads use the Hugging Face Diffusers ecosystem

##### Main Components
a. User CLI Layer:
- Scripts: lav_relight, lav_paint, lav_cog_relight, lav_wan_relight, sam2
- Config Loader: reads `configs/*.yaml`
b. Data Layer:
- Input Videos (`input_*/*.mp4`)
- Masks (`masks_animatediff/*`) or SAM2-generated masks
c. Model Layer:
- Image Relighting Model (IC-Light)
- Video-Diffusion Backbones (AnimateDiff, CogVideoX-2b, Wan2.1)
- Pretrained Repositories (Hugging Face downloads)
d. Pipeline Layer (`src/`):
- AnimatediffPipeline, InpaintPipeline, CogVideoPipeline, WanPipeline, IC-LightPipe
- Each pipeline integrates CLA modifications and calls Diffusers schedulers
e. Processing Modules:
- CLA Module (insertion into self-attention)
- PLF Module (per-frame blending with original video)
f. Utility Layer:
- tools.py: video decoding/encoding, mask I/O, blending functions
g. Output Layer:
- Relighted Videos → `output/output_mp4/`

##### Component Interactions and Data Flow
1. User invokes a CLI script with a YAML config.
2. Config Loader picks pipeline type and loads parameters.
3. If `lav_paint.py`: run `sam2.py` to generate masks → load masks.
4. Load input video frames and optional masks.
5. Instantiate the appropriate pipeline class from `src/`:
– Pipeline loads pretrained video-diffusion model & IC-Light model.
– Inject CLA into the attention blocks of IC-Light.
– For each frame (or frame sequence), run the diffusion model to generate relit frames.
– Apply PLF: linear blend of relit frame + original frame for temporal smoothness.
6. Write frames to video file via utilities → final MP4 in `output/`.

##### Architectural Patterns & Principles
- Strategy Pattern: multiple pipeline implementations behind a unified CLI interface.
- Configuration-Driven: all experiment parameters (model names, schedulers, blending weights) in YAML.
- Modular Layers: clear separation of CLI, data I/O, model orchestration, postprocessing, utilities.
- Plugin-Style Model Download: automatic fetching from Hugging Face.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User CLI Layer
    subgraph "User CLI Layer"
        LR["lav_relight.py"]:::cli
        LP["lav_paint.py"]:::cli
        LC["lav_cog_relight.py"]:::cli
        LW["lav_wan_relight.py"]:::cli
        SAM["sam2.py"]:::cli
    end

    %% Config Loader & Configs
    CL["Config Loader"]:::cli
    subgraph "Config Files"
        CR["configs/relight/*.yaml"]:::data
        CPI["configs/relight_inpaint/*.yaml"]:::data
        CCR["configs/cog_relight/*.yaml"]:::data
        CWR["configs/wan_relight/*.yaml"]:::data
    end

    %% Data Layer
    subgraph "Data Layer"
        IA["input_animatediff/*.mp4"]:::data
        ICog["input_cog/*.mp4"]:::data
        IW["input_wan/*.mp4"]:::data
        MasksAD["masks_animatediff/"]:::data
    end

    %% Pipeline Layer
    subgraph "Pipeline Layer"
        AP["AnimatediffPipeline"]:::pipeline
        IP["InpaintPipeline"]:::pipeline
        CP["CogVideoPipeline"]:::pipeline
        WP["WanPipeline"]:::pipeline
        ILP["IC-LightPipe"]:::pipeline
    end

    %% Model Layer
    subgraph "Model Layer"
        ICModel["src/ic_light.py"]:::model
        ICPipe["src/ic_light_pipe.py"]:::model
        ED["src/animatediff_eul.py"]:::model
        ADPipe["src/animatediff_pipe.py"]:::model
        AIPipe["src/animatediff_inpaint_pipe.py"]:::model
        CDDIM["src/cogvideo_ddim.py"]:::model
        CPipe["src/cogvideo_pipe.py"]:::model
        WFlow["src/wan_flowmatch.py"]:::model
        WPipe["src/wan_pipe.py"]:::model
        MDL["models/model_download_here"]:::external
    end

    %% Processing Modules
    CLA["Consistent Light Attention"]:::utils
    PLF["Progressive Light Fusion"]:::utils

    %% Utility Layer
    Tools["utils/tools.py"]:::utils

    %% Output Layer
    Out["output/output_mp4/"]:::output

    %% External Dependencies
    subgraph "External Dependencies"
        HF["Hugging Face / Diffusers"]:::external
    end

    %% Connections
    LR -->|invoke| CL
    LP -->|invoke| CL
    LC -->|invoke| CL
    LW -->|invoke| CL
    LP -->|requires masks| SAM
    SAM -->|generate masks| MasksAD
    CL -->|load params| CR
    CL -->|"load params"| CPI
    CL -->|"load params"| CCR
    CL -->|"load params"| CWR

    CL -->|select strategy| AP
    CL -->|select strategy| IP
    CL -->|select strategy| CP
    CL -->|select strategy| WP

    AP -->|load video| IA
    CP -->|load video| ICog
    WP -->|load video| IW
    IP -->|load masks| MasksAD

    AP --> ICPipe
    IP --> ICPipe
    CP --> ICPipe
    WP --> ICPipe
    ICPipe --> ICModel

    AP --> ADPipe
    IP --> AIPipe
    CP --> CPipe
    WP --> WPipe

    ADPipe --> ED
    AIPipe --> ED
    CPipe --> CDDIM
    WPipe --> WFlow

    AP -->|download| MDL
    IP -->|download| MDL
    CP -->|download| MDL
    WP -->|download| MDL
    ICPipe -->|download| MDL

    AP --> CLA
    IP --> CLA
    CP --> CLA
    WP --> CLA

    AP --> PLF
    IP --> PLF
    CP --> PLF
    WP --> PLF

    CLA --> Tools
    PLF --> Tools

    Tools -->|write frames| Out

    MDL --> HF
    HF --> ED
    HF --> CDDIM
    HF --> WFlow
    HF --> ICModel

    %% Click Events
    click LR "https://github.com/bcmi/light-a-video/blob/main/lav_relight.py"
    click LP "https://github.com/bcmi/light-a-video/blob/main/lav_paint.py"
    click LC "https://github.com/bcmi/light-a-video/blob/main/lav_cog_relight.py"
    click LW "https://github.com/bcmi/light-a-video/blob/main/lav_wan_relight.py"
    click SAM "https://github.com/bcmi/light-a-video/blob/main/sam2.py"
    click CR "https://github.com/bcmi/light-a-video/blob/main/configs/relight/*.yaml"
    click CPI "https://github.com/bcmi/light-a-video/blob/main/configs/relight_inpaint/*.yaml"
    click CCR "https://github.com/bcmi/light-a-video/blob/main/configs/cog_relight/*.yaml"
    click CWR "https://github.com/bcmi/light-a-video/blob/main/configs/wan_relight/*.yaml"
    click ICModel "https://github.com/bcmi/light-a-video/blob/main/src/ic_light.py"
    click ICPipe "https://github.com/bcmi/light-a-video/blob/main/src/ic_light_pipe.py"
    click ED "https://github.com/bcmi/light-a-video/blob/main/src/animatediff_eul.py"
    click ADPipe "https://github.com/bcmi/light-a-video/blob/main/src/animatediff_pipe.py"
    click AIPipe "https://github.com/bcmi/light-a-video/blob/main/src/animatediff_inpaint_pipe.py"
    click CDDIM "https://github.com/bcmi/light-a-video/blob/main/src/cogvideo_ddim.py"
    click CPipe "https://github.com/bcmi/light-a-video/blob/main/src/cogvideo_pipe.py"
    click WFlow "https://github.com/bcmi/light-a-video/blob/main/src/wan_flowmatch.py"
    click WPipe "https://github.com/bcmi/light-a-video/blob/main/src/wan_pipe.py"
    click Tools "https://github.com/bcmi/light-a-video/blob/main/utils/tools.py"

    %% Styles
    classDef cli fill:#D0E6FB,stroke:#333,stroke-width:1px
    classDef pipeline fill:#A0C4FF,stroke:#333,stroke-width:1px
    classDef model fill:#FFD6A5,stroke:#333,stroke-width:1px
    classDef data fill:#FDFFB6,stroke:#333,stroke-width:1px
    classDef utils fill:#BDB2FF,stroke:#333,stroke-width:1px
    classDef external fill:#C4FCEF,stroke:#333,stroke-width:1px
    classDef output fill:#CAFFBF,stroke:#333,stroke-width:1px
```
