How bcmi/light-a-video works
Project Type and Purpose
• This is an open-source, training-free video relighting framework (“Light-A-Video”).
• It provides zero-shot illumination control on arbitrary videos by orchestrating image-relighting models (IC-Light) and video-diffusion models (AnimateDiff, CogVideoX, Wan2.1), plus postprocessing (Progressive Light Fusion).
File Structure
• Top-level scripts (lav_relight.py, lav_paint.py, lav_cog_relight.py, lav_wan_relight.py, sam2.py) correspond to different workflows:
• lav_relight.py: frame-by-frame relighting with AnimateDiff
• lav_paint.py: segmentation + inpaint relighting
• lav_cog_relight.py: CogVideoX-based relighting
• lav_wan_relight.py: Wan2.1-based relighting
• sam2.py: generates foreground masks via SAM2
• configs/: YAML configs for each pipeline and each video example
• input_*, masks_animatediff/: data directories for raw videos and precomputed masks
• models/: placeholder for automatic download of pretrained checkpoints
• src/: implementation of modular “pipeline” classes—one per video-diffusion backbone and for the IC-Light relight model
• utils/tools.py: helper functions (video I/O, blending, attention manipulations)
README Insights
• Two key methodological blocks:
a. Consistent Light Attention (CLA) – cross-frame attention tweaks in the image relight model
b. Progressive Light Fusion (PLF) – linear blending strategy for temporal smoothness
• Supports three video diffusion backbones: AnimateDiff, CogVideoX, Wan2.1
• Automatic model downloads use the Hugging Face Diffusers ecosystem
Main Components
a. User CLI Layer:
• Scripts: lav_relight, lav_paint, lav_cog_relight, lav_wan_relight, sam2
• Config Loader: reads configs/*.yaml
b. Data Layer:
• Input Videos (input_*/*.mp4)
• Masks (masks_animatediff/*) or SAM2-generated masks
c. Model Layer:
• Image Relighting Model (IC-Light)
• Video-Diffusion Backbones (AnimateDiff, CogVideoX-2b, Wan2.1)
• Pretrained Repositories (Hugging Face downloads)
d. Pipeline Layer (src/):
• AnimatediffPipeline, InpaintPipeline, CogVideoPipeline, WanPipeline, IC-LightPipe
• Each pipeline integrates CLA modifications and calls Diffusers schedulers
e. Processing Modules:
• CLA Module (insertion into self-attention)
• PLF Module (per-frame blending with original video)
f. Utility Layer:
• tools.py: video decoding/encoding, mask I/O, blending functions
g. Output Layer:
• Relighted Videos → output/output_mp4/
Component Interactions and Data Flow
1. User invokes a CLI script with a YAML config.
2. Config Loader picks pipeline type and loads parameters.
3. If lav_paint.py: run sam2.py to generate masks → load masks.
4. Load input video frames and optional masks.
5. Instantiate the appropriate pipeline class from src/:
– Pipeline loads pretrained video-diffusion model & IC-Light model.
– Inject CLA into the attention blocks of IC-Light.
– For each frame (or frame sequence), run the diffusion model to generate relit frames.
– Apply PLF: linear blend of relit frame + original frame for temporal smoothness.
6. Write frames to video file via utilities → final MP4 in output/.
Architectural Patterns & Principles
• Strategy Pattern: multiple pipeline implementations behind a unified CLI interface.
• Configuration-Driven: all experiment parameters (model names, schedulers, blending weights) in YAML.
• Modular Layers: clear separation of CLI, data I/O, model orchestration, postprocessing, utilities.
• Plugin-Style Model Download: automatic fetching from Hugging Face.