How leolinpotato/drive works
Project type and purpose
• DRIVE is a command‐line and script-driven video‐editing pipeline (not a webapp—although a minimal Web UI exists) designed for instance-level control. It glues together three major subcomponents:
• A preprocessing/tracking stage (mask generation via SAMURAI or similar).
• A diffusion–denoising engine (built on the RAVE codebase plus EasyInv for inversion).
• A harmonization post‐processing stage (the Harmonizer codebase).
File structure
• Top‐level directories:
• Harmonizer/ – Contains the “Harmonizer” module: model code, training code, pre‐trained weights, video & image demos.
• RAVE/ – Contains the RAVE diffusion engine, a large “annotator” collection of vision submodules, YAML configs, pipelines/, scripts/, main.py, webui.py.
• configs/, data/, pipelines/, scripts/, utils/ – Orchestrate the DRIVE pipeline at a high level.
• Key files:
• main.py & webui.py – Entry points for CLI and minimal UI.
• pipelines/*.py – Glue code that sequences preprocessing, denoising, composition, harmonization.
• video_harmonization.sh – Shell wrapper that invokes Harmonizer/demo/video_harmonization/run.py.
• configs/*.yaml – Parameterize mask/denoise/harmonize settings.
• Harmonizer/pretrained & RAVE/annotator/* – Store pretrained models and vision backbones.
README for additional insights
• The methodology section breaks the pipeline into five clear stages: Preprocessing, Denoising, Composition, Harmonization, Detail Refinement (EasyInv).
• External dependencies: SAMURAI for masks; EasyInv for DDIM inversion; PyTorch & associated vision libraries.
a. Main components
1. User Interface Layer
• CLI (main.py, scripts/run_experiment*.py)
• Notebook (demo_notebook.ipynb)
• Web UI (webui.py)
2. Configuration Store
• YAML configs under configs/
3. Preprocessing Module
• Mask generator (external: SAMURAI)
• Preprocessor utils (RAVE/utils/preprocesser_utils.py)
4. Diffusion Engine (RAVE)
• Core denoising (RAVE/annotator + pipelines orchestrator)
• EasyInv inversion logic integrated into pipelines
5. Composition Module
• Mask‐guided compositional denoising formula
6. Harmonization Module (Harmonizer)
• Foreground‐background blending (Harmonizer/src/model, demo/video_harmonization)
7. Output Store
• Edited video frames/video files in data/ or demo output folders
b. Relationships and interactions
• UI triggers pipeline entry point → loads configs → calls Preprocessing → invokes Diffusion Engine → produces latent or frame outputs → feeds Composition logic → hands off to Harmonization → writes final video.
• Preprocessing writes masked frames to disk or memory, consumed by diffusion.
• Diffusion Engine may call external EasyInv routines.
• Harmonizer/demo scripts read foreground, background, mask videos to refine and harmonize.
c. Architectural patterns and principles
• Pipeline (staged‐line) pattern: each stage is a self‐contained module with a clear input/output contract (e.g., frames + masks → denoised latents → composited frames → harmonized video).
• Plugin/annotator pattern in RAVE: lots of interchangeable vision modules (backbones, filters) under annotator/
• Configuration‐driven orchestration (YAML + command‐line flags) decouple code from parameters.
d. Key technologies
• Python 3, PyTorch
• RAVE’s “annotator” subpackage for vision backbones
• Harmonizer’s model/filter modules for post‐processing
• EasyInv code invoked via pipelines
• SAMURAI (external mask generator)