How nalin0503/fyp_imagemorpher works
Project Purpose and Type
• The project is an advanced image morphing pipeline designed for creating smooth transitions between images. It combines keyframe generation using diffusion-based models (handled by the DiffMorpher submodule with LCM-LoRA acceleration) and recursive FILM-based frame interpolation.
• It supports both a command-line interface (via run_morphing.py) and a web-based UI (via app.py using Streamlit) for interactive control.
• The system is intended for research and production use (NTU Final Year Project) and leverages state-of-the-art techniques in image processing.
Key Components and Their Roles
a. Frontend/UI Layer:
• app.py: Provides a web interface built with Streamlit. It allows users to input images, text prompts, and morphing parameters.
b. Orchestration/Controller:
• run_morphing.py: Acts as the main controller that coordinates the overall morphing process. It triggers keyframe generation and interpolation according to user-specified parameters.
• Makefile: Automates initialization, dependency setup, and triggering of morphing tasks.
c. Core Processing Modules:
• DiffMorpher Submodule: Responsible for generating keyframes using diffusion models and integrates LCM-LoRA for accelerating diffusion. Within this submodule, main.py serves as the entry point.
• FILM.py: Implements the recursive frame interpolation algorithm that smooths transitions between keyframes.
d. Supporting Components:
• requirements.txt: Manages the Python package dependencies.
• Sample_runs folder: Contains example outputs and sample morphing results, useful for testing and verification.
• Configuration Files/Assets:
• .gitmodules file: Indicates the use of submodules (like DiffMorpher).
Relationships and Data Flow
• The user (through either CLI or web UI) provides the initial images and associated text prompts.
• The input is first passed to the DiffMorpher component that performs diffusion-based keyframe generation (possibly enhanced by LCM-LoRA).
• Once the keyframes are generated, they are processed by the FILM.py component to create smoothly interpolated frames.
• Final outputs (videos and intermediate frames) are saved under designated directories (sample_runs) and optionally can be previewed through the web interface.
• From “User Input” to “UI Layer (app.py)” or “Direct CLI (run_morphing.py)”
• From “Controller” to “DiffMorpher Submodule” (keyframe generation)
• From “Controller” to “FILM Module” (frame interpolation)
• From both processing modules to “Output Storage” (sample_runs)
Architectural Patterns and Technologies
• Modular Architecture: Separation between the frontend (Streamlit UI), controller/orchestration layer (run_morphing.py), and two core processing modules (DiffMorpher and FILM).
• Submodule Integration: The DiffMorpher is integrated as a submodule, reflecting the dependency on an external keyframe generation framework.
• Acceleration Mechanism: LCM-LoRA integration for accelerating the reverse diffusion process.
• Technologies:
• Python 3.12 as the primary language.
• CUDA and GPU acceleration for diffusion and interpolation tasks.
• Streamlit for the web-based user interface.
• External libraries from Stable Diffusion, Latent Consistency Models, and FILM for frame interpolation.