How mlsakiit/forgetube works
Project Type and Purpose
• Project Type: This is a multi-modal automated video generation pipeline that integrates AI-based text generation, image synthesis, and text-to-speech (TTS) to automatically generate narrated videos.
• Purpose: The system takes a text prompt, uses a large language model (Gemini API) for script generation, then leverages diffusion models (Stable Diffusion XL) to create images, and converts text to audio (using Kokoro) before assembling the final video using video processing libraries (MoviePy/FFmpeg).
File Structure
• Key Directories and Files:
• assembly/: Contains scripts and templates (video_template.json) for video assembly, likely using MoviePy and FFmpeg.
• diffusion/: Contains scripts (generate_image.py, generate_image_local.py, Modelfile, etc.) for image generation via diffusion models.
• tts/: Contains scripts (generate_audio.py) for the text-to-speech functionality.
• main.py and main_local.py: The central entry points that orchestrate the entire pipeline.
• resources/: Holds assets like intro videos, fonts, and potentially placeholders for images and audio.
• Additional files such as pyproject.toml, .python-version, and uv.lock help manage project dependencies and Python versioning.
Components, Interactions, and Architectural Patterns
• Component Breakdown:
• Script Generation Module: Uses Gemini API (and Serp API) to generate a JSON-based script with text, audio, and visual prompts. (Interaction point: main.py/main_local.py where API keys are provided.)
• Image Generation Module (Diffusion): Processes the script’s prompts to generate images. It supports running the model either on Modal (cloud GPU) or locally. The diffusion folder contains different scripts that handle various image generation scenarios.
• Audio Generation Module (TTS): Converts text into speech using Kokoro. This is implemented in tts/scripts/generate_audio.py.
• Video Assembly Module: Collects the generated images, audio, and additional resources (intro, fonts, subtitles) to create the final video. This is handled by the assembly/scripts/assembly_video.py.
• Resource and Asset Management: The resources/ directory manages static files (e.g., intros, fonts) and the pipeline uses these assets during video finalization.
• Architectural Principles:
• Modular Architecture: Each functionality (script generation, image creation, audio synthesis, video assembly) is well encapsulated in its own folder.
• Sequential Pipeline: The output of one stage feeds into the next (script → images/audio → video assembly).
• Integration with External Services: Gemini and Serp APIs, FFmpeg for video processing, and Modal for cloud-based GPU acceleration.
• Dependencies and Technologies:
• Python 3.11, PyTorch, Diffusers (Hugging Face), MoviePy/FFmpeg, and Kokoro for TTS along with UV and Modal for dependency and resource management.
Final Summary
• The top layer represents external interactions and inputs (user prompts, API calls).
• The middle layer is divided into three main processing components (Script Generation, Image Generation, TTS).
• The bottom layer is the Video Assembly which integrates outputs from the processing layer along with resource assets.
• Modularity and sequential processing, which is central to this automated video generation pipeline.