How huanshere/videolingo works
Main Components
• Frontend/UI:
– A Streamlit-based user interface (st.py and st_components) that allows users to start a one-click process, select languages, and manage settings.
• Core Processing Pipeline (located in the “core” folder):
– A sequential series of steps (e.g., step1_ytdlp.py, step2_whisperX.py, step3 (NLP segmentation with spaCy utilities), step4 (summarization/translation), step5-day tasks, step6_gen_audio, etc.) that process the video from download to final audio merging.
• Transcription and Audio Processing:
– Integration with WhisperX (with additional modules under core/all_whisper_methods), handling audio pre-processing and background noise handling.
• Text-to-Speech (TTS) Engines:
– Multiple TTS modules (found in core/all_tts_functions) supporting providers such as Azure, OpenAI, GPT-SoVITS, FishTTS, and others. Include external API integrations like 302.ai, and configuration points for custom TTS adjustments.
• Batch Processing:
– The “batch” directory shows scripts and utilities (such as OneKeyBatch.bat and utils for video and settings processing) for running multiple videos or tasks automatically.
• Configuration & Support:
– Configuration files (config.yaml, custom_terms.xlsx) specifying settings for API keys, TTS choices, and video processing parameters.
– Utility scripts like onekeycleanup.py for maintenance.
• External Dependencies & Deployment:
– Docker (via Dockerfile) for containerized deployment.
– FFmpeg and CUDA/CUDNN support as external dependencies.
– Additional documentation and translations, supporting multi-language capabilities.
Component Relationships and Data Flow
– The UI (st.py) passes video URLs and settings to the processing pipeline.
• In the Core Pipeline:
– Step 1: Download the video (yt-dlp, as seen in step1_ytdlp.py).
– Step 2: Perform transcription with WhisperX (step2_whisperX.py, along with additional audio pre-processing modules).
– Steps 3 & 4: Segment the subtitles using NLP (with a series of spacy_utils and split methods) and then translate/summarize as needed.
– Steps 5-12: Sequential audio generation for dubbing: generating TTS audio (via modules in core/all_tts_functions), merging subtitle audio, aligning subtitles, and finally merging dubbed audio back into the video.
• TTS Engine Integration:
– The choice of TTS engines (azure_tts, openai_tts, GPT-SoVITS, etc.) is abstracted as interchangeable modules, allowing users to switch providers.
• Batch Mode:
– Scripts in the “batch” folder offer an alternative one-click processing experience, connecting to the same core engine.
• External API Connections:
– Incorporate external LLM and WhisperX API calls as needed (e.g., connecting to 302.ai for unified key management).
• Deployment and Containerization:
– Dockerfile defines the containerized environment, encapsulating all dependencies including FFmpeg and CUDA support for Windows/Linux.
Architectural Patterns
• Pipeline Architecture:
– Steps in the “core” directory reflect a sequential pipeline, where the output of one stage is the input for the next.
• Modular Design:
– Separation of concerns is maintained with dedicated folders for TTS functions, Whisper methods, NLP utilities, and batch processing.
• Extensibility Points:
– The design supports plugging in new TTS providers or custom NLP segmentation, as evidenced by the “custom_tts.py” and modular spacy utilities.