How medahmedkrichen/vidubb works
Project Type and Purpose
• ViDubb is a full-stack AI video dubbing application with a CLI and a Gradio web interface. It orchestrates a multi-stage media processing pipeline—downloading or ingesting video, performing audio analysis (diarization, transcription, translation, emotion analysis), generating new speech (voice cloning, TTS), optionally lip-syncing with Wav2Lip, and finally merging audio/video.
Main Components
a. User Interface
• CLI (inference.py)
• Gradio Web App (app.py)
b. Orchestration Layer
• Controller scripts (inference.py, app.py) that coordinate all sub-modules.
c. Video Input Module
• YouTube Downloader (via youtube-dl or pytube)
• Local File Loader
d. Audio Processing Pipeline
1. Speaker Diarization (pyannote.audio via Hugging Face API)
2. Speech-to-Text Transcription (Whisper)
3. Sentence Segmentation (NLTK)
4. Text Translation
• MarianMT (Hugging Face)
• Optional LLM (Groq “llama3-70b”)
5. Emotion Analysis (SpeechBrain or similar, optional)
e. Text-to-Speech (TTS)
• Voice cloning engine to synthesize translated sentences in cloned voices
f. Video/Lip-Sync Pipeline
1. Frame Extraction (OpenCV + FFmpeg)
2. Face Detection & Alignment (S3FD)
3. Lip-Sync Model (Wav2Lip)
g. Synchronization & Mixing
• FFmpeg and PyDub to align and mix generated audio with frames, preserve background sound
h. Storage & Assets
• Local directories: Wav2Lip/checkpoints (models), audio_chunks/, results/, temp/, videos/, speakers_audio/, speakers_image/
i. Configuration & Dependencies
• .env (API tokens), hparams.py, requirements.txt, requirementsCPU.txt, Conda environment, CUDA
Relationships & Data Flow
User → Controller → Video Input → Audio Processing Pipeline → TTS → Audio Output
Controller → Video/Lip-Sync Pipeline → Video Frames → Lip-Sync Model → Dubbed Video Frames
Audio Output + Video Frames → Synchronization & Mixing → Final Dubbed Video → results/
• External API calls: Speaker Diarization (Hugging Face), Translation (Hugging Face or Groq), optional Emotion API
Architectural Patterns & Principles
• Modular pipeline: each stage is an independent module with clear input/output
• Separation of concerns: UI layer, orchestration layer, compute‐intensive ML modules, I/O modules
• Extensibility: ability to swap transcription, translation, TTS, lip-sync engines by updating requirements and model paths
• Scalability: GPU acceleration for heavy ML tasks, CPU fallbacks available via requirementsCPU.txt
Technologies & Frameworks
• Python 3.10, Conda environment
• PyTorch (CUDA)
• FFmpeg, OpenCV, PyDub
• Wav2Lip (GAN-based lip sync)
• pyannote.audio (speaker diarization)
• Whisper (speech-to-text)
• MarianMT, llama3-70b (translation)
• SpeechBrain (emotion)
• TTS library for voice cloning
• Gradio for web interface