# medahmedkrichen/vidubb architecture

> Architecture diagram of the GitHub repository medahmedkrichen/vidubb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/medahmedkrichen/vidubb
- Repository: https://github.com/medahmedkrichen/vidubb

## Overview

##### Project Type and Purpose
- ViDubb is a full-stack AI video dubbing application with a CLI and a Gradio web interface. It orchestrates a multi-stage media processing pipeline—downloading or ingesting video, performing audio analysis (diarization, transcription, translation, emotion analysis), generating new speech (voice cloning, TTS), optionally lip-syncing with Wav2Lip, and finally merging audio/video.

##### Main Components
a. User Interface
- CLI (inference.py)
- Gradio Web App (app.py)
b. Orchestration Layer
- Controller scripts (inference.py, app.py) that coordinate all sub-modules.
c. Video Input Module
- YouTube Downloader (via youtube-dl or pytube)
- Local File Loader
d. Audio Processing Pipeline
1. Speaker Diarization (pyannote.audio via Hugging Face API)
2. Speech-to-Text Transcription (Whisper)
3. Sentence Segmentation (NLTK)
4. Text Translation
- MarianMT (Hugging Face)
- Optional LLM (Groq “llama3-70b”)
5. Emotion Analysis (SpeechBrain or similar, optional)
e. Text-to-Speech (TTS)
- Voice cloning engine to synthesize translated sentences in cloned voices
f. Video/Lip-Sync Pipeline
1. Frame Extraction (OpenCV + FFmpeg)
2. Face Detection & Alignment (S3FD)
3. Lip-Sync Model (Wav2Lip)
g. Synchronization & Mixing
- FFmpeg and PyDub to align and mix generated audio with frames, preserve background sound
h. Storage & Assets
- Local directories: Wav2Lip/checkpoints (models), audio_chunks/, results/, temp/, videos/, speakers_audio/, speakers_image/
i. Configuration & Dependencies
- .env (API tokens), hparams.py, requirements.txt, requirementsCPU.txt, Conda environment, CUDA

##### Relationships & Data Flow
User → Controller → Video Input → Audio Processing Pipeline → TTS → Audio Output
Controller → Video/Lip-Sync Pipeline → Video Frames → Lip-Sync Model → Dubbed Video Frames
Audio Output + Video Frames → Synchronization & Mixing → Final Dubbed Video → results/
- External API calls: Speaker Diarization (Hugging Face), Translation (Hugging Face or Groq), optional Emotion API

##### Architectural Patterns & Principles
- Modular pipeline: each stage is an independent module with clear input/output
- Separation of concerns: UI layer, orchestration layer, compute‐intensive ML modules, I/O modules
- Extensibility: ability to swap transcription, translation, TTS, lip-sync engines by updating requirements and model paths
- Scalability: GPU acceleration for heavy ML tasks, CPU fallbacks available via requirementsCPU.txt

##### Technologies & Frameworks
- Python 3.10, Conda environment
- PyTorch (CUDA)
- FFmpeg, OpenCV, PyDub
- Wav2Lip (GAN-based lip sync)
- pyannote.audio (speaker diarization)
- Whisper (speech-to-text)
- MarianMT, llama3-70b (translation)
- SpeechBrain (emotion)
- TTS library for voice cloning
- Gradio for web interface

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    subgraph "UI Layer"
        CLI["CLI Interface"]:::io
        WebUI["Gradio Web App"]:::io
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        Controller["Controller (inference.py, app.py)"]:::compute
    end

    %% Audio Processing Pipeline
    subgraph "Audio Pipeline"
        Diarization["Speaker Diarization\n(pyannote.audio)"]:::compute
        Transcription["Transcription\n(Whisper)"]:::compute
        Segmentation["Sentence Segmentation\n(NLTK)"]:::compute
        Translation["Text Translation\n(MarianMT/LLM)"]:::compute
        Emotion{"Emotion Analysis\n(SpeechBrain)\nOptional?"}:::decision
        TTS["TTS (Voice Cloning)"]:::compute
    end

    %% Video Processing Pipeline
    subgraph "Video Pipeline"
        FrameExt["Frame Extraction\n(OpenCV + FFmpeg)"]:::compute
        FaceDetect["Face Detection & Alignment\n(S3FD)"]:::compute
        LipSyncGate{"LipSync Enabled?"}:::decision
        LipSync1["Wav2Lip Model\n(wav2lip.py)"]:::compute
        SyncNet["SyncNet\n(syncnet.py)"]:::compute
    end

    %% Synchronization & Mixing
    Sync["Synchronization & Mixing\n(FFmpeg + PyDub)"]:::compute

    %% Storage & Assets
    subgraph "Storage & Assets"
        Checkpoints["Model Checkpoints"]:::data
        Temp["Temp Files"]:::data
        Results1["results/"]:::data
        Results2["Wav2Lip/results/"]:::data
        AudioChunks["audio_chunks/"]:::data
        SpeakersAudio["speakers_audio/"]:::data
        SpeakersImage["speakers_image/"]:::data
        VideosStorage["videos/"]:::data
        Env["Config & Env\n(.env, requirements)"]:::data
        HP["Hyperparameters\n(hparams.py)"]:::data
    end

    %% Data Stores
    GeneratedAudio["Generated Audio"]:::data
    DubFrames["Dubbed Video Frames"]:::data
    FinalVideo["Final Dubbed Video"]:::data

    %% Connections
    CLI -->|uses| Controller
    WebUI -->|uses| Controller

    Controller -->|loads video| VideoLoader

    Controller -->|start| Diarization
    Diarization -->|segments| Transcription
    Transcription -->|sentences| Segmentation
    Segmentation -->|texts| Translation
    Translation -->|translated text| Emotion
    Emotion -->|text| TTS
    TTS --> GeneratedAudio

    Controller -->|start| FrameExt
    FrameExt --> FaceDetect
    FaceDetect --> LipSyncGate
    LipSyncGate -->|yes| LipSync1
    LipSyncGate -->|yes| SyncNet
    LipSyncGate -->|no| DubFrames
    LipSync1 --> DubFrames
    SyncNet --> DubFrames

    GeneratedAudio --> Sync
    DubFrames --> Sync
    Sync --> FinalVideo
    FinalVideo --> Results1
    FinalVideo --> Results2

    VideoLoader["Video Input\n(YouTube/Local)"]:::io

    %% External API Calls
    Diarization -.->|HuggingFace API| externalHF["HuggingFace"]:::external
    Translation -.->|HuggingFace API| externalHF
    Translation -.->|Groq API| externalGroq["Groq"]:::external
    Emotion -.->|optional API| externalEmotion["Emotion API"]:::external

    %% Click Events
    click CLI "https://github.com/medahmedkrichen/vidubb/blob/main/inference.py"
    click WebUI "https://github.com/medahmedkrichen/vidubb/blob/main/app.py"
    click VideoLoader "https://github.com/medahmedkrichen/vidubb/blob/main/tools/utils.py"
    click FrameExt "https://github.com/medahmedkrichen/vidubb/blob/main/Wav2Lip/preprocess.py"
    click FaceDetect "https://github.com/medahmedkrichen/vidubb/tree/main/Wav2Lip/face_detection/"
    click LipSync1 "https://github.com/medahmedkrichen/vidubb/blob/main/Wav2Lip/models/wav2lip.py"
    click SyncNet "https://github.com/medahmedkrichen/vidubb/blob/main/Wav2Lip/models/syncnet.py"
    click Wav2Lip "https://github.com/medahmedkrichen/vidubb/tree/main/Wav2Lip/"
    click Checkpoints "https://github.com/medahmedkrichen/vidubb/tree/main/Wav2Lip/checkpoints/"
    click Temp "https://github.com/medahmedkrichen/vidubb/tree/main/Wav2Lip/temp/"
    click Results1 "https://github.com/medahmedkrichen/vidubb/tree/main/results/"
    click Results2 "https://github.com/medahmedkrichen/vidubb/tree/main/Wav2Lip/results/"
    click AudioChunks "https://github.com/medahmedkrichen/vidubb/tree/main/audio_chunks/"
    click SpeakersAudio "https://github.com/medahmedkrichen/vidubb/tree/main/speakers_audio/"
    click SpeakersImage "https://github.com/medahmedkrichen/vidubb/tree/main/speakers_image/"
    click VideosStorage "https://github.com/medahmedkrichen/vidubb/tree/main/videos/"
    click HP "https://github.com/medahmedkrichen/vidubb/blob/main/Wav2Lip/hparams.py"
    click Env "https://github.com/medahmedkrichen/vidubb/blob/main/.env"

    %% Styles
    classDef compute fill:#cfe2f3,stroke:#6fa8dc,stroke-width:1px
    classDef data fill:#d9ead3,stroke:#6aa84f,stroke-width:1px
    classDef io fill:#fff2cc,stroke:#e69138,stroke-width:1px
    classDef decision fill:#f4cccc,stroke:#cc0000,stroke-width:1px
    classDef external fill:#f0f0f0,stroke:#888888,stroke-dasharray: 2 2,stroke-width:1px
```
