# huanshere/videolingo architecture

> Architecture diagram of the GitHub repository huanshere/videolingo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huanshere/videolingo
- Repository: https://github.com/huanshere/videolingo

## Overview

##### Main Components
- Frontend/UI:
– A Streamlit-based user interface (st.py and st_components) that allows users to start a one-click process, select languages, and manage settings.
- Core Processing Pipeline (located in the “core” folder):
– A sequential series of steps (e.g., step1_ytdlp.py, step2_whisperX.py, step3 (NLP segmentation with spaCy utilities), step4 (summarization/translation), step5-day tasks, step6_gen_audio, etc.) that process the video from download to final audio merging.
- Transcription and Audio Processing:
– Integration with WhisperX (with additional modules under core/all_whisper_methods), handling audio pre-processing and background noise handling.
- Text-to-Speech (TTS) Engines:
– Multiple TTS modules (found in core/all_tts_functions) supporting providers such as Azure, OpenAI, GPT-SoVITS, FishTTS, and others. Include external API integrations like 302.ai, and configuration points for custom TTS adjustments.
- Batch Processing:
– The “batch” directory shows scripts and utilities (such as OneKeyBatch.bat and utils for video and settings processing) for running multiple videos or tasks automatically.
- Configuration & Support:
– Configuration files (config.yaml, custom_terms.xlsx) specifying settings for API keys, TTS choices, and video processing parameters.
– Utility scripts like onekeycleanup.py for maintenance.
- External Dependencies & Deployment:
– Docker (via Dockerfile) for containerized deployment.
– FFmpeg and CUDA/CUDNN support as external dependencies.
– Additional documentation and translations, supporting multi-language capabilities.

##### Component Relationships and Data Flow
– The UI (st.py) passes video URLs and settings to the processing pipeline.
- In the Core Pipeline:
– Step 1: Download the video (yt-dlp, as seen in step1_ytdlp.py).
– Step 2: Perform transcription with WhisperX (step2_whisperX.py, along with additional audio pre-processing modules).
– Steps 3 & 4: Segment the subtitles using NLP (with a series of spacy_utils and split methods) and then translate/summarize as needed.
– Steps 5-12: Sequential audio generation for dubbing: generating TTS audio (via modules in core/all_tts_functions), merging subtitle audio, aligning subtitles, and finally merging dubbed audio back into the video.
- TTS Engine Integration:
– The choice of TTS engines (azure_tts, openai_tts, GPT-SoVITS, etc.) is abstracted as interchangeable modules, allowing users to switch providers.
- Batch Mode:
– Scripts in the “batch” folder offer an alternative one-click processing experience, connecting to the same core engine.
- External API Connections:
– Incorporate external LLM and WhisperX API calls as needed (e.g., connecting to 302.ai for unified key management).
- Deployment and Containerization:
– Dockerfile defines the containerized environment, encapsulating all dependencies including FFmpeg and CUDA support for Windows/Linux.

##### Architectural Patterns
- Pipeline Architecture:
– Steps in the “core” directory reflect a sequential pipeline, where the output of one stage is the input for the next.
- Modular Design:
– Separation of concerns is maintained with dedicated folders for TTS functions, Whisper methods, NLP utilities, and batch processing.
- Extensibility Points:
– The design supports plugging in new TTS providers or custom NLP segmentation, as evidenced by the “custom_tts.py” and modular spacy utilities.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Frontend/UI Subgraph
    subgraph "Frontend/UI"
        UI["Streamlit UI (st.py)"]:::frontend
        stComp1["Download Video Section"]:::frontend
        stComp2["Imports and Utils"]:::frontend
        stComp3["Sidebar Settings"]:::frontend
    end

    %% Batch Processing Subgraph
    subgraph "Batch Processing"
        B1["OneKeyBatch"]:::batch
        B2["Batch Utils"]:::batch
    end

    %% Core Processing Pipeline Subgraph
    subgraph "Core Processing Pipeline"
        P1["Step1: Video Download (yt-dlp)"]:::pipeline
        P2["Step2: Transcription (WhisperX)"]:::pipeline
        P3A["Step3A: NLP spacy split"]:::pipeline
        P3B["Step3B: NLP splitbymeaning"]:::pipeline
        P4A["Step4A: Summarize"]:::pipeline
        P4B["Step4B: Translate"]:::pipeline
        P5["Step5: Subtitle Split"]:::pipeline
        P6["Step6: Generate Timeline"]:::pipeline
        P7["Step7: Merge Subtitles to Video"]:::pipeline
        P8A["Step8A: Gen Audio Task"]:::pipeline
        P8B["Step8B: Gen Dub Chunks"]:::pipeline
        P9["Step9: Extract Reference Audio"]:::pipeline
        P10["Step10: Gen Final Audio"]:::pipeline
        P11["Step11: Merge Full Audio"]:::pipeline
        P12["Step12: Merge Dub Audio to Video"]:::pipeline
    end

    %% Transcription & Audio Processing Subgraph
    subgraph "Transcription & Audio Processing"
        T1["WhisperX 302"]:::transcription
        T2["WhisperX Local"]:::transcription
        T3["Audio Preprocess"]:::transcription
        T4["Demucs VL"]:::transcription
    end

    %% Text-to-Speech (TTS) Engines Subgraph
    subgraph "TTS Engines"
        TT1["Azure TTS"]:::tts
        TT2["OpenAI TTS"]:::tts
        TT3["Custom TTS"]:::tts
        TT4["Fish TTS"]:::tts
        TT5["GPT-SoVITS TTS"]:::tts
        TT6["Edge TTS"]:::tts
        TT7["SF CosyVoice2"]:::tts
        TT8["SF FishTTS"]:::tts
        TT9["302_f5TTS"]:::tts
        TT10["Estimate Duration"]:::tts
        TT11["TTS Main"]:::tts
    end

    %% Configuration & Support Subgraph
    subgraph "Configuration & Support"
        C1["config.yaml"]:::config
        C2["custom_terms.xlsx"]:::config
        C3["OneKey Cleanup"]:::config
    end

    %% External Dependencies & Deployment Subgraph
    subgraph "External Dependencies & Deployment"
        E1["Dockerfile"]:::external
    end

    %% Connections: Frontend/UI to Core Pipeline
    UI -->|"triggers"| P1
    stComp1 --- UI
    stComp2 --- UI
    stComp3 --- UI

    %% Batch Processing triggers Core Pipeline
    B1 -->|"triggers"| P1
    B2 --- B1

    %% Core Pipeline sequential flow
    P1 -->|"next"| P2
    P2 -->|"next"| P3A
    P3A -->|"next"| P3B
    P3B -->|"next"| P4A
    P4A -->|"next"| P4B
    P4B -->|"next"| P5
    P5 -->|"next"| P6
    P6 -->|"next"| P7
    P7 -->|"next"| P8A
    P8A -->|"next"| P8B
    P8B -->|"next"| P9
    P9 -->|"next"| P10
    P10 -->|"next"| P11
    P11 -->|"next"| P12

    %% Integration: Transcription & Audio Processing with Step2
    P2 -->|"uses"| T1
    P2 -->|"uses"| T2
    P2 -->|"uses"| T3
    P2 -->|"uses"| T4

    %% Integration: TTS Engines with Audio Generation Step
    P8A -->|"calls"| TT1

    %% Configuration & Support influences Pipeline
    C1 -->|"configures"| P1
    C2 -->|"configures"| P1
    C3 -->|"supports"| P1

    %% External Dependencies support Pipeline
    E1 -->|"provides"| P1

    %% Click Events for Frontend/UI
    click UI "https://github.com/huanshere/videolingo/blob/main/st.py"
    click stComp1 "https://github.com/huanshere/videolingo/blob/main/st_components/download_video_section.py"
    click stComp2 "https://github.com/huanshere/videolingo/blob/main/st_components/imports_and_utils.py"
    click stComp3 "https://github.com/huanshere/videolingo/blob/main/st_components/sidebar_setting.py"

    %% Click Events for Core Processing Pipeline steps
    click P1 "https://github.com/huanshere/videolingo/blob/main/core/step1_ytdlp.py"
    click P2 "https://github.com/huanshere/videolingo/blob/main/core/step2_whisperX.py"
    click P3A "https://github.com/huanshere/videolingo/blob/main/core/step3_1_spacy_split.py"
    click P3B "https://github.com/huanshere/videolingo/blob/main/core/step3_2_splitbymeaning.py"
    click P4A "https://github.com/huanshere/videolingo/blob/main/core/step4_1_summarize.py"
    click P4B "https://github.com/huanshere/videolingo/blob/main/core/step4_2_translate_all.py"
    click P5 "https://github.com/huanshere/videolingo/blob/main/core/step5_splitforsub.py"
    click P6 "https://github.com/huanshere/videolingo/blob/main/core/step6_generate_final_timeline.py"
    click P7 "https://github.com/huanshere/videolingo/blob/main/core/step7_merge_sub_to_vid.py"
    click P8A "https://github.com/huanshere/videolingo/blob/main/core/step8_1_gen_audio_task.py"
    click P8B "https://github.com/huanshere/videolingo/blob/main/core/step8_2_gen_dub_chunks.py"
    click P9 "https://github.com/huanshere/videolingo/blob/main/core/step9_extract_refer_audio.py"
    click P10 "https://github.com/huanshere/videolingo/blob/main/core/step10_gen_audio.py"
    click P11 "https://github.com/huanshere/videolingo/blob/main/core/step11_merge_full_audio.py"
    click P12 "https://github.com/huanshere/videolingo/blob/main/core/step12_merge_dub_to_vid.py"

    %% Click Events for Transcription & Audio Processing
    click T1 "https://github.com/huanshere/videolingo/blob/main/core/all_whisper_methods/whisperX_302.py"
    click T2 "https://github.com/huanshere/videolingo/blob/main/core/all_whisper_methods/whisperX_local.py"
    click T3 "https://github.com/huanshere/videolingo/blob/main/core/all_whisper_methods/audio_preprocess.py"
    click T4 "https://github.com/huanshere/videolingo/blob/main/core/all_whisper_methods/demucs_vl.py"

    %% Click Events for TTS Engines
    click TT1 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/azure_tts.py"
    click TT2 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/openai_tts.py"
    click TT3 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/custom_tts.py"
    click TT4 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/fish_tts.py"
    click TT5 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/gpt_sovits_tts.py"
    click TT6 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/edge_tts.py"
    click TT7 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/sf_cosyvoice2.py"
    click TT8 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/sf_fishtts.py"
    click TT9 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/_302_f5tts.py"
    click TT10 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/estimate_duration.py"
    click TT11 "https://github.com/huanshere/videolingo/blob/main/core/all_tts_functions/tts_main.py"

    %% Click Events for Batch Processing
    click B1 "https://github.com/huanshere/videolingo/blob/main/batch/OneKeyBatch.bat"
    click B2 "https://github.com/huanshere/videolingo/tree/main/batch/utils/"

    %% Click Events for Configuration & Support
    click C1 "https://github.com/huanshere/videolingo/blob/main/config.yaml"
    click C2 "https://github.com/huanshere/videolingo/blob/main/custom_terms.xlsx"
    click C3 "https://github.com/huanshere/videolingo/blob/main/core/onekeycleanup.py"

    %% Click Event for External Dependencies & Deployment
    click E1 "https://github.com/huanshere/videolingo/tree/main/Dockerfile"

    %% Styles
    classDef frontend fill:#a0d8f1,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#c4f0c5,stroke:#333,stroke-width:2px;
    classDef batch fill:#fef7a1,stroke:#333,stroke-width:2px;
    classDef transcription fill:#e3d4f2,stroke:#333,stroke-width:2px;
    classDef tts fill:#f9d6e1,stroke:#333,stroke-width:2px;
    classDef config fill:#fce2b2,stroke:#333,stroke-width:2px;
    classDef external fill:#d0e6a5,stroke:#333,stroke-width:2px;
```
