# szymon-25/dubbing-engine architecture

> Architecture diagram of the GitHub repository szymon-25/dubbing-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/szymon-25/dubbing-engine
- Repository: https://github.com/szymon-25/dubbing-engine

## Overview

##### Project type and purpose
- This is an open-source TypeScript project (running on Bun) that implements an AI-powered video dubbing engine.
- Purpose: take an input video, transcribe its audio, translate text, clone voices, perform audio/video processing (lip-sync, subtitles), then produce a dubbed output video.

##### File structure
- Top-level scripts and configs:
- start.sh – orchestrates dependency checks and launch
- package.json, tsconfig.json, bun.lockb – build and runtime config
- .env.example – environment variables for API keys
- src/core/index.ts – central orchestrator that calls all submodules
- src/llm, src/transcription, src/speech, src/separator, src/lipsync, src/ffmpeg, src/subtitles, src/smart-sync – functional modules
- src/utils and src/types – shared utilities and type declarations
- input/ and output/ directories – I/O targets for video files

##### README for additional insights
- The readme walks through a six-step pipeline: configuration, transcription & analysis, translation, audio processing, voice generation, final assembly.
- It lists external services/APIs: OpenAI, Gladia, ElevenLabs, Lalal.ai, SyncLab (AWS for lipsync), FFmpeg.

a. Main components:
1. Client/CLI Interface (start.sh + config)
2. Core Orchestrator (src/core/index.ts)
3. Transcription Service (src/transcription/* + Gladia API)
4. Translation & Prompt Builder (src/llm/* + OpenAI API)
5. Audio Separation (src/separator/spleeter.ts + Lalal.ai)
6. SmartSync Adaptation (src/smart-sync/adaptation.ts)
7. Voice Generation (src/speech/speechGenerator.ts + ElevenLabs API)
8. Video & Audio Utilities (src/ffmpeg/*)
9. Subtitles Generator (src/subtitles/subtitles-generator.ts)
10. Lip Sync Module (src/lipsync/lipsync.ts + SyncLab/AWS)
11. I/O Storage (input/, output/, AWS S3 for lipsync)
12. Environment & Config (src/utils/config.ts, .env)

b. Relationships and interactions:
- start.sh → reads .env → invokes core/index
- core/index sequences modules in pipeline order
- transcription calls Gladia, returns text segments
- translation module sends segments to OpenAI
- audio separation calls Lalal.ai to split voices/background
- smart-sync adapts timing metadata
- speechGenerator invokes ElevenLabs to produce dubbed audio segments
- ffmpeg utilities stitch audio/video segments, adjust levels
- subtitles-generator creates subtitle files from translated text
- lipsync module sends original video + new audio to SyncLab (via AWS S3) to adjust mouth movements
- final ffmpeg merge outputs the dubbed video to output/

c. Architectural patterns and design principles:
- Pipeline / orchestration pattern: discrete stages, each with a well-defined input/output
- Modular design: each feature in its own folder under src/
- Dependency injection via configuration in src/utils/config.ts
- External API adaptors: each service is wrapped in its own module (single responsibility)
- Use of asynchronous operations for API calls and ffmpeg processing

d. Relevant technologies, frameworks, and libraries:
- Bun runtime + TypeScript
- FFmpeg CLI or WebAssembly wrapper
- Gladia transcription API
- OpenAI for text translation and prompt building
- ElevenLabs for voice cloning and TTS
- Lalal.ai for audio separation
- SyncLab + AWS S3 for lip-sync
- Spleeter library (via Lalal.ai or local wrapper)
- Environment config management via .env

##### Detail without over-splitting
- Of the six pipeline steps as components
- Entry and exit points (input directory, output directory or S3 bucket)
- Where ffmpeg transforms data vs. where AI services are invoked

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client and Configuration
    subgraph "Client & Config"
        SH["start.sh"]:::internal
        ENV[".env.example"]:::config
    end

    %% Storage
    subgraph "Storage"
        InputDir["input/"]:::storage
        OutputDir["output/"]:::storage
        S3Bucket["AWS S3"]:::storage
    end

    %% Core Orchestrator
    SH --> Core["Core Orchestrator"]:::internal
    ENV --> Core
    InputDir --> SH

    %% Processing Pipeline
    Core --> Transcription["Transcription Service"]:::module
    Core --> Translation["Translation & Prompt Builder"]:::module
    Core --> Separation["Audio Separation Service"]:::module
    Core --> SmartSync["SmartSync Adaptation"]:::module
    Core --> SpeechGen["Voice Generation Service"]:::module
    Core --> SubGen["Subtitles Generator"]:::module
    Core --> LipSyncMod["Lip Sync Module"]:::module
    Core --> FFMpegUtils["FFmpeg Utilities"]:::module

    %% External Services
    Transcription -->|"HTTP/REST"| GladiaAPI["Gladia API"]:::external
    Translation -->|"HTTP/REST"| OpenAIAPI["OpenAI API"]:::external
    Separation -->|"HTTP/REST"| Lalalai["Lalal.ai API"]:::external
    SpeechGen -->|"HTTP/REST"| ElevenAPI["ElevenLabs API"]:::external
    LipSyncMod -->|"HTTP/REST"| SyncLab["SyncLab"]:::external
    LipSyncMod -->|"uses S3"| S3Bucket

    %% Inter-module Data Flows
    SpeechGen -->|"audio segments"| FFMpegUtils
    LipSyncMod -->|"video & audio"| FFMpegUtils
    SubGen -->|"subtitle files"| OutputDir
    FFMpegUtils -->|"final merge"| OutputDir

    %% Click Events
    click SH "https://github.com/szymon-25/dubbing-engine/blob/main/start.sh"
    click ENV "https://github.com/szymon-25/dubbing-engine/blob/main/.env.example"
    click Core "https://github.com/szymon-25/dubbing-engine/blob/main/src/core/index.ts"
    click Transcription "https://github.com/szymon-25/dubbing-engine/blob/main/src/transcription/transcriber.ts"
    click Translation "https://github.com/szymon-25/dubbing-engine/blob/main/src/llm/openai.ts"
    click Translation "https://github.com/szymon-25/dubbing-engine/blob/main/src/llm/prompt-builder.ts"
    click Separation "https://github.com/szymon-25/dubbing-engine/blob/main/src/separator/spleeter.ts"
    click SmartSync "https://github.com/szymon-25/dubbing-engine/blob/main/src/smart-sync/adaptation.ts"
    click SpeechGen "https://github.com/szymon-25/dubbing-engine/blob/main/src/speech/speechGenerator.ts"
    click SpeechGen "https://github.com/szymon-25/dubbing-engine/blob/main/src/elevenlabs/elevenlabs.ts"
    click FFMpegUtils "https://github.com/szymon-25/dubbing-engine/blob/main/src/ffmpeg/audio-utils.ts"
    click FFMpegUtils "https://github.com/szymon-25/dubbing-engine/blob/main/src/ffmpeg/video-utils.ts"
    click FFMpegUtils "https://github.com/szymon-25/dubbing-engine/blob/main/src/ffmpeg/ffmpegPatch.ts"
    click SubGen "https://github.com/szymon-25/dubbing-engine/blob/main/src/subtitles/subtitles-generator.ts"
    click LipSyncMod "https://github.com/szymon-25/dubbing-engine/blob/main/src/lipsync/lipsync.ts"
    click InputDir "https://github.com/szymon-25/dubbing-engine/tree/main/input/"
    click OutputDir "https://github.com/szymon-25/dubbing-engine/tree/main/output/"
    click Config "https://github.com/szymon-25/dubbing-engine/blob/main/src/utils/config.ts"

    %% Styles
    classDef internal fill:#c5f4e7,stroke:#333,stroke-width:1px
    classDef external fill:#a0c4ff,stroke:#333,stroke-width:1px
    classDef module fill:#d0f4de,stroke:#333,stroke-width:1px
    classDef storage fill:#fff3b0,stroke:#333,stroke-width:1px
    classDef config fill:#ffe5d9,stroke:#333,stroke-width:1px
```
