How szymon-25/dubbing-engine works
Project type and purpose
• This is an open-source TypeScript project (running on Bun) that implements an AI-powered video dubbing engine.
• Purpose: take an input video, transcribe its audio, translate text, clone voices, perform audio/video processing (lip-sync, subtitles), then produce a dubbed output video.
File structure
• Top-level scripts and configs:
• start.sh – orchestrates dependency checks and launch
• package.json, tsconfig.json, bun.lockb – build and runtime config
• .env.example – environment variables for API keys
• src/core/index.ts – central orchestrator that calls all submodules
• src/llm, src/transcription, src/speech, src/separator, src/lipsync, src/ffmpeg, src/subtitles, src/smart-sync – functional modules
• src/utils and src/types – shared utilities and type declarations
• input/ and output/ directories – I/O targets for video files
README for additional insights
• The readme walks through a six-step pipeline: configuration, transcription & analysis, translation, audio processing, voice generation, final assembly.
• It lists external services/APIs: OpenAI, Gladia, ElevenLabs, Lalal.ai, SyncLab (AWS for lipsync), FFmpeg.
a. Main components:
1. Client/CLI Interface (start.sh + config)
2. Core Orchestrator (src/core/index.ts)
3. Transcription Service (src/transcription/* + Gladia API)
4. Translation & Prompt Builder (src/llm/* + OpenAI API)
5. Audio Separation (src/separator/spleeter.ts + Lalal.ai)
6. SmartSync Adaptation (src/smart-sync/adaptation.ts)
7. Voice Generation (src/speech/speechGenerator.ts + ElevenLabs API)
8. Video & Audio Utilities (src/ffmpeg/*)
9. Subtitles Generator (src/subtitles/subtitles-generator.ts)
10. Lip Sync Module (src/lipsync/lipsync.ts + SyncLab/AWS)
11. I/O Storage (input/, output/, AWS S3 for lipsync)
12. Environment & Config (src/utils/config.ts, .env)
b. Relationships and interactions:
• start.sh → reads .env → invokes core/index
• core/index sequences modules in pipeline order
• transcription calls Gladia, returns text segments
• translation module sends segments to OpenAI
• audio separation calls Lalal.ai to split voices/background
• smart-sync adapts timing metadata
• speechGenerator invokes ElevenLabs to produce dubbed audio segments
• ffmpeg utilities stitch audio/video segments, adjust levels
• subtitles-generator creates subtitle files from translated text
• lipsync module sends original video + new audio to SyncLab (via AWS S3) to adjust mouth movements
• final ffmpeg merge outputs the dubbed video to output/
c. Architectural patterns and design principles:
• Pipeline / orchestration pattern: discrete stages, each with a well-defined input/output
• Modular design: each feature in its own folder under src/
• Dependency injection via configuration in src/utils/config.ts
• External API adaptors: each service is wrapped in its own module (single responsibility)
• Use of asynchronous operations for API calls and ffmpeg processing
d. Relevant technologies, frameworks, and libraries:
• Bun runtime + TypeScript
• FFmpeg CLI or WebAssembly wrapper
• Gladia transcription API
• OpenAI for text translation and prompt building
• ElevenLabs for voice cloning and TTS
• Lalal.ai for audio separation
• SyncLab + AWS S3 for lip-sync
• Spleeter library (via Lalal.ai or local wrapper)
• Environment config management via .env
Detail without over-splitting
• Of the six pipeline steps as components
• Entry and exit points (input directory, output directory or S3 bucket)
• Where ffmpeg transforms data vs. where AI services are invoked