How aravindan20/voice_clone_usingindic works
Project type & purpose (from README + tree)
Project type: Single-node, local-first AI audio dubbing web app (not a distributed backend/frontend deployment).
Purpose: Take an input audio clip → transcribe (Gemini 2.0 Flash) → translate to an Indian language (Gemini) → allow manual edit → voice-clone TTS (IndicF5 via PyTorch/Hugging Face) → output dubbed audio.
UI: Gradio served on http://localhost:12000.
What the file tree implies architecturally
Top-level files map cleanly to “one service” with external dependencies:
• dubsync_complete.py — main application/orchestrator (Gradio UI + pipeline)
• app.py — likely an alternate entrypoint/prototype (include as “optional/legacy runner” unless you confirm usage)
• requirements.txt — Python deps (Gradio, librosa, soundfile, google genai libs, etc.)
• setup_environment.sh — environment bootstrap (conda + torch cuda + IndicF5 install)
• .env.example — shows required secrets: GEMINI_API_KEY, HUGGINGFACE_TOKEN
• .browser_screenshots/ — non-runtime artifacts (documentation/QA evidence)
User Browser
• uploads/records audio
• edits translation text
• downloads dubbed audio
DubSync Application (Python process)
• Gradio Web Server/UI
• Pipeline Orchestrator (the glue logic in dubsync_complete.py)
• Audio Pre/Post-processing (librosa/soundfile; resample to 24kHz mono)
• ASR + Translation Client (Gemini API calls)
• Voice Cloning TTS Engine (IndicF5 running locally via PyTorch)
• Temp File Storage / Working Directory (where uploaded audio, intermediate wav, final wav live)
External Services
• Google Gemini API (transcription + translation)
• Hugging Face Hub (model weights/token-auth download) *(even if the model is installed, weights are typically fetched/validated here)*
User → Gradio: Edit translated text (manual review)
8. Gradio → IndicF5 Adapter: Provide (a) normalized source audio (voice reference) + (b) edited translation text + (c) target language
Gradio → Browser: Return downloadable dubbed audio
• App startup → Hugging Face Hub: authenticate + download/load model artifacts (token)
• App startup → Env Vars: GEMINI_API_KEY, HUGGINGFACE_TOKEN
• Error path: IndicF5 load failure → “placeholder audio” → Browser
Important architectural patterns/design principles present
• Orchestrator pipeline pattern: one coordinator (in dubsync_complete.py) sequencing distinct AI steps.
• Human-in-the-loop: translation is not final until user edits; model inference depends on user-approved text.
• Adapter pattern for external AI: Gemini and IndicF5 are best drawn as adapters to keep the core pipeline clean.
• Local inference + remote inference hybrid: Gemini is remote API; IndicF5 is local GPU-heavy inference.
What to label explicitly (to make it “detailed” and accurate)
• Technology: “Gradio”, “PyTorch + IndicF5”, “Gemini 2.0 Flash API”, “librosa/soundfile”
• Data contracts:
• Input formats supported: WAV/MP3/M4A/FLAC
• Internal canonical format: 24kHz mono
• Text: transcript + translated text + edited text
• Constraints:
• “<= 30s recommended”
• “GPU 4GB+ recommended”
• Secrets:
• .env provides GEMINI_API_KEY, HUGGINGFACE_TOKEN
Don’t overthink it: split into *more* components than fewer
Even if everything is implemented in one Python file, represent it as:
• UI layer
• preprocessing layer
• external API adapter
• human review step
• inference engine
• storage
• configuration
• setup-time provisioning