How amitvermanit/audio_call works
Project Type and Purpose
• This is a full-stack WebRTC application with integrated Python ML pipelines for text-to-speech (TTS) and voice-cloning.
• The backend (NestJS + Mediasoup) handles WebRTC signaling and media routing.
• The frontend (React + Vite) is a browser client that captures/transmits audio/video.
• The ml2 folder contains standalone Python services that run ML workers for seamless TTS and voice clone.
File Structure Analysis
• Top-level:
• backend/ (Node/NestJS + Mediasoup)
• frontend/ (React + Vite)
• ml2/ (Python environment + pipelines)
• backend/src/calls: Mediasoup.service, signaling.gateway – handles SFU, signaling via WebSockets.
• backend/src/lib: ffmpeg and SDP utilities for media transformation.
• frontend/src: mediasoupClient.ts, socket.ts – WebRTC client logic and socket.io signaling.
• ml2/pipelines: two sub-folders (seamless_t2s and voice_clone) each with a Python worker.
Key README Insights
• Backend & frontend must run first; then Python server and workers.
• Two Chrome instances join the call via WebRTC for peer communication.
• ML pipelines run as independent Python processes, consuming media and producing transformed audio streams.