How fomn16/upwork-cb-translator works
Project type and purpose
• This is a full-stack, real-time audio streaming and translation/processing application.
• Frontend (React + Vite) captures user media and connects via WebRTC.
• Backend (NestJS + mediasoup) handles signaling, media routing, and spawns FFmpeg processes.
• ML service (Python + GStreamer + custom translation utils) receives audio over UDP/WebRTC, processes it (e.g., transcription/translation), and returns text results.
File structure
Top-level directories:
• frontend/ – React app with socket.ts and mediasoupClient integration.
• backend/ – NestJS project with modules under src/calls (Mediasoup.service, signaling.gateway).
• ml/ – Python scripts (webrtc_python.py, gstreamer.py, translator utils) plus UDP senders.
Key files and patterns:
• backend/src/calls – implements an event-driven Signaling Gateway and a Mediasoup service.
• backend/src/lib – helpers for FFmpeg spawning and SDP/port management.
• ml/ – GStreamer pipelines, Python “webrtc_python.py” as a media endpoint, UDP-based raw audio ingestion, translator utilities.
Extract insights from the README
• Startup order: backend → python server → frontend
• A constant “announcedIp” must be set in backend/src/calls/constants.ts for mediasoup.
• Communication paths: WebRTC over the browser ↔ backend, UDP sockets or Python WebRTC ↔ ML translator.
Main components
A. Frontend Client
– React + Vite
– WebRTC Peer (mediasoupClient)
– WebSocket (socket.ts) for signaling
B. Backend Service
– NestJS Application
– Signaling Gateway (WebSocket)
– Mediasoup Worker & Router
– FFmpeg subprocesses (spawned via lib/ffmpeg.ts)
– UDP or RTP forwarders (lib/port.ts, custom code)
C. ML Translation Service
– Python WebRTC endpoint (webrtc_python.py) or UDP receiver (udp.js / udpSender.js)
– GStreamer pipelines (gstreamer.py)
– Translator utils (streaming_translator_utils.py, seamlessm4t_translator_utils.py)
– Output: text results (transcripts/translations)
D. (Optional) External dependencies
– System’s IPv4 network interfaces
– FFmpeg binary
– Python packages (in requirements.txt)
Relationships and data flows
1) User browser ⇄ Backend Signaling (WebSocket)
• Negotiates WebRTC transports and media parameters with NestJS Signaling Gateway.
2) User browser ⇄ Mediasoup Worker (RTP/DTLS/SRTP)
• Live audio stream from client’s microphone.
3) Backend ⇄ FFmpeg
• Receives RTP from Mediasoup, decodes/encodes, forwards raw audio frames via UDP/RTP to ML service.
4) Backend ⇄ ML Service
• UDP socket or Python WebRTC connection transports raw audio frames to Python.
5) ML Service → Translator pipeline → returns text output
• Process stream in real time, generate transcripts or translations.
6) ML Service → Backend (via UDP or WebSocket) → Frontend
• Textual results are pushed back through the NestJS signaling layer to the UI.
Architectural patterns and principles
• Modular separation: frontend / backend / ML microservice
• Event-driven signaling via WebSockets
• Media pipeline pattern: client → mediasoup → FFmpeg → ML → results
• Use of external processes (FFmpeg, GStreamer) for media transformations
• Lightweight, stateless transports (UDP, WebRTC)