# fomn16/upwork-cb-translator architecture

> Architecture diagram of the GitHub repository fomn16/upwork-cb-translator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fomn16/upwork-cb-translator
- Repository: https://github.com/fomn16/upwork-cb-translator

## Overview

##### Project type and purpose
- This is a full-stack, real-time audio streaming and translation/processing application.
- Frontend (React + Vite) captures user media and connects via WebRTC.
- Backend (NestJS + mediasoup) handles signaling, media routing, and spawns FFmpeg processes.
- ML service (Python + GStreamer + custom translation utils) receives audio over UDP/WebRTC, processes it (e.g., transcription/translation), and returns text results.

##### File structure
Top-level directories:
- frontend/ – React app with socket.ts and mediasoupClient integration.
- backend/ – NestJS project with modules under src/calls (Mediasoup.service, signaling.gateway).
- ml/ – Python scripts (webrtc_python.py, gstreamer.py, translator utils) plus UDP senders.

Key files and patterns:
- backend/src/calls – implements an event-driven Signaling Gateway and a Mediasoup service.
- backend/src/lib – helpers for FFmpeg spawning and SDP/port management.
- ml/ – GStreamer pipelines, Python “webrtc_python.py” as a media endpoint, UDP-based raw audio ingestion, translator utilities.

##### Extract insights from the README
- Startup order: backend → python server → frontend
- A constant “announcedIp” must be set in backend/src/calls/constants.ts for mediasoup.
- Communication paths: WebRTC over the browser ↔ backend, UDP sockets or Python WebRTC ↔ ML translator.

##### Main components
A. Frontend Client
– React + Vite
– WebRTC Peer (mediasoupClient)
– WebSocket (socket.ts) for signaling
B. Backend Service
– NestJS Application
– Signaling Gateway (WebSocket)
– Mediasoup Worker & Router
– FFmpeg subprocesses (spawned via lib/ffmpeg.ts)
– UDP or RTP forwarders (lib/port.ts, custom code)
C. ML Translation Service
– Python WebRTC endpoint (webrtc_python.py) or UDP receiver (udp.js / udpSender.js)
– GStreamer pipelines (gstreamer.py)
– Translator utils (streaming_translator_utils.py, seamlessm4t_translator_utils.py)
– Output: text results (transcripts/translations)
D. (Optional) External dependencies
– System’s IPv4 network interfaces
– FFmpeg binary
– Python packages (in requirements.txt)

##### Relationships and data flows
1) User browser ⇄ Backend Signaling (WebSocket)
- Negotiates WebRTC transports and media parameters with NestJS Signaling Gateway.
2) User browser ⇄ Mediasoup Worker (RTP/DTLS/SRTP)
- Live audio stream from client’s microphone.
3) Backend ⇄ FFmpeg
- Receives RTP from Mediasoup, decodes/encodes, forwards raw audio frames via UDP/RTP to ML service.
4) Backend ⇄ ML Service
- UDP socket or Python WebRTC connection transports raw audio frames to Python.
5) ML Service → Translator pipeline → returns text output
- Process stream in real time, generate transcripts or translations.
6) ML Service → Backend (via UDP or WebSocket) → Frontend
- Textual results are pushed back through the NestJS signaling layer to the UI.

##### Architectural patterns and principles
- Modular separation: frontend / backend / ML microservice
- Event-driven signaling via WebSockets
- Media pipeline pattern: client → mediasoup → FFmpeg → ML → results
- Use of external processes (FFmpeg, GStreamer) for media transformations
- Lightweight, stateless transports (UDP, WebRTC)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Frontend Client"
        direction TB
        A1["main.tsx<br/>(React+Vite)"]:::frontend
        A2["App.tsx"]:::frontend
        A3["mediasoupClient.ts<br/>(WebRTC Peer)"]:::frontend
        A4["socket.ts<br/>(WebSocket Signaling)"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Service"
        direction TB
        B1["main.ts<br/>(NestJS App)"]:::backend
        B2["app.module.ts"]:::backend
        B3["signaling.gateway.ts<br/>(WebSocket Signaling)"]:::backend
        B4["Mediasoup.service.ts<br/>(Mediasoup Worker & Router)"]:::backend
        B5["constants.ts<br/>(announcedIp)"]:::backend
        subgraph "Lib Helpers"
            direction TB
            B6["ffmpeg.ts<br/>(FFmpeg Subprocess)"]:::backend
            B7["port.ts<br/>(RTP/UDP Port Mgmt)"]:::backend
            B8["sdp.ts<br/>(SDP Utilities)"]:::backend
            B9["lib.ts<br/>(Shared Helpers)"]:::backend
        end
    end

    %% ML Translation Layer
    subgraph "ML Translation Service"
        direction TB
        C1["main.py<br/>(Orchestrator)"]:::ml
        C2["webrtc_python.py<br/>(Python WebRTC)"]:::ml
        C3["udp.js<br/>(UDP Receiver)"]:::ml
        C4["udpSender.js<br/>(UDP Sender)"]:::ml
        C5["gstreamer.py<br/>(GStreamer Pipelines)"]:::ml
        C6["streaming_translator_utils.py"]:::ml
        C7["seamlessm4t_translator_utils.py"]:::ml
        C8["requirements.txt"]:::ml
    end

    %% Data Flows
    A4 <-->|"WebSocket Signaling"| B3
    B3 -->|"WebSocket Signaling"| A4
    A3 -->|"WebRTC/RTP"| B4
    B4 -->|"RTP Stream"| B6
    B6 -->|"UDP/RTP Raw Audio"| C3
    B6 -->|"UDP/RTP Raw Audio"| C2
    C1 -->|"invoke WebRTC"| C2
    C1 -->|"invoke UDP Sender"| C4
    C2 -->|"feed Pipeline"| C5
    C3 -->|"feed Pipeline"| C5
    C5 -->|"audio frames"| C6
    C5 -->|"audio frames"| C7
    C6 -->|"text results"| B3
    C7 -->|"text results"| B3
    B3 -->|"text results"| A2

    %% Click Events
    click A1 "https://github.com/fomn16/upwork-cb-translator/blob/master/frontend/src/main.tsx"
    click A2 "https://github.com/fomn16/upwork-cb-translator/blob/master/frontend/src/App.tsx"
    click A3 "https://github.com/fomn16/upwork-cb-translator/blob/master/frontend/src/mediasoupClient.ts"
    click A4 "https://github.com/fomn16/upwork-cb-translator/blob/master/frontend/src/socket.ts"
    click B1 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/main.ts"
    click B2 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/app.module.ts"
    click B3 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/calls/signaling.gateway.ts"
    click B4 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/calls/Mediasoup.service.ts"
    click B5 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/calls/constants.ts"
    click B6 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/lib/ffmpeg.ts"
    click B7 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/lib/port.ts"
    click B8 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/lib/sdp.ts"
    click B9 "https://github.com/fomn16/upwork-cb-translator/blob/master/backend/src/lib/lib.ts"
    click C1 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/main.py"
    click C2 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/webrtc_python.py"
    click C3 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/udp.js"
    click C4 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/udpSender.js"
    click C5 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/gstreamer.py"
    click C6 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/streaming_translator_utils.py"
    click C7 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/seamlessm4t_translator_utils.py"
    click C8 "https://github.com/fomn16/upwork-cb-translator/blob/master/ml/requirements.txt"

    %% Styles
    classDef frontend fill:#D6EAF8,stroke:#2980B9,color:#154360
    classDef backend fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef ml fill:#FCF3CF,stroke:#F4D03F,color:#7D6608
```
