# amitvermanit/audio_call architecture

> Architecture diagram of the GitHub repository amitvermanit/audio_call, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amitvermanit/audio_call
- Repository: https://github.com/amitvermanit/audio_call

## Overview

##### Project Type and Purpose
- This is a full-stack WebRTC application with integrated Python ML pipelines for text-to-speech (TTS) and voice-cloning.
- The backend (NestJS + Mediasoup) handles WebRTC signaling and media routing.
- The frontend (React + Vite) is a browser client that captures/transmits audio/video.
- The ml2 folder contains standalone Python services that run ML workers for seamless TTS and voice clone.

##### File Structure Analysis
- Top-level:
- backend/ (Node/NestJS + Mediasoup)
- frontend/ (React + Vite)
- ml2/ (Python environment + pipelines)
- backend/src/calls: Mediasoup.service, signaling.gateway – handles SFU, signaling via WebSockets.
- backend/src/lib: ffmpeg and SDP utilities for media transformation.
- frontend/src: mediasoupClient.ts, socket.ts – WebRTC client logic and socket.io signaling.
- ml2/pipelines: two sub-folders (seamless_t2s and voice_clone) each with a Python worker.

##### Key README Insights
- Backend & frontend must run first; then Python server and workers.
- Two Chrome instances join the call via WebRTC for peer communication.
- ML pipelines run as independent Python processes, consuming media and producing transformed audio streams.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Browser Clients
    subgraph "Browser Clients" 
        direction TB
        Client1["Chrome Instance"]:::frontend
        Client2["Chrome Instance"]:::frontend
    end

    %% Frontend Application
    subgraph "Frontend App" 
        direction TB
        MainFE["main.tsx"]:::frontend
        MediaClient["mediasoupClient.ts"]:::frontend
        SocketClient["socket.ts"]:::frontend
    end

    %% Backend Server
    subgraph "Backend Server"
        direction TB
        MainBE["main.ts"]:::backend
        AppModule["app.module.ts"]:::backend
        AppController["app.controller.ts"]:::backend
        AppService["app.service.ts"]:::backend

        subgraph "Calls Module"
            direction TB
            SignalingModule["signaling.module.ts"]:::backend
            SignalingGateway["signaling.gateway.ts"]:::backend
            MediasoupService["Mediasoup.service.ts"]:::backend
        end

        subgraph "Media Utils"
            direction TB
            FFmpegUtil["ffmpeg.ts"]:::backend
            SDPUtil["sdp.ts"]:::backend
        end
    end

    %% Python ML Server
    subgraph "Python ML Server"
        direction TB
        EnvDef["environment.yml"]:::ml
        MainML["_main.py"]:::ml
        subgraph "Pipelines"
            direction TB
            TTSWorker["seamless_t2S_worker.py"]:::ml
            VoiceCloneWorker["voice_clone_worker.py"]:::ml
            EmbeddingsStore["voice_embeddings/"]:::ml
        end
    end

    %% Connections: Signaling & Media Flow
    Client1 -->|"capture/transmit"| MediaClient
    Client2 -->|"capture/transmit"| MediaClient
    MediaClient -->|"signal (socket.io)"| SocketClient
    SocketClient -->|"signal"| SignalingGateway
    SignalingGateway -->|"create transports"| MediasoupService
    MediasoupService -->|"route RTP streams"| MediasoupService
    MediasoupService -->|"pipe media"| FFmpegUtil
    FFmpegUtil -->|"forward audio"| MainML
    MainML -->|"dispatch to"| TTSWorker
    MainML -->|"dispatch to"| VoiceCloneWorker
    TTSWorker -->|"synthesized audio"| MainML
    VoiceCloneWorker -->|"cloned audio"| MainML
    MainML -->|"return processed audio"| MediasoupService
    MediasoupService -->|"distribute media"| MediaClient
    MediaClient -->|"render playback"| Client1
    MediaClient -->|"render playback"| Client2

    %% Click events
    click MainFE "https://github.com/amitvermanit/audio_call/blob/main/frontend/src/main.tsx"
    click MediaClient "https://github.com/amitvermanit/audio_call/blob/main/frontend/src/mediasoupClient.ts"
    click SocketClient "https://github.com/amitvermanit/audio_call/blob/main/frontend/src/socket.ts"
    click SignalingGateway "https://github.com/amitvermanit/audio_call/blob/main/backend/src/calls/signaling.gateway.ts"
    click SignalingModule "https://github.com/amitvermanit/audio_call/blob/main/backend/src/calls/signaling.module.ts"
    click MediasoupService "https://github.com/amitvermanit/audio_call/blob/main/backend/src/calls/Mediasoup.service.ts"
    click MainBE "https://github.com/amitvermanit/audio_call/blob/main/backend/src/main.ts"
    click AppController "https://github.com/amitvermanit/audio_call/blob/main/backend/src/app.controller.ts"
    click AppService "https://github.com/amitvermanit/audio_call/blob/main/backend/src/app.service.ts"
    click AppModule "https://github.com/amitvermanit/audio_call/blob/main/backend/src/app.module.ts"
    click FFmpegUtil "https://github.com/amitvermanit/audio_call/blob/main/backend/src/lib/ffmpeg.ts"
    click SDPUtil "https://github.com/amitvermanit/audio_call/blob/main/backend/src/lib/sdp.ts"
    click MainML "https://github.com/amitvermanit/audio_call/blob/main/ml2/_main.py"
    click EnvDef "https://github.com/amitvermanit/audio_call/blob/main/ml2/environment.yml"
    click TTSWorker "https://github.com/amitvermanit/audio_call/blob/main/ml2/pipelines/seamless_t2s/seamless_t2S_worker.py"
    click VoiceCloneWorker "https://github.com/amitvermanit/audio_call/blob/main/ml2/pipelines/voice_clone/voice_clone_worker.py"
    click EmbeddingsStore "https://github.com/amitvermanit/audio_call/tree/main/ml2/pipelines/voice_clone/voice_embeddings/"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#0366d6,color:#0366d6
    classDef backend fill:#d4edda,stroke:#28a745,color:#155724
    classDef ml fill:#fff3cd,stroke:#ffc107,color:#856404
```
