# facebookresearch/seamless_communication architecture

> Architecture diagram of the GitHub repository facebookresearch/seamless_communication, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/seamless_communication
- Repository: https://github.com/facebookresearch/seamless_communication

## Overview

##### Project Type and Purpose
– This repository is a hybrid Python/C++/Web ML framework (“Seamless Communication”) for end-to-end multilingual speech and text translation.
– It packages:
- A core Python library (`src/seamless_communication`) exposing CLI commands for inference, evaluation, finetuning, streaming, toxicity checks, etc.
- A C/C++ tensor‐library backend (`ggml`) plus a small C++ “unity.cpp” executable for on-device GGML inference.
- Web/Gradio demos (`demo/`) for online interactive UIs.
- Extensive documentation (`docs/`) and test suites (`tests/`).

##### File Structure and Architectural Patterns
– Top-level directories:
- `src/seamless_communication/`: the core Python package, subdivided into modules for CLI, inference, streaming, denoise, models, datasets, toxicity, etc.
- `ggml/`: the C tensor library, build scripts, examples, and the on-device “unity” binary.
- `demo/`: two Gradio/Python demo apps (`expressive`, `m4tv*`) and a pure-JS/HTML demo (`dino_pretssel`).
- `docs/`: architecture and usage guides per subsystem.
- `tests/`: unit and integration tests verifying each module.
– Architectural patterns:
- _Modular CLI pattern_: top-level console commands (`m4t_predict`, `expressivity_predict`, `streaming_evaluate`, etc.) hook into `src/seamless_communication/cli/*`.
- _Builder/Loader/Model_ pattern under `models/`: each model type has a builder, loader, and model definition.
- _Pipeline/Agent_ pattern in `streaming/agents`: discrete streaming components chained via SimulEval.
- _Configuration via YAML cards_: model hyperparameters live in `cards/*.yaml`.

##### README Insights
– Describes four major “products”: SeamlessM4T (all-in-one translation), SeamlessExpressive (prosody + voice style), SeamlessStreaming (real-time streaming), and the unified Seamless model.
– Highlights external library dependencies: `fairseq2`, `SONAR`, `stopes`, `SimulEval`.
– Shows usage flows: CLI invocation, Gradio demo startup, on-device unity.cpp usage.

##### Main Components
a. Presentation Layer
- CLI commands: `m4t_predict`, `expressivity_predict`, `streaming_evaluate`, `etox`, `mutox`.
- Demo UIs: Gradio apps (`demo/expressive/app.py`, `demo/m4tv*`), JS/HTML demo (`demo/dino_pretssel/`).

b. Application/Orchestration Layer
- Inference orchestrator (`src/seamless_communication/inference/*`): routes input -> generator, translator, transcriber.
- Streaming pipeline (`src/seamless_communication/streaming/agents/*`): agents implement SimulEval interface, chained to process streaming audio/text.

c. Domain/Model Layer
- Models (speech encoder, aligner, generator, monotonic decoder, unity, vocoder, toxicity): each in `models/*` with builder, loader, model code.
- Pre/Post-processing: denoise (`denoise/demucs.py`), segmentation (`segment/silero_vad.py`), dataset adapters (`datasets/*`).
- Config cards (`cards/*.yaml`) drive hyperparameters & checkpoint locations.

d. Infrastructure Layer
- External libraries: fairseq2, SONAR, stopes, SimulEval.
- On-device backend: `ggml` (C build, Makefiles, examples) & `unity.cpp` binary using GGML models.
- Storage: local file system (audio inputs, outputs, checkpoints), Hugging Face model hub.

e. Data Layer
- Input: audio files, text, streaming audio chunks.
- Intermediate: waveform tensors, mel-filterbanks, token sequences, unit representations.
- Output: translated text, synthesized audio.

##### Relationships and Data Flows
1) User → CLI or UI → Orchestrator
2) Orchestrator → Pre-processing → Model inference → Post-processing
3) Model inference → calls into fairseq2 (Python) or ggml (C)
4) Streaming: microphone → online encoder → decoder agents → vocoder → speaker output
5) On-device: `unity` binary → GGML runtime → model file → output

##### Architectural Patterns & Design Principles
– Separation of concerns: distinct packages for CLI, inference, streaming, model definitions, backend.
– Extensibility: builder/loader abstractions allow new models to plug in.
– Pipeline/agent pattern: streaming assembled as modular agents via SimulEval interface.
– Configuration-driven: YAML “cards” decouple hyperparams/checkpoints from code.

##### Technology & Frameworks
– Python: `src/seamless_communication/*`, CLI via `argparse` or `click`.
– C/C++: `ggml` for tensor ops; `unity.cpp` for GGML on-device inference.
– Web: Gradio (Python demos), HTML/JS (`wavesurfer.js`) for audio visualization.
– Build: CMake, Makefile, Zig (build.zig) under `ggml`.
– Testing: `pytest` under `tests/`.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        User["User"]:::external
        M4T_CLI["M4T CLI"]:::presentation
        Expressive_CLI["Expressive CLI"]:::presentation
        Streaming_CLI["Streaming CLI"]:::presentation
        Toxicity_CLI["Toxicity CLI"]:::presentation
        Expressive_UI["Expressive Gradio UI"]:::presentation
        M4TV1_UI["M4TV1 Gradio UI"]:::presentation
        M4TV2_UI["M4TV2 Gradio UI"]:::presentation
        Dino_UI["Dino Pretssel Demo"]:::presentation
    end

    %% Application Layer
    subgraph "Application/Orchestration Layer"
        direction TB
        Orchestrator["Inference Orchestrator"]:::application
        Stream_Agents["Streaming Pipeline Agents"]:::application
        Stream_Loaders["Streaming Data Loaders"]:::application
    end

    %% Domain/Model Layer
    subgraph "Domain/Model Layer"
        direction TB
        Config_Cards["Configuration Cards (YAML)"]:::domain
        Denoise["Speech Denoising"]:::domain
        VAD["Voice Activity Detection"]:::domain
        Datasets["Dataset Adapters"]:::domain
        Aligner["Aligner Model"]:::domain
        Conformer["Conformer Shaw Encoder"]:::domain
        Generator["Generator Model"]:::domain
        Monotonic["Monotonic Decoder"]:::domain
        Pretssel["Pretssel (Prosody) Model"]:::domain
        UnitExtractor["Unit Extractor"]:::domain
        UnityModel["Unity Unified Model"]:::domain
        Vocoder["Vocoder"]:::domain
        Toxicity["Toxicity Modules"]:::domain
    end

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        direction TB
        Fairseq2["fairseq2"]:::external
        SONAR["SONAR"]:::external
        STOPES["stopes"]:::external
        SimulEval["SimulEval"]:::external
        Unity_Binary["unity.cpp Binary"]:::infra
        GGML_Code["GGML Tensor Library"]:::infra
        GGML_Build["GGML Build Scripts"]:::infra
        GGML_Py["Python-GGML Bindings"]:::infra
        FileSystem["Local FileSystem"]:::infra
        HF_Hub["Hugging Face Hub"]:::infra
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Audio_In["Audio Files/Text Input"]:::data
        Stream_Chunks["Streaming Audio Chunks"]:::data
        Wave_Tensor["Waveform Tensors"]:::data
        Mel_FB["Mel-filterbanks"]:::data
        Token_Seq["Token Sequences"]:::data
        Unit_Reps["Unit Representations"]:::data
        Text_Out["Translated Text"]:::data
        Audio_Out["Synthesized Audio"]:::data
    end

    %% Relationships
    User --> M4T_CLI
    User --> Expressive_CLI
    User --> Streaming_CLI
    User --> Toxicity_CLI
    User --> Expressive_UI
    User --> M4TV1_UI
    User --> M4TV2_UI
    User --> Dino_UI

    M4T_CLI --> Orchestrator
    Expressive_CLI --> Orchestrator
    Toxicity_CLI --> Orchestrator
    Expressive_UI --> Orchestrator
    M4TV1_UI --> Orchestrator
    M4TV2_UI --> Orchestrator

    Streaming_CLI --> Stream_Loaders
    Streaming_CLI --> Stream_Agents
    Expressive_UI --> Stream_Agents

    Orchestrator --> Config_Cards
    Orchestrator --> Denoise
    Orchestrator --> VAD
    Orchestrator --> Datasets
    Orchestrator --> Aligner
    Orchestrator --> Conformer
    Orchestrator --> Generator
    Orchestrator --> Monotonic
    Orchestrator --> Pretssel
    Orchestrator --> UnitExtractor
    Orchestrator --> UnityModel
    Orchestrator --> Vocoder
    Orchestrator --> Toxicity

    Audio_In --> Denoise
    Audio_In --> VAD
    Stream_Chunks --> Stream_Loaders
    Stream_Loaders --> Stream_Agents
    Stream_Agents --> Vocoder
    Vocoder --> Audio_Out

    Denoise --> Wave_Tensor
    VAD --> Wave_Tensor
    Wave_Tensor --> Mel_FB
    Mel_FB --> Token_Seq
    Token_Seq --> Unit_Reps
    Unit_Reps --> Generator
    Generator --> Audio_Out
    Token_Seq --> Conformer
    Conformer --> Token_Seq
    Token_Seq --> Monotonic
    Monotonic --> Token_Seq
    Token_Seq --> Aligner
    Aligner --> Token_Seq
    Token_Seq --> Pretssel
    Pretssel --> Token_Seq
    Token_Seq --> UnitExtractor
    UnitExtractor --> Unit_Reps
    UnityModel --> Audio_Out
    Toxicity --> Text_Out
    Text_Out --> User

    Aligner --> Fairseq2
    Conformer --> Fairseq2
    Generator --> Fairseq2
    Monotonic --> Fairseq2
    Pretssel --> Fairseq2
    UnitExtractor --> Fairseq2
    Vocoder --> Fairseq2

    Unity_Binary --> GGML_Code
    GGML_Code --> GGML_Build
    GGML_Build --> Unity_Binary
    Unity_Binary --> UnityModel
    Unity_Binary --> Audio_Out
    GGML_Py --> Unity_Binary
    UnityModel --> GGML_Py

    Config_Cards --> Orchestrator

    %% Storage
    Audio_In --> FileSystem
    Audio_Out --> FileSystem
    UnityModel --> HF_Hub

    %% Click Events
    click M4T_CLI "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/cli/m4t/"
    click Expressive_CLI "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/cli/expressivity/"
    click Streaming_CLI "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/cli/streaming/"
    click Toxicity_CLI "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/cli/toxicity/"
    click Orchestrator "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/inference/"
    click Stream_Agents "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/streaming/agents/"
    click Stream_Loaders "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/streaming/dataloaders/"
    click Denoise "https://github.com/facebookresearch/seamless_communication/blob/main/src/seamless_communication/denoise/demucs.py"
    click VAD "https://github.com/facebookresearch/seamless_communication/blob/main/src/seamless_communication/segment/silero_vad.py"
    click Datasets "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/datasets/"
    click Aligner "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/aligner/"
    click Conformer "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/conformer_shaw/"
    click Generator "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/generator/"
    click Monotonic "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/monotonic_decoder/"
    click Pretssel "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/pretssel/"
    click UnitExtractor "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/unit_extractor/"
    click UnityModel "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/unity/"
    click Vocoder "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/models/vocoder/"
    click Config_Cards "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/cards/"
    click Toxicity "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/toxicity/"
    click Unity_Binary "https://github.com/facebookresearch/seamless_communication/blob/main/ggml/examples/unity/unity.cpp"
    click GGML_Code "https://github.com/facebookresearch/seamless_communication/tree/main/ggml/src/"
    click GGML_Code "https://github.com/facebookresearch/seamless_communication/tree/main/ggml/include/ggml/"
    click GGML_Build "https://github.com/facebookresearch/seamless_communication/blob/main/ggml/CMakeLists.txt"
    click GGML_Build "https://github.com/facebookresearch/seamless_communication/tree/main/ggml/Makefile"
    click GGML_Build "https://github.com/facebookresearch/seamless_communication/blob/main/ggml/build.zig"
    click GGML_Py "https://github.com/facebookresearch/seamless_communication/blob/main/ggml/ggml.py"
    click GGML_Py "https://github.com/facebookresearch/seamless_communication/blob/main/ggml/ggml_convert.py"
    click Expressive_UI "https://github.com/facebookresearch/seamless_communication/blob/main/demo/expressive/app.py"
    click M4TV1_UI "https://github.com/facebookresearch/seamless_communication/blob/main/demo/m4tv1/app.py"
    click M4TV2_UI "https://github.com/facebookresearch/seamless_communication/blob/main/demo/m4tv2/app.py"
    click Dino_UI "https://github.com/facebookresearch/seamless_communication/blob/main/demo/dino_pretssel/index.html"
    click Config_Cards "https://github.com/facebookresearch/seamless_communication/tree/main/src/seamless_communication/cards/"
    click Documentation "https://github.com/facebookresearch/seamless_communication/tree/main/docs/"
    click Tests "https://github.com/facebookresearch/seamless_communication/tree/main/tests/"

    %% Styles
    classDef presentation fill:#DEEBFF,stroke:#0366d6
    classDef application fill:#E6FFED,stroke:#28a745
    classDef domain fill:#FFF4E5,stroke:#D08B00
    classDef infra fill:#F0F0F0,stroke:#636363
    classDef data fill:#E8F4F8,stroke:#1C7ED6
    classDef external fill:#F0F0F0,stroke:#636363,stroke-dasharray: 5 5
```
