# ictnlp/llama-omni architecture

> Architecture diagram of the GitHub repository ictnlp/llama-omni, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ictnlp/llama-omni
- Repository: https://github.com/ictnlp/llama-omni

## Overview

##### Project Type & Purpose
- The README highlights low-latency, simultaneous generation of both text and speech responses. It also details components for local inference and a Gradio-based web demo.

##### Analyzing the File Structure
- Top-level Files: Standard configuration files (.gitignore, LICENSE, README.md, pyproject.toml) and a cog.yaml (typically for containerization or deployment).
- Core Directory (omni_speech): Contains all the main code. Within this directory:
- General utility modules (arguments.py, constants.py, conversation.py, utils.py).
- A "datasets" subdirectory with preprocessing scripts indicating an interface to prepare or preload data.
- An "infer" directory containing inference scripts, examples with sample JSONL files and WAV audio files, and a shell script for local inference.
- A "model" subdirectory broken down into:
- language_model: Contains adapters or wrappers for Llama variants.
- speech_encoder, speech_generator, speech_projector: Each with a builder and a main entry, reflecting modular, object-oriented approaches for handling input (speech encoder), generating speech (speech generator), and aligning modalities or projecting features (speech projector).
- A "serve" directory with components to actually run the service:
- controller.py: Likely coordinates or manages request routing.
- gradio_web_server.py: Provides the web interface through Gradio.
- model_worker.py: Manages actual model inference and processing.
- Additional files like predict.py can serve as another entry point for quick prediction or testing.

##### Key Components
A. Frontend / Presentation Layer:
B. Controller & Routing:
- Controller Service (omni_speech/serve/controller.py): Acts as a mediator between the Web UI and model workers.
C. Inference & Model Worker:
- Model Worker (omni_speech/serve/model_worker.py) that handles inbound requests and passes them through the model pipeline:
- This pipeline consists of the speech encoder, language model, speech generator, and speech projector (inhabiting modules under omni_speech/model/).
D. Core Model Components (under omni_speech/model):
- Speech Encoder (omni_speech/model/speech_encoder/…): Converts speech input into an intermediate representation.
- Speech Generator (omni_speech/model/speech_generator/…): Converts processed info into synthesized speech.
- Speech Projector (omni_speech/model/speech_projector/…): Likely aligns and fuses different modalities.
E. Supporting Utilities:
- Datasets and Preprocessing (omni_speech/datasets): Prepares input data.
- Inference Scripts (omni_speech/infer) and helper scripts (predict.py).

##### Patterns & Interactions
- The Gradio Web UI receives speech input from a user.
- The Controller routes the requests to the Model Worker.
- The Model Worker orchestrates the operations using:
- The speech encoder to interpret and transcribe the audio input.
- The language model to generate a text response.
- The speech generator (and possibly speech projector) to produce corresponding audio output.
- In local inference, the “infer” directory scripts process pre-organized speech instructions.
- Integration of external libraries like fairseq, flash-attn, and Whisper, which support various stages of the data handling and model execution.

## Diagram (Mermaid)

```mermaid
flowchart TD
    User("User"):::external

    subgraph "Frontend Layer"
        GWS("Gradio Web Server"):::frontend
    end

    subgraph "Middleware Layer"
        CTRL("Controller"):::middleware
        MW("Model Worker"):::middleware
    end

    subgraph "Core Model Components"
        SE("Speech Encoder"):::model
        LM("Language Model"):::model
        SG("Speech Generator"):::model
        SP("Speech Projector"):::model
    end

    subgraph "Support Components"
        DP("Datasets & Preprocessing"):::support
        IS("Inference Scripts"):::support
        PE("Predict Entry Point"):::support
    end

    Note1("External Tools: fairseq, flash-attn, Whisper"):::externalTool

    User -->|"initiates"| GWS
    GWS -->|"sends_request"| CTRL
    CTRL -->|"routes_to"| MW
    MW -->|"processes_audio"| SE
    SE -->|"transcribes_to"| LM
    LM -->|"generates_text"| SG
    LM -->|"aligns_modalities"| SP
    DP -->|"prepares_data"| MW
    IS -->|"triggers_local_inference"| MW
    PE -->|"quick_prediction"| MW
    Note1 -.-> SE

    click GWS "https://github.com/ictnlp/llama-omni/blob/main/omni_speech/serve/gradio_web_server.py"
    click CTRL "https://github.com/ictnlp/llama-omni/blob/main/omni_speech/serve/controller.py"
    click MW "https://github.com/ictnlp/llama-omni/blob/main/omni_speech/serve/model_worker.py"
    click LM "https://github.com/ictnlp/llama-omni/tree/main/omni_speech/model/language_model/"
    click SE "https://github.com/ictnlp/llama-omni/tree/main/omni_speech/model/speech_encoder/"
    click SG "https://github.com/ictnlp/llama-omni/tree/main/omni_speech/model/speech_generator/"
    click SP "https://github.com/ictnlp/llama-omni/tree/main/omni_speech/model/speech_projector/"
    click DP "https://github.com/ictnlp/llama-omni/tree/main/omni_speech/datasets/"
    click IS "https://github.com/ictnlp/llama-omni/tree/main/omni_speech/infer/"
    click PE "https://github.com/ictnlp/llama-omni/blob/main/predict.py"

    classDef frontend fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef middleware fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef model fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef support fill:#D3D3D3,stroke:#000,stroke-width:1px;
    classDef external fill:#FFC0CB,stroke:#000,stroke-width:1px;
    classDef externalTool fill:#FFFF99,stroke:#000,stroke-width:1px;
```
