# vita-mllm/freeze-omni architecture

> Architecture diagram of the GitHub repository vita-mllm/freeze-omni, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vita-mllm/freeze-omni
- Repository: https://github.com/vita-mllm/freeze-omni

## Overview

##### Main System Components

a) Core Processing Pipeline:
- Speech Encoder (Streaming Input)
- Speech Decoder (AR-based)
- State Prediction System

b) Server Architecture:
- Model Pool Management
- VAD (Voice Activity Detection)
- Request Queue Handler
- User Session Management

##### Component Relationships and Data Flow

INPUT LAYER:
- Chunk-wise streaming input processing
- To VAD system (from web/vad.py)

PROCESSING LAYER:
- Speech Encoder (models/encoder/*)
- Subcomponents: attention, CMVN, transformer
- LLM Integration (models/audioLLM.py)
- Frozen state
- Decoder System (models/decoder/*)
- TiCodec integration
- AR-based speech generation

OUTPUT LAYER:
- Streaming speech output
- State prediction feedback loop

##### Specific Architectural Elements

RECTANGLES (Blue):
- Core processing modules
- Server components

- Model weights/checkpoints
- Cache storage

DIAMONDS (Yellow):
- Decision points
- State prediction nodes

- Solid: Direct data flow

##### Technical Details

Annotations for:
- Input chunk size
- Model configurations
- Latency metrics
- Cache management strategy
- Load balancing logic

##### Implementation Notes

- Main inference pipeline (bin/inference.py)
- Server implementation (bin/server.py)
- Web interface components (web/*)

##### Chunk-level State Prediction

This will create a comprehensive yet clear representation of the system's architecture that accurately reflects its implementation and design choices.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Input
        SI[Streaming Input] --> VAD
        VAD[Voice Activity Detection] --> CW[Chunk-wise Processing]
        click VAD "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/web/vad.py"
    end

    subgraph Processing
        direction TB
        CW --> SE[Speech Encoder]
        click SE "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/encoder/encoder.py"

        subgraph Encoder["Speech Encoder Components"]
            direction TB
            AT[Attention] --> TR[Transformer]
            CM[CMVN] --> TR
            click AT "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/encoder/attention.py"
            click CM "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/encoder/cmvn.py"
            click TR "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/encoder/transformer.py"
        end

        SE --> LLM[LLM Integration]
        click LLM "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/audioLLM.py"

        LLM --> AD[Adapter Layer]
        click AD "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/adapter.py"

        AD --> SD[Speech Decoder]
        click SD "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/decoder/decoder.py"

        subgraph Decoder["Decoder Components"]
            direction TB
            TI[TiCodec] --> L2T[LLM to TTS]
            click TI "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/decoder/ticodec/models.py"
            click L2T "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/decoder/llm2tts.py"
        end
    end

    subgraph Server["Server Architecture"]
        direction TB
        MP[Model Pool] --> RQ[Request Queue]
        RQ --> US[User Session]
        click MP "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/web/pool.py"
        click RQ "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/web/queue.py"
    end

    subgraph Pipeline["Core Pipeline"]
        direction TB
        INF[Inference System] --> PL[Pipeline Manager]
        click INF "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/bin/inference.py"
        click PL "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/pipeline.py"
    end

    subgraph Web["Web Interface"]
        WI[Web Demo]
        click WI "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/web/resources/demo.html"
    end

    subgraph Utils["Utility Components"]
        MH[Mask Handler] --> UT[Utils]
        click MH "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/masks.py"
        click UT "https://github.com/VITA-MLLM/Freeze-Omni/blob/main/models/utils.py"
    end

    %% Connections
    Input --> Processing
    Processing --> Server
    Server --> Pipeline
    Pipeline --> Web
    Utils --> Processing

    %% Styles
    classDef default fill:#f9f9f9,stroke:#333,stroke-width:2px
    classDef input fill:#d4e6f1,stroke:#2874a6
    classDef process fill:#d5f5e3,stroke:#196f3d
    classDef server fill:#fdebd0,stroke:#d35400
    classDef web fill:#ebdef0,stroke:#8e44ad
    classDef utils fill:#f2f4f4,stroke:#566573

    class SI,VAD,CW input
    class SE,LLM,AD,SD,AT,CM,TR,TI,L2T process
    class MP,RQ,US server
    class WI web
    class MH,UT utils
```
