# vita-mllm/vita architecture

> Architecture diagram of the GitHub repository vita-mllm/vita, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vita-mllm/vita
- Repository: https://github.com/vita-mllm/vita

## Overview

##### Main Components (Top Level)

a) Core Processing Units:
- Vision Processing Module (InternViT-based)
- Audio Processing Module (Speech Recognition & TTS)
- Language Processing Module (Qwen2.5-based)
- Multimodal Integration Hub

b) Interface Layer:
- Web Demo Interface
- Real-time Interactive System
- Evaluation Framework

##### Detailed Component Breakdown

a) Vision Processing Pipeline:
- Image Encoder (InternViT-300M-448px)
- Video Frame Processor
- Visual Feature Extractor

b) Audio Processing Pipeline:
- ASR Module
- End-to-end TTS Module
- VAD (Voice Activity Detection)

c) Language Model Integration:
- Qwen2.5 Base Model
- Multimodal Projector
- Cross-modal Attention Mechanisms

##### Key Architectural Elements

a) Data Flow Paths:
- Image/Video Input → Vision Processing → Multimodal Integration
- Audio Input → Speech Processing → Text Conversion
- Text Generation → TTS → Audio Output

b) Integration Points:
- Vision-Language Fusion
- Audio-Language Alignment
- Cross-modal Feature Fusion

##### System Features to Highlight

a) Real-time Processing:
- Low-latency Pipeline (1.5s end-to-end)
- Dynamic Patch Processing
- Parallel Processing Units

b) Model Components:
- VITA-1.5 Checkpoint
- Multimodal Encoders
- Progressive Training Integration

##### Additional Elements

a) Support Systems:
- VLMEvalKit Integration
- Benchmark Evaluation Pipeline
- Web Demo Infrastructure

b) Development Tools:
- Training Pipeline
- Evaluation Framework
- Demo Deployment System

##### Performance Metrics Display

- ASR WER Performance (7.5)
- Multimodal Benchmark Scores (70.8)
- Latency Metrics (1.5s)

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Core
        MM[Multimodal Integration Hub]:::integration
        VP[Vision Processing Module]:::vision
        AP[Audio Processing Module]:::audio
        LP[Language Processing Module]:::language
    end

    subgraph Vision_Pipeline
        IE[Image Encoder<br>InternViT-300M]:::vision
        VFP[Video Frame Processor]:::vision
        VFE[Visual Feature Extractor]:::vision
    end

    subgraph Audio_Pipeline
        ASR[ASR Module]:::audio
        TTS[TTS Module]:::audio
        VAD[Voice Activity Detection]:::audio
    end

    subgraph Language_Integration
        QM[Qwen2.5 Base Model]:::language
        MP[Multimodal Projector]:::integration
        CMA[Cross-modal Attention]:::integration
    end

    subgraph Interface
        WD[Web Demo Interface]:::interface
        RTS[Real-time System]:::interface
        EF[Evaluation Framework]:::interface
    end

    subgraph Performance
        PM1[ASR WER: 7.5]:::metric
        PM2[Benchmark: 70.8]:::metric
        PM3[Latency: 1.5s]:::metric
    end

    %% Core Connections
    VP --> MM
    AP --> MM
    LP --> MM

    %% Vision Pipeline
    VP --> IE
    IE --> VFP
    VFP --> VFE
    VFE --> MM

    %% Audio Pipeline
    AP --> ASR
    AP --> TTS
    AP --> VAD
    ASR --> MM
    TTS --> MM
    VAD --> RTS

    %% Language Integration
    LP --> QM
    QM --> MP
    MP --> CMA
    CMA --> MM

    %% Interface Connections
    MM --> WD
    MM --> RTS
    MM --> EF

    %% Performance Metrics
    EF --> PM1
    EF --> PM2
    EF --> PM3

    %% Click Events
    click VP "https://github.com/VITA-MLLM/VITA/tree/main/vita/model/multimodal_encoder/internvit/"
    click AP "https://github.com/VITA-MLLM/VITA/tree/main/vita/model/vita_tts/"
    click LP "https://github.com/VITA-MLLM/VITA/blob/main/vita/model/language_model/vita_qwen2.py"
    click MM "https://github.com/VITA-MLLM/VITA/tree/main/vita/model/multimodal_projector/"
    click WD "https://github.com/VITA-MLLM/VITA/tree/main/web_demo/vita_html/"
    click RTS "https://github.com/VITA-MLLM/VITA/tree/main/web_demo/wakeup_and_vad/"
    click EF "https://github.com/VITA-MLLM/VITA/tree/main/VLMEvalKit/"
    click VAD "https://github.com/VITA-MLLM/VITA/blob/main/web_demo/wakeup_and_vad/wakeup_and_vad.py"
    click TTS "https://github.com/VITA-MLLM/VITA/blob/main/vita/model/vita_tts/pipeline.py"

    %% Styling
    classDef vision fill:#a8d5ff,stroke:#333
    classDef audio fill:#98ff98,stroke:#333
    classDef language fill:#ffb366,stroke:#333
    classDef integration fill:#e6b3ff,stroke:#333
    classDef interface fill:#f2f2f2,stroke:#333
    classDef metric fill:#ffcccc,stroke:#333
```
