How vita-mllm/freeze-omni works
Main System Components
a) Core Processing Pipeline:
• Speech Encoder (Streaming Input)
• Speech Decoder (AR-based)
• State Prediction System
b) Server Architecture:
• Model Pool Management
• VAD (Voice Activity Detection)
• Request Queue Handler
• User Session Management
Component Relationships and Data Flow
INPUT LAYER:
• Chunk-wise streaming input processing
• To VAD system (from web/vad.py)
PROCESSING LAYER:
• Speech Encoder (models/encoder/*)
• Subcomponents: attention, CMVN, transformer
• LLM Integration (models/audioLLM.py)
• Frozen state
• Decoder System (models/decoder/*)
• TiCodec integration
• AR-based speech generation
OUTPUT LAYER:
• Streaming speech output
• State prediction feedback loop
Specific Architectural Elements
RECTANGLES (Blue):
• Core processing modules
• Server components
• Model weights/checkpoints
• Cache storage
DIAMONDS (Yellow):
• Decision points
• State prediction nodes
• Solid: Direct data flow
Technical Details
Annotations for:
• Input chunk size
• Model configurations
• Latency metrics
• Cache management strategy
• Load balancing logic
Implementation Notes
• Main inference pipeline (bin/inference.py)
• Server implementation (bin/server.py)
• Web interface components (web/*)
Chunk-level State Prediction
This will create a comprehensive yet clear representation of the system's architecture that accurately reflects its implementation and design choices.