How ictnlp/llama-omni works
Project Type & Purpose
• The README highlights low-latency, simultaneous generation of both text and speech responses. It also details components for local inference and a Gradio-based web demo.
Analyzing the File Structure
• Top-level Files: Standard configuration files (.gitignore, LICENSE, README.md, pyproject.toml) and a cog.yaml (typically for containerization or deployment).
• Core Directory (omni_speech): Contains all the main code. Within this directory:
• General utility modules (arguments.py, constants.py, conversation.py, utils.py).
• A "datasets" subdirectory with preprocessing scripts indicating an interface to prepare or preload data.
• An "infer" directory containing inference scripts, examples with sample JSONL files and WAV audio files, and a shell script for local inference.
• A "model" subdirectory broken down into:
• language_model: Contains adapters or wrappers for Llama variants.
• speech_encoder, speech_generator, speech_projector: Each with a builder and a main entry, reflecting modular, object-oriented approaches for handling input (speech encoder), generating speech (speech generator), and aligning modalities or projecting features (speech projector).
• A "serve" directory with components to actually run the service:
• controller.py: Likely coordinates or manages request routing.
• gradio_web_server.py: Provides the web interface through Gradio.
• model_worker.py: Manages actual model inference and processing.
• Additional files like predict.py can serve as another entry point for quick prediction or testing.
Key Components
A. Frontend / Presentation Layer:
B. Controller & Routing:
• Controller Service (omni_speech/serve/controller.py): Acts as a mediator between the Web UI and model workers.
C. Inference & Model Worker:
• Model Worker (omni_speech/serve/model_worker.py) that handles inbound requests and passes them through the model pipeline:
• This pipeline consists of the speech encoder, language model, speech generator, and speech projector (inhabiting modules under omni_speech/model/).
D. Core Model Components (under omni_speech/model):
• Speech Encoder (omni_speech/model/speech_encoder/…): Converts speech input into an intermediate representation.
• Speech Generator (omni_speech/model/speech_generator/…): Converts processed info into synthesized speech.
• Speech Projector (omni_speech/model/speech_projector/…): Likely aligns and fuses different modalities.
E. Supporting Utilities:
• Datasets and Preprocessing (omni_speech/datasets): Prepares input data.
• Inference Scripts (omni_speech/infer) and helper scripts (predict.py).
Patterns & Interactions
• The Gradio Web UI receives speech input from a user.
• The Controller routes the requests to the Model Worker.
• The Model Worker orchestrates the operations using:
• The speech encoder to interpret and transcribe the audio input.
• The language model to generate a text response.
• The speech generator (and possibly speech projector) to produce corresponding audio output.
• In local inference, the “infer” directory scripts process pre-organized speech instructions.
• Integration of external libraries like fairseq, flash-attn, and Whisper, which support various stages of the data handling and model execution.