How apple/ml-ferret works
Main Components
• Core Model & Training: Represented by the “ferret” directory. This includes:
– Model definition and utility modules (e.g., ferret/model with submodules for language models, multimodal encoder, and projector).
– Training scripts (ferret/train, including training routines, hyperparameter settings, and delta application via apply_delta).
– Evaluation routines (ferret/eval and its corresponding scripts to run evaluations on different datasets and benchmarks).
• Serving/Inference: Found under “ferret/serve”. It includes:
– Controller (controller.py that orchestrates model workers).
– Model workers (model_worker.py, register_worker.py) that load the checkpoint and perform inference on GPUs.
– Auxiliary serving components (gradio web server for UI integration and additional utilities).
• User Interface – FerretUI: The “ferretui” directory provides a UI-centric module.
– Web interfaces and Gradio components (gradio_web_server.py, CLI scripts and examples).
– It re-implements model-serving logic (mirrors many of the “ferret” modules) and includes additional evaluation and demo components.
• Experiment Scripts & Supporting Material:
– “experiments” folder houses bash scripts for training variants (7B and 13B).
– “scripts”, “figs”, and additional playground/sample data directories serve for demo displays, testing, and visualization.
• External Dependencies & Data:
– The README details the use of external pre-trained models (Vicuna, LLaVA projector weights).
– Evaluation datasets and checkpoints (found in ferret/eval/ferret_gpt4_data and referenced in the README).
Relationships and Interactions
• The Training block (ferret/train) prepares the updated model parameters and applies “delta” differences to the base Vicuna checkpoint.
• The Core Model block (ferret/model) serves as the foundation that both the inference system and evaluation modules use.
• The Evaluation block (ferret/eval & ferretui/ferretui/eval) takes outputs from the model and performs various assessments (referring, grounding, review scripts).
• The Gradio Web Server interacts with the Controller to receive and display inference results.
• The UI block (ferretui) encapsulates a web front-end that interacts with the Serving layer.
• Additionally, illustrate the external dependency “Vicuna checkpoint & LLaVA’s Projector” feeding into the Core Model Training block.
Important Architectural Patterns & Design Principles
• Separation of Concerns: Different directories for training, evaluation, serving, and UI. This separation supports modularity.
• Distributed Serving & Worker Architecture: The serving framework uses a central controller that dispatches demands to independent model worker processes.
• Hybrid Model Design: The Core Model block comprises both language and multimodal (visual) modules, highlighting the “Hybrid Region Representation” and “Spatial-aware Visual Sampler”.
• Extensibility: The separate FerretUI module provides a dedicated interface and evaluation setup that can be plugged into the core system.
Specific Technologies, Frameworks, or Libraries
• Python is the base language.
• Gradio is used for the web interface.
• PyTorch (implicitly, given the training and model serving context) for model training and inference.
• Shell scripting (bash scripts in the “experiments” and “scripts” directories) to manage experiments.
• External dependencies: Vicuna and LLaVA projectors for the base models and weights.