# apple/ml-ferret architecture

> Architecture diagram of the GitHub repository apple/ml-ferret, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apple/ml-ferret
- Repository: https://github.com/apple/ml-ferret

## Overview

##### Main Components
- Core Model & Training: Represented by the “ferret” directory. This includes:
– Model definition and utility modules (e.g., ferret/model with submodules for language models, multimodal encoder, and projector).
– Training scripts (ferret/train, including training routines, hyperparameter settings, and delta application via apply_delta).
– Evaluation routines (ferret/eval and its corresponding scripts to run evaluations on different datasets and benchmarks).
- Serving/Inference: Found under “ferret/serve”. It includes:
– Controller (controller.py that orchestrates model workers).
– Model workers (model_worker.py, register_worker.py) that load the checkpoint and perform inference on GPUs.
– Auxiliary serving components (gradio web server for UI integration and additional utilities).
- User Interface – FerretUI: The “ferretui” directory provides a UI-centric module.
– Web interfaces and Gradio components (gradio_web_server.py, CLI scripts and examples).
– It re-implements model-serving logic (mirrors many of the “ferret” modules) and includes additional evaluation and demo components.
- Experiment Scripts & Supporting Material:
– “experiments” folder houses bash scripts for training variants (7B and 13B).
– “scripts”, “figs”, and additional playground/sample data directories serve for demo displays, testing, and visualization.
- External Dependencies & Data:
– The README details the use of external pre-trained models (Vicuna, LLaVA projector weights).
– Evaluation datasets and checkpoints (found in ferret/eval/ferret_gpt4_data and referenced in the README).

##### Relationships and Interactions
- The Training block (ferret/train) prepares the updated model parameters and applies “delta” differences to the base Vicuna checkpoint.
- The Core Model block (ferret/model) serves as the foundation that both the inference system and evaluation modules use.
- The Evaluation block (ferret/eval & ferretui/ferretui/eval) takes outputs from the model and performs various assessments (referring, grounding, review scripts).
- The Gradio Web Server interacts with the Controller to receive and display inference results.
- The UI block (ferretui) encapsulates a web front-end that interacts with the Serving layer.
- Additionally, illustrate the external dependency “Vicuna checkpoint & LLaVA’s Projector” feeding into the Core Model Training block.

##### Important Architectural Patterns & Design Principles
- Separation of Concerns: Different directories for training, evaluation, serving, and UI. This separation supports modularity.
- Distributed Serving & Worker Architecture: The serving framework uses a central controller that dispatches demands to independent model worker processes.
- Hybrid Model Design: The Core Model block comprises both language and multimodal (visual) modules, highlighting the “Hybrid Region Representation” and “Spatial-aware Visual Sampler”.
- Extensibility: The separate FerretUI module provides a dedicated interface and evaluation setup that can be plugged into the core system.

##### Specific Technologies, Frameworks, or Libraries
- Python is the base language.
- Gradio is used for the web interface.
- PyTorch (implicitly, given the training and model serving context) for model training and inference.
- Shell scripting (bash scripts in the “experiments” and “scripts” directories) to manage experiments.
- External dependencies: Vicuna and LLaVA projectors for the base models and weights.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Dependencies
    E["Base Model Weights (Vicuna & LLaVA)"]:::external

    %% Core Model & Training Layer
    subgraph "Core Model & Training"
        T["Training Scripts"]:::modelTraining
        M["Core Model"]:::modelTraining
    end

    %% Serving/Inference Layer
    subgraph "Serving/Inference"
        C["Controller.py"]:::serving
        MW["Model Worker.py"]:::serving
        RS["Gradio Web Server.py"]:::serving
    end

    %% Evaluation Layer
    subgraph "Evaluation"
        EVAL["Backend Eval (ferret/eval)"]:::evaluation
        EVALUI["UI Eval (ferretui/eval)"]:::evaluation
    end

    %% User Interface Layer
    subgraph "User Interface"
        UI["FerretUI Module"]:::ui
    end

    %% Experiments & Supporting Material
    subgraph "Experiments & Support"
        EXP["Experiments Scripts"]:::experiments
        SCR["Additional Scripts"]:::experiments
        FIG["Figs (Visualizations)"]:::experiments
        PL["Playground Data"]:::experiments
    end

    %% End User
    EU["End User"]:::external

    %% Connections
    E -->|"feeds"| T
    T -->|"applies_delta"| M
    M -->|"provides_model"| C
    M -->|"evaluation_output"| EVAL
    M -->|"evaluation_output"| EVALUI

    C -->|"dispatches"| MW
    MW -->|"replies"| C

    RS -->|"queries"| C
    C -->|"sends_results"| RS

    C -->|"inference_commands"| UI
    UI -->|"interactive_demo"| EU

    EXP -->|"triggers_training"| T
    SCR -->|"supports_training"| T
    FIG -->|"visual_assets"| UI
    PL -->|"sample_data"| UI

    %% Styles
    classDef modelTraining fill:#cce5ff,stroke:#0066cc,stroke-width:2px;
    classDef evaluation fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef serving fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef ui fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef experiments fill:#f0f0f0,stroke:#000000,stroke-width:2px;
    classDef external fill:#e2e3e5,stroke:#41464b,stroke-width:2px;

    %% Click Events
    click T "https://github.com/apple/ml-ferret/tree/main/ferret/train"
    click M "https://github.com/apple/ml-ferret/tree/main/ferret/model"
    click EVAL "https://github.com/apple/ml-ferret/tree/main/ferret/eval"
    click C "https://github.com/apple/ml-ferret/blob/main/ferret/serve/controller.py"
    click MW "https://github.com/apple/ml-ferret/blob/main/ferret/serve/model_worker.py"
    click RS "https://github.com/apple/ml-ferret/blob/main/ferret/serve/gradio_web_server.py"
    click UI "https://github.com/apple/ml-ferret/tree/main/ferretui"
    click EXP "https://github.com/apple/ml-ferret/tree/main/experiments"
    click SCR "https://github.com/apple/ml-ferret/tree/main/scripts"
    click FIG "https://github.com/apple/ml-ferret/tree/main/figs"
    click PL "https://github.com/apple/ml-ferret/tree/main/playground"
```
