# salesforceairesearch/latte architecture

> Architecture diagram of the GitHub repository salesforceairesearch/latte, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salesforceairesearch/latte
- Repository: https://github.com/salesforceairesearch/latte

## Overview

##### Project Type and Purpose
- TACO is an open-source research toolkit (not a traditional full-stack web app) for multi-modal “chains-of-thought-and-action” (CoTA) inference, evaluation, and training of large action models.
- Key features:
- Inference orchestration of multi-step tasks (OCR, depth estimation, calculator, etc.) via LLM APIs
- Built-in evaluation (VLMEvalKit) against benchmarks
- Training pipelines for two model families (Mantis and LLaVA-OneVision)

##### File-Structure Analysis
- Top-level directories, by responsibility:
- taco/: Core inference orchestrator (agent + action + parser + executor + prompt + data)
- VLMEvalKit/: Standalone evaluation framework (run.py, run_eval_on_preds.py, dataset definitions, API wrappers)
- autogen/: Shared internal library for LLM API clients, agent chat framework, caching, code execution, logging
- llava/: LLaVA-OneVision model definition, data preprocessing, training scripts, serving code
- mantis/: Mantis model definition, training configurations, training scripts
- scripts/: Convenient bash wrappers for inference+evaluation and training
- Entry points:
- Inference: `python -m taco.run_multimodal_agent` (or via `scripts/infer_and_eval_taco.sh`)
- Evaluation: `python VLMEvalKit/run_eval_on_preds.py`
- Training: `scripts/train_taco_mantis_llava.sh` (Mantis) and `scripts/train_taco_llava_onevision.sh` (LLaVA)

##### README Insights
- Workflow described: install → inference → evaluation → (optionally) training with separate envs.
- External tools invoked during inference (OCR, depth, calculator) are represented by “action” modules under taco/ and by API wrappers under autogen/oai and taco/api.

1. CLI Layer
- Bash scripts (scripts/)
- Python entrypoints (taco/run_multimodal_agent, VLMEvalKit/run_*.py)
2. Inference Orchestrator
- taco.core: agent, action, parser, executor, prompt, observation, data
- autogen.agentchat & autogen.oai: LLM API adapters (OpenAI, Anthropic, Gemini, Qwen, etc.)
- Tool modules under taco.action_utils (OCR, depth, math)
3. Model Backends
- LLaVA-OneVision (llava/model, llava/serve)
- Mantis (mantis/models)
- Other VL models via VLMEvalKit/vlmeval/vlm and its `api/` adapters
4. Data & Evaluation
- VLMEvalKit dataset definitions & utils
- run_eval_on_preds for metric computation
5. Training Pipelines
- llava/train & mantis/train: trainers, data configs, accelerate/zero setups
6. External Services
- LLM APIs (OpenAI, Anthropic, Gemini, Qwen, etc.)
- External tool services (OCR, Depth Estimation, Calculator)

1. CLI → Inference Orchestrator: user invokes `run_multimodal_agent`
2. Orchestrator → autogen.oai: send prompt for chain-of-thought
3. Orchestrator → taco.action modules: dispatch intermediate actions (OCR, depth, math)
4. Action results → Orchestrator: integrate into next prompt
5. Orchestrator → Model Backend: if using LLaVA or Mantis local model server via llava/serve or huggingface caches
6. Orchestrator → VLMEvalKit: write predictions to disk → run_eval_on_preds reads results → metrics
7. Training Scripts → llava/train or mantis/train: launch distributed training on GPUs

c. Architectural Patterns & Principles
- Modular plugin architecture for API clients (autogen/oai) and model backends (vlmeval/vlm, llava, mantis)
- Orchestrator-centric design: centralized taco agent controlling multi-step pipelines
- Separation of concerns: inference vs. evaluation vs. training

d. Key Technologies
- Python-based CLI entrypoints and bash wrappers
- PyTorch / Accelerate / DeepSpeed for training
- Hugging Face Transformers for model loading inside llava and mantis
- External LLM APIs (OpenAI, Anthropic, Gemini, Qwen) via autogen.oai
- VLMEvalKit for standardized benchmark evaluation

##### Final Advice
- Subsystem as its own cluster to avoid clutter.
- Detail all external dependencies (LLM APIs, CUDA/Accelerate) in a footer or side panel.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer" 
        direction TB
        Scripts["Bash Scripts"]:::cli
        EntryPoints["Python Entrypoints"]:::cli
    end

    subgraph "Bash Scripts" 
        direction TB
        InferEval["infer_and_eval_taco.sh"]:::cli
        TrainLLA["train_taco_llava_onevision.sh"]:::cli
        TrainMAN["train_taco_mantis_llava.sh"]:::cli
    end

    subgraph "Python Entrypoints"
        direction TB
        RunAgent["run_multimodal_agent"]:::cli
        RunEval["run.py"]:::cli
        EvalPreds["run_eval_on_preds.py"]:::cli
    end

    %% Inference Orchestrator
    subgraph "Inference Orchestrator (taco)" 
        direction TB
        Agent["agent.py"]:::core
        Action["action.py"]:::core
        ActionUtils["action_utils.py"]:::api
        Parser["parser.py"]:::core
        Executor["executor.py"]:::core
        Prompt["prompt.py"]:::core
        Observation["observation.py"]:::core
        Data["data.py"]:::core
        Config["config.py"]:::core
    end

    %% API & Tool Adapters
    subgraph "API & Tool Adapters (autogen)" 
        direction TB
        OAI["oai/"]:::api
        AgentChat["agentchat/"]:::api
        Cache["cache/"]:::api
        Logger["logger/"]:::api
        Coding["coding/"]:::api
        CodeUtils["code_utils.py"]:::api
        BrowserIO["browser_utils.py"]:::api
        IO["io/"]:::api
    end

    %% Model Backends
    subgraph "Model Backends" 
        direction TB
        subgraph "LLaVA-OneVision" 
            direction TB
            LLModel["model/"]:::backend
            LLServe["serve/"]:::backend
            LLEval["eval/"]:::backend
        end
        subgraph "Mantis" 
            direction TB
            MANModels["models/"]:::backend
        end
        subgraph "VL Models (eval)" 
            direction TB
            VLEvalModel["vlm/"]:::backend
        end
    end

    %% Evaluation Subsystem
    subgraph "Evaluation (VLMEvalKit)" 
        direction TB
        VLData["dataset/"]:::eval
        VLAPI["api/"]:::eval
        VLTools["tools.py"]:::eval
        EvalEntry["run_eval_on_preds.py"]:::eval
    end

    %% Training Pipelines
    subgraph "Training Pipelines" 
        direction TB
        subgraph "LLaVA Training" 
            direction TB
            LLTrainScripts["scripts/train/"]:::train
            LLTrainCode["train.py"]:::train
            LLTrainer["llava_trainer.py"]:::train
            LLTRL["trl/"]:::train
        end
        subgraph "Mantis Training" 
            direction TB
            MANTrainEntry["train/"]:::train
            MANDataCfg["data_configs/"]:::train
            MANAccel["accelerate_configs/"]:::train
            MANZero["zero_configs/"]:::train
            MANScripts["scripts/"]:::train
        end
    end

    %% Shared Utilities & External Services
    subgraph "Shared & External" 
        direction TB
        App["app.py"]:::core
        ReqAll["requirements.txt"]:::core
        ReqLLA["requirements_llava.txt"]:::core
        ReqMAN["requirements_mantis.txt"]:::core
        LLMAPIs["LLM APIs"]:::ext
        ToolsSvc["External Tool Services"]:::ext
    end

    %% Connections
    InferEval -->|invokes| RunAgent
    RunAgent -->|uses| Agent
    Agent -->|dispatches| Action
    Action -->|uses| ActionUtils
    Agent -->|parses| Parser
    Agent -->|executes| Executor
    Agent -->|loads prompts| Prompt
    Agent -->|records| Observation
    Agent -->|handles data| Data
    Agent -->|reads config| Config

    Agent -->|calls| OAI
    Agent -->|calls| AgentChat
    Agent -->|caches| Cache
    Agent -->|logs| Logger
    Agent -->|executes code| Coding
    Agent -->|utils| CodeUtils
    Agent -->|browser IO| BrowserIO
    Agent -->|IO| IO

    Agent -->|executes| ActionUtils

    Agent -->|routes to| LLModel
    Agent -->|routes to| LLServe
    Agent -->|routes to| LLEval
    Agent -->|routes to| MANModels
    Agent -->|routes to| VLEvalModel

    RunEval -->|reads preds| EvalEntry
    EvalEntry -->|uses| VLData
    EvalEntry -->|uses| VLAPI
    EvalEntry -->|uses| VLTools

    TrainLLA -->|launches| LLTrainScripts
    LLTrainScripts -->|runs| LLTrainCode
    LLTrainCode -->|uses| LLTrainer
    LLTrainer -->|uses| LLTRL

    TrainMAN -->|launches| MANTrainEntry
    MANTrainEntry -->|loads data| MANDataCfg
    MANTrainEntry -->|configures| MANAccel
    MANTrainEntry -->|configures| MANZero
    MANTrainEntry -->|uses scripts| MANScripts

    Agent -->|optionally| EvalEntry
    Agent -->|optionally| LLTrainScripts
    Agent -->|optionally| MANTrainEntry

    App -->|serves| LLServe
    LLMAPIs -->|external| OAI
    ToolsSvc -->|external| ActionUtils

    %% Click Events
    click InferEval "https://github.com/salesforceairesearch/latte/blob/main/scripts/infer_and_eval_taco.sh"
    click TrainLLA "https://github.com/salesforceairesearch/latte/blob/main/scripts/train_taco_llava_onevision.sh"
    click TrainMAN "https://github.com/salesforceairesearch/latte/blob/main/scripts/train_taco_mantis_llava.sh"
    click RunAgent "https://github.com/salesforceairesearch/latte/blob/main/taco/run_multimodal_agent.py"
    click RunEval "https://github.com/salesforceairesearch/latte/blob/main/VLMEvalKit/run.py"
    click EvalPreds "https://github.com/salesforceairesearch/latte/blob/main/VLMEvalKit/run_eval_on_preds.py"
    click Agent "https://github.com/salesforceairesearch/latte/blob/main/taco/agent.py"
    click Action "https://github.com/salesforceairesearch/latte/blob/main/taco/action.py"
    click ActionUtils "https://github.com/salesforceairesearch/latte/blob/main/taco/action_utils.py"
    click Parser "https://github.com/salesforceairesearch/latte/blob/main/taco/parser.py"
    click Executor "https://github.com/salesforceairesearch/latte/blob/main/taco/executor.py"
    click Prompt "https://github.com/salesforceairesearch/latte/blob/main/taco/prompt.py"
    click Observation "https://github.com/salesforceairesearch/latte/blob/main/taco/observation.py"
    click Data "https://github.com/salesforceairesearch/latte/blob/main/taco/data.py"
    click Config "https://github.com/salesforceairesearch/latte/blob/main/taco/config.py"
    click OAI "https://github.com/salesforceairesearch/latte/tree/main/autogen/oai/"
    click AgentChat "https://github.com/salesforceairesearch/latte/tree/main/autogen/agentchat/"
    click Cache "https://github.com/salesforceairesearch/latte/tree/main/autogen/cache/"
    click Logger "https://github.com/salesforceairesearch/latte/tree/main/autogen/logger/"
    click Coding "https://github.com/salesforceairesearch/latte/tree/main/autogen/coding/"
    click CodeUtils "https://github.com/salesforceairesearch/latte/blob/main/autogen/code_utils.py"
    click BrowserIO "https://github.com/salesforceairesearch/latte/blob/main/autogen/browser_utils.py"
    click IO "https://github.com/salesforceairesearch/latte/tree/main/autogen/io/"
    click LLModel "https://github.com/salesforceairesearch/latte/tree/main/llava/model/"
    click LLServe "https://github.com/salesforceairesearch/latte/tree/main/llava/serve/"
    click LLEval "https://github.com/salesforceairesearch/latte/tree/main/llava/eval/"
    click MANModels "https://github.com/salesforceairesearch/latte/tree/main/mantis/models/"
    click VLEvalModel "https://github.com/salesforceairesearch/latte/tree/main/VLMEvalKit/vlmeval/vlm/"
    click VLData "https://github.com/salesforceairesearch/latte/tree/main/VLMEvalKit/vlmeval/dataset/"
    click VLAPI "https://github.com/salesforceairesearch/latte/tree/main/VLMEvalKit/vlmeval/api/"
    click VLTools "https://github.com/salesforceairesearch/latte/blob/main/VLMEvalKit/vlmeval/tools.py"
    click LLTrainScripts "https://github.com/salesforceairesearch/latte/tree/main/llava/scripts/train/"
    click LLTrainCode "https://github.com/salesforceairesearch/latte/blob/main/llava/train/train.py"
    click LLTrainer "https://github.com/salesforceairesearch/latte/blob/main/llava/train/llava_trainer.py"
    click LLTRL "https://github.com/salesforceairesearch/latte/tree/main/llava/train/trl/"
    click MANTrainEntry "https://github.com/salesforceairesearch/latte/tree/main/mantis/train/"
    click MANDataCfg "https://github.com/salesforceairesearch/latte/tree/main/mantis/train/data_configs/"
    click MANAccel "https://github.com/salesforceairesearch/latte/tree/main/mantis/train/accelerate_configs/"
    click MANZero "https://github.com/salesforceairesearch/latte/tree/main/mantis/train/zero_configs/"
    click MANScripts "https://github.com/salesforceairesearch/latte/tree/main/mantis/train/scripts/"
    click App "https://github.com/salesforceairesearch/latte/blob/main/app.py"
    click ReqAll "https://github.com/salesforceairesearch/latte/blob/main/requirements.txt"
    click ReqLLA "https://github.com/salesforceairesearch/latte/blob/main/requirements_llava.txt"
    click ReqMAN "https://github.com/salesforceairesearch/latte/blob/main/requirements_mantis.txt"

    %% Styles
    classDef cli fill:#CCCCCC,stroke:#333,stroke-width:1px
    classDef core fill:#AED6F1,stroke:#333,stroke-width:1px
    classDef api fill:#A9DFBF,stroke:#333,stroke-width:1px
    classDef backend fill:#F5B041,stroke:#333,stroke-width:1px
    classDef eval fill:#D7BDE2,stroke:#333,stroke-width:1px
    classDef train fill:#F8C471,stroke:#333,stroke-width:1px
    classDef ext fill:#ABB2B9,stroke:#333,stroke-width:1px
```
