How salesforceairesearch/latte works
Project Type and Purpose
• TACO is an open-source research toolkit (not a traditional full-stack web app) for multi-modal “chains-of-thought-and-action” (CoTA) inference, evaluation, and training of large action models.
• Key features:
• Inference orchestration of multi-step tasks (OCR, depth estimation, calculator, etc.) via LLM APIs
• Built-in evaluation (VLMEvalKit) against benchmarks
• Training pipelines for two model families (Mantis and LLaVA-OneVision)
File-Structure Analysis
• Top-level directories, by responsibility:
• taco/: Core inference orchestrator (agent + action + parser + executor + prompt + data)
• VLMEvalKit/: Standalone evaluation framework (run.py, run_eval_on_preds.py, dataset definitions, API wrappers)
• autogen/: Shared internal library for LLM API clients, agent chat framework, caching, code execution, logging
• llava/: LLaVA-OneVision model definition, data preprocessing, training scripts, serving code
• mantis/: Mantis model definition, training configurations, training scripts
• scripts/: Convenient bash wrappers for inference+evaluation and training
• Entry points:
• Inference: python -m taco.run_multimodal_agent (or via scripts/infer_and_eval_taco.sh)
• Evaluation: python VLMEvalKit/run_eval_on_preds.py
• Training: scripts/train_taco_mantis_llava.sh (Mantis) and scripts/train_taco_llava_onevision.sh (LLaVA)
README Insights
• Workflow described: install → inference → evaluation → (optionally) training with separate envs.
• External tools invoked during inference (OCR, depth, calculator) are represented by “action” modules under taco/ and by API wrappers under autogen/oai and taco/api.
1. CLI Layer
• Bash scripts (scripts/)
• Python entrypoints (taco/run_multimodal_agent, VLMEvalKit/run_*.py)
2. Inference Orchestrator
• taco.core: agent, action, parser, executor, prompt, observation, data
• autogen.agentchat & autogen.oai: LLM API adapters (OpenAI, Anthropic, Gemini, Qwen, etc.)
• Tool modules under taco.action_utils (OCR, depth, math)
3. Model Backends
• LLaVA-OneVision (llava/model, llava/serve)
• Mantis (mantis/models)
• Other VL models via VLMEvalKit/vlmeval/vlm and its api/ adapters
4. Data & Evaluation
• VLMEvalKit dataset definitions & utils
• run_eval_on_preds for metric computation
5. Training Pipelines
• llava/train & mantis/train: trainers, data configs, accelerate/zero setups
6. External Services
• LLM APIs (OpenAI, Anthropic, Gemini, Qwen, etc.)
• External tool services (OCR, Depth Estimation, Calculator)
1. CLI → Inference Orchestrator: user invokes run_multimodal_agent
2. Orchestrator → autogen.oai: send prompt for chain-of-thought
3. Orchestrator → taco.action modules: dispatch intermediate actions (OCR, depth, math)
4. Action results → Orchestrator: integrate into next prompt
5. Orchestrator → Model Backend: if using LLaVA or Mantis local model server via llava/serve or huggingface caches
6. Orchestrator → VLMEvalKit: write predictions to disk → run_eval_on_preds reads results → metrics
7. Training Scripts → llava/train or mantis/train: launch distributed training on GPUs
c. Architectural Patterns & Principles
• Modular plugin architecture for API clients (autogen/oai) and model backends (vlmeval/vlm, llava, mantis)
• Orchestrator-centric design: centralized taco agent controlling multi-step pipelines
• Separation of concerns: inference vs. evaluation vs. training
d. Key Technologies
• Python-based CLI entrypoints and bash wrappers
• PyTorch / Accelerate / DeepSpeed for training
• Hugging Face Transformers for model loading inside llava and mantis
• External LLM APIs (OpenAI, Anthropic, Gemini, Qwen) via autogen.oai
• VLMEvalKit for standardized benchmark evaluation
Final Advice
• Subsystem as its own cluster to avoid clutter.
• Detail all external dependencies (LLM APIs, CUDA/Accelerate) in a footer or side panel.