# salesforce/lavis architecture

> Architecture diagram of the GitHub repository salesforce/lavis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salesforce/lavis
- Repository: https://github.com/salesforce/lavis

## Overview

LAVIS is a language-vision intelligence library for multimodal models, datasets, tasks, training, and evaluation; the tree also includes a Streamlit-style demo application. A user can choose a demo task, provide an input, and receive a model result. Internally, the library unifies model and processor loading with task and dataset components, and supports training and evaluation through runners. The README documents broader task and dataset support than the sampled implementation excerpts show; those capabilities are represented at subsystem level, with uncertain internal links omitted.

Main entry points: [`app/main.py`](https://github.com/salesforce/lavis/blob/main/app/main.py) (Demo navigation), [`lavis/models/__init__.py`](https://github.com/salesforce/lavis/blob/main/lavis/models/__init__.py) (Model loading), [`lavis/runners/runner_base.py`](https://github.com/salesforce/lavis/blob/main/lavis/runners/runner_base.py) (Training runners), [`lavis/common/registry.py`](https://github.com/salesforce/lavis/blob/main/lavis/common/registry.py) (Component registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_demo["Demo application"]
  node_demo_router["Demo navigation<br/>[main.py]"]
  node_caption_demo["Captioning demo<br/>[caption.py]"]
  node_vqa_demo["Visual QA demo<br/>[vqa.py]"]
  node_search_demo["Multimodal search"]
  node_itm_demo["Image-text matching"]
  node_localization_demo["Text localization"]
  node_classification_demo["Classification demo<br/>[classification.py]"]
end

subgraph group_runtime["Model runtime"]
  node_model_api["Model loading<br/>[__init__.py]"]
  node_model_zoo["Model implementations"]
  node_base_model["Model base<br/>[base_model.py]"]
  node_processors["Input processors"]
  node_registry["Component registry<br/>[registry.py]"]
end

subgraph group_data["Data and tasks"]
  node_datasets["Dataset builders"]
  node_data_utils["Dataset utilities<br/>[data_utils.py]"]
  node_task_runtime["Task execution<br/>[base_task.py]"]
  node_task_types["Task implementations"]
end

subgraph group_operations["Training and evaluation"]
  node_config["Configuration<br/>[config.py]"]
  node_runner["Training runners<br/>[runner_base.py]"]
  node_evaluation["Evaluation<br/>[evaluate.py]"]
  node_distributed["Distributed utilities<br/>[dist_utils.py]"]
end

node_user(("User"))

node_user -->|"chooses task"| node_demo_router
node_demo_router -->|"routes to"| node_caption_demo
node_demo_router -->|"routes to"| node_vqa_demo
node_demo_router -->|"routes to"| node_search_demo
node_demo_router -->|"routes to"| node_itm_demo
node_demo_router -->|"routes to"| node_localization_demo
node_demo_router -->|"routes to"| node_classification_demo
node_search_demo -->|"loads model"| node_model_api
node_search_demo -->|"uses processors"| node_processors
node_model_api -->|"looks up models"| node_registry
node_model_api -->|"loads implementation"| node_model_zoo
node_model_zoo -->|"extends"| node_base_model
node_model_api -->|"builds processors"| node_processors
node_datasets -->|"registers builders"| node_registry
node_datasets -->|"uses data utilities"| node_data_utils
node_task_runtime -->|"registers tasks"| node_registry
node_task_types -->|"extends task base"| node_task_runtime
node_task_runtime -->|"uses data utilities"| node_data_utils
node_user -.->|"selects task"| node_task_types
node_user -.->|"chooses dataset"| node_datasets
node_config -->|"resolves components"| node_registry
node_runner -->|"prepares datasets"| node_data_utils
node_runner -.->|"reads settings"| node_config
node_evaluation -.->|"evaluates tasks"| node_task_types
node_runner -.->|"uses coordination"| node_distributed
node_user -.->|"trains models"| node_runner
node_user -.->|"benchmarks models"| node_evaluation

click node_demo_router "https://github.com/salesforce/lavis/blob/main/app/main.py"
click node_caption_demo "https://github.com/salesforce/lavis/blob/main/app/caption.py"
click node_vqa_demo "https://github.com/salesforce/lavis/blob/main/app/vqa.py"
click node_search_demo "https://github.com/salesforce/lavis/blob/main/app/multimodal_search.py"
click node_itm_demo "https://github.com/salesforce/lavis/blob/main/app/image_text_match.py"
click node_localization_demo "https://github.com/salesforce/lavis/blob/main/app/text_localization.py"
click node_classification_demo "https://github.com/salesforce/lavis/blob/main/app/classification.py"
click node_model_api "https://github.com/salesforce/lavis/blob/main/lavis/models/__init__.py"
click node_model_zoo "https://github.com/salesforce/lavis/tree/main/lavis/models"
click node_base_model "https://github.com/salesforce/lavis/blob/main/lavis/models/base_model.py"
click node_processors "https://github.com/salesforce/lavis/tree/main/lavis/processors"
click node_registry "https://github.com/salesforce/lavis/blob/main/lavis/common/registry.py"
click node_datasets "https://github.com/salesforce/lavis/tree/main/lavis/datasets/builders"
click node_data_utils "https://github.com/salesforce/lavis/blob/main/lavis/datasets/data_utils.py"
click node_task_runtime "https://github.com/salesforce/lavis/blob/main/lavis/tasks/base_task.py"
click node_task_types "https://github.com/salesforce/lavis/tree/main/lavis/tasks"
click node_config "https://github.com/salesforce/lavis/blob/main/lavis/common/config.py"
click node_runner "https://github.com/salesforce/lavis/blob/main/lavis/runners/runner_base.py"
click node_evaluation "https://github.com/salesforce/lavis/blob/main/evaluate.py"
click node_distributed "https://github.com/salesforce/lavis/blob/main/lavis/common/dist_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo_router,node_caption_demo,node_vqa_demo,node_search_demo,node_itm_demo,node_localization_demo,node_classification_demo,node_user toneBlue
class node_model_api,node_model_zoo,node_base_model,node_processors,node_registry toneAmber
class node_datasets,node_data_utils,node_task_runtime,node_task_types toneMint
class node_config,node_runner,node_evaluation,node_distributed toneRose
```

## Components

### Demo application

- [Demo navigation](https://github.com/salesforce/lavis/blob/main/app/main.py): `app/main.py`
- [Captioning demo](https://github.com/salesforce/lavis/blob/main/app/caption.py): `app/caption.py`
- [Visual QA demo](https://github.com/salesforce/lavis/blob/main/app/vqa.py): `app/vqa.py`
- [Multimodal search](https://github.com/salesforce/lavis/blob/main/app/multimodal_search.py): `app/multimodal_search.py`
- [Image-text matching](https://github.com/salesforce/lavis/blob/main/app/image_text_match.py): `app/image_text_match.py`
- [Text localization](https://github.com/salesforce/lavis/blob/main/app/text_localization.py): `app/text_localization.py`
- [Classification demo](https://github.com/salesforce/lavis/blob/main/app/classification.py): `app/classification.py`

### Model runtime

- [Model loading](https://github.com/salesforce/lavis/blob/main/lavis/models/__init__.py): `lavis/models/__init__.py`
- [Model implementations](https://github.com/salesforce/lavis/tree/main/lavis/models): `lavis/models`
- [Model base](https://github.com/salesforce/lavis/blob/main/lavis/models/base_model.py): `lavis/models/base_model.py`
- [Input processors](https://github.com/salesforce/lavis/tree/main/lavis/processors): `lavis/processors`
- [Component registry](https://github.com/salesforce/lavis/blob/main/lavis/common/registry.py): `lavis/common/registry.py`

### Data and tasks

- [Dataset builders](https://github.com/salesforce/lavis/tree/main/lavis/datasets/builders): `lavis/datasets/builders`
- [Dataset utilities](https://github.com/salesforce/lavis/blob/main/lavis/datasets/data_utils.py): `lavis/datasets/data_utils.py`
- [Task execution](https://github.com/salesforce/lavis/blob/main/lavis/tasks/base_task.py): `lavis/tasks/base_task.py`
- [Task implementations](https://github.com/salesforce/lavis/tree/main/lavis/tasks): `lavis/tasks`

### Training and evaluation

- [Configuration](https://github.com/salesforce/lavis/blob/main/lavis/common/config.py): `lavis/common/config.py`
- [Training runners](https://github.com/salesforce/lavis/blob/main/lavis/runners/runner_base.py): `lavis/runners/runner_base.py`
- [Evaluation](https://github.com/salesforce/lavis/blob/main/evaluate.py): `evaluate.py`
- [Distributed utilities](https://github.com/salesforce/lavis/blob/main/lavis/common/dist_utils.py): `lavis/common/dist_utils.py`

### Other components

- **User**

## Connections

- User → Demo navigation: chooses task (evidence: `app/main.py`)
- Demo navigation → Captioning demo: routes to (evidence: `app/main.py`)
- Demo navigation → Visual QA demo: routes to (evidence: `app/main.py`)
- Demo navigation → Multimodal search: routes to (evidence: `app/main.py`)
- Demo navigation → Image-text matching: routes to (evidence: `app/main.py`)
- Demo navigation → Text localization: routes to (evidence: `app/main.py`)
- Demo navigation → Classification demo: routes to (evidence: `app/main.py`)
- Multimodal search → Model loading: loads model (evidence: `app/multimodal_search.py`)
- Multimodal search → Input processors: uses processors (evidence: `app/multimodal_search.py`)
- Model loading → Component registry: looks up models (evidence: `lavis/models/__init__.py`)
- Model loading → Model implementations: loads implementation (evidence: `lavis/models/__init__.py`)
- Model implementations → Model base: extends (evidence: `lavis/models/__init__.py`)
- Model loading → Input processors: builds processors (evidence: `lavis/models/__init__.py`)
- Dataset builders → Component registry: registers builders (evidence: `lavis/datasets/builders/retrieval_builder.py`)
- Dataset builders → Dataset utilities: uses data utilities (evidence: `lavis/datasets/data_utils.py`)
- Task execution → Component registry: registers tasks (evidence: `lavis/tasks/base_task.py`)
- Task implementations → Task execution: extends task base (evidence: `lavis/tasks/retrieval.py`)
- Task execution → Dataset utilities: uses data utilities (evidence: `lavis/tasks/base_task.py`)
- User → Task implementations: selects task
- User → Dataset builders: chooses dataset
- Configuration → Component registry: resolves components (evidence: `lavis/common/config.py`)
- Training runners → Dataset utilities: prepares datasets (evidence: `lavis/runners/runner_base.py`)
- Training runners → Configuration: reads settings
- Evaluation → Task implementations: evaluates tasks
- Training runners → Distributed utilities: uses coordination (evidence: `lavis/runners/runner_base.py`)
- User → Training runners: trains models
- User → Evaluation: benchmarks models
