How salesforce/lavis works
LAVIS is a language-vision intelligence library for multimodal models, datasets, tasks, training, and evaluation; the tree also includes a Streamlit-style demo application. A user can choose a demo task, provide an input, and receive a model result. Internally, the library unifies model and processor loading with task and dataset components, and supports training and evaluation through runners. The README documents broader task and dataset support than the sampled implementation excerpts show; those capabilities are represented at subsystem level, with uncertain internal links omitted.
Main entry points: main.py Demo navigation · __init__.py Model loading · runner_base.py Training runners · registry.py Component registry
Components 21
Demo application
- Demo navigation
app/main.py - Captioning demo
app/caption.py - Visual QA demo
app/vqa.py - Multimodal search
app/multimodal_search.py - Image-text matching
app/image_text_match.py - Text localization
app/text_localization.py - Classification demo
app/classification.py
Model runtime
- Model loading
lavis/models/__init__.py - Model implementations
lavis/models - Model base
lavis/models/base_model.py - Input processors
lavis/processors - Component registry
lavis/common/registry.py
Data and tasks
- Dataset builders
lavis/datasets/builders - Dataset utilities
lavis/datasets/data_utils.py - Task execution
lavis/tasks/base_task.py - Task implementations
lavis/tasks
Training and evaluation
- Configuration
lavis/common/config.py - Training runners
lavis/runners/runner_base.py - Evaluation
evaluate.py - Distributed utilities
lavis/common/dist_utils.py
Other components
- User
Connections 27
- User → Demo navigation: chooses task
app/main.py - Demo navigation → Captioning demo: routes to
app/main.py - Demo navigation → Visual QA demo: routes to
app/main.py - Demo navigation → Multimodal search: routes to
app/main.py - Demo navigation → Image-text matching: routes to
app/main.py - Demo navigation → Text localization: routes to
app/main.py - Demo navigation → Classification demo: routes to
app/main.py - Multimodal search → Model loading: loads model
app/multimodal_search.py - Multimodal search → Input processors: uses processors
app/multimodal_search.py - Model loading → Component registry: looks up models
lavis/models/__init__.py - Model loading → Model implementations: loads implementation
lavis/models/__init__.py - Model implementations → Model base: extends
lavis/models/__init__.py - Model loading → Input processors: builds processors
lavis/models/__init__.py - Dataset builders → Component registry: registers builders
lavis/datasets/builders/retrieval_builder.py - Dataset builders → Dataset utilities: uses data utilities
lavis/datasets/data_utils.py - Task execution → Component registry: registers tasks
lavis/tasks/base_task.py - Task implementations → Task execution: extends task base
lavis/tasks/retrieval.py - Task execution → Dataset utilities: uses data utilities
lavis/tasks/base_task.py - User → Task implementations: selects task
- User → Dataset builders: chooses dataset
- Configuration → Component registry: resolves components
lavis/common/config.py - Training runners → Dataset utilities: prepares datasets
lavis/runners/runner_base.py - Training runners → Configuration: reads settings
- Evaluation → Task implementations: evaluates tasks
- Training runners → Distributed utilities: uses coordination
lavis/runners/runner_base.py - User → Training runners: trains models
- User → Evaluation: benchmarks models