# nvlabs/vila architecture

> Architecture diagram of the GitHub repository nvlabs/vila, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvlabs/vila
- Repository: https://github.com/nvlabs/vila

## Overview

##### Main System Components

a) Data Processing Pipeline
- Multiple dataset processors (MMC4, COYO, SFT datasets)
- Data preparation workflows (downloading, splitting, filtering)

b) Core Model Architecture
- Three main components:
- Multimodal Encoder (llava/model/multimodal_encoder)
- Language Model (llava/model/language_model)
- Multimodal Projector (llava/model/multimodal_projector)

c) Training System
- Training pipeline with:
- Training workflows (alignment, pretraining, fine-tuning)
- Sequence parallel training system
- Optimization components (FP8 training, quantization)

d) Inference & Deployment
- Include:
- Multiple deployment backends (TinyChat, TensorRT-LLM)
- API server implementation
- Evaluation systems

##### Component Relationships

Use these connection types:
- Double lines: API boundaries

##### Detailed Subcomponents

a) Data Processing:
**[Data Processing]**
- Dataset Processors
- MMC4 Pipeline
- COYO Pipeline
- SFT Datasets
- Data Preparation Utils

b) Model Architecture:
**[Model Architecture]**
- Multimodal Encoder
- Vision Encoder
- CLIP Encoder
- Other Encoders
- Language Model
- Base LLM
- Multimodal Adaptations
- Projector Layer

c) Training System:
**[Training Pipeline]**
- Alignment Stage
- Pretraining Stage
- Fine-tuning Stage
- Optimization
- FP8 Training
- Quantization
- Sequence Parallel

##### Implementation Details

Technical annotations for:
- Supported model sizes (3B/8B/13B/40B)
- Quantization methods (AWQ, 4-bit)
- Deployment targets (A100, 4090, Orin)
- Performance metrics

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Data_Processing
        MMC4[MMC4 Processing]
        COYO[COYO Processing]
        SFT[SFT Datasets]
        DataCore[Dataset Core]
        DataReg[Dataset Registry]
    end

    subgraph Model_Architecture
        subgraph Encoder
            MultiEnc[Multimodal Encoder]
            VisionEnc[Vision Encoder]
            CLIPEnc[CLIP Encoder]
        end
        
        subgraph LangModel
            BaseLLM[Base LLM]
            MultiAdapt[Multimodal Adaptations]
        end
        
        Projector[Multimodal Projector]
    end

    subgraph Training_System
        SeqParallel[Sequence Parallel Training]
        FP8[FP8 Training]
        TrainFlow[Training Workflows]
        
        subgraph Optimization
            Quant[Quantization Tools]
            RealQuant[Real Quantization]
        end
    end

    subgraph Deployment
        TRT[TensorRT-LLM Backend]
        API[API Server]
        Eval[Evaluation Systems]
    end

    %% Data Flow
    MMC4 --> DataCore
    COYO --> DataCore
    SFT --> DataCore
    DataCore --> DataReg
    DataReg --> TrainFlow

    MultiEnc --> Projector
    BaseLLM --> MultiAdapt
    MultiAdapt --> Projector

    TrainFlow --> Optimization
    SeqParallel --> TrainFlow
    FP8 --> TrainFlow

    Projector --> TRT
    Projector --> API
    API --> Eval

    %% Styling
    classDef data fill:#a8d1f0
    classDef model fill:#90EE90
    classDef training fill:#FFB347
    classDef deployment fill:#DDA0DD

    class MMC4,COYO,SFT,DataCore,DataReg data
    class MultiEnc,VisionEnc,CLIPEnc,BaseLLM,MultiAdapt,Projector model
    class SeqParallel,FP8,TrainFlow,Quant,RealQuant training
    class TRT,API,Eval deployment

    %% Click Events
    click MMC4 "https://github.com/nvlabs/vila/tree/main/data_prepare/mmc4/"
    click COYO "https://github.com/nvlabs/vila/tree/main/data_prepare/coyo/"
    click SFT "https://github.com/nvlabs/vila/tree/main/data_prepare/sft/"
    click MultiEnc "https://github.com/nvlabs/vila/tree/main/llava/model/multimodal_encoder/"
    click BaseLLM "https://github.com/nvlabs/vila/tree/main/llava/model/language_model/"
    click Projector "https://github.com/nvlabs/vila/tree/main/llava/model/multimodal_projector/"
    click SeqParallel "https://github.com/nvlabs/vila/tree/main/llava/train/sequence_parallel/"
    click FP8 "https://github.com/nvlabs/vila/tree/main/llava/model/coat/"
    click TrainFlow "https://github.com/nvlabs/vila/tree/main/scripts/NVILA/"
    click TRT "https://github.com/nvlabs/vila/tree/main/demo_trt_llm/"
    click API "https://github.com/nvlabs/vila/blob/main/server.py"
    click Eval "https://github.com/nvlabs/vila/tree/main/llava/eval/"
    click DataCore "https://github.com/nvlabs/vila/tree/main/llava/data/dataset_impl/"
    click DataReg "https://github.com/nvlabs/vila/tree/main/llava/data/registry/"
    click Quant "https://github.com/nvlabs/vila/tree/main/llava/model/quantization/"
    click RealQuant "https://github.com/nvlabs/vila/tree/main/llava/model/realquantize/"
```
