# skyworkai/vitron architecture

> Architecture diagram of the GitHub repository skyworkai/vitron, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skyworkai/vitron
- Repository: https://github.com/skyworkai/vitron

## Overview

##### Project type and purpose
- Vitron is a research‐oriented, full-stack machine-learning toolkit and demo application for a pixel-level vision LLM (images + videos).
- It bundles: a Gradio demo (app.py), data pipelines, pre-trained checkpoints, multi-stage training/inference scripts, and plug-in modules (GLIGEN, SEEM, StableVideo, i2vgen-xl).

##### File structure
- Top-level entrypoints:
- app.py / app_utils.py — launches Gradio demo (frontend).
- inference_image.py, scripts/finetune_lora.sh — CLI/inference and fine-tuning wrappers.
- Data & assets:
- data/, examples/ — raw video/image samples and orchestration JSON.
- checkpoints/ — downloaded pre-trained model weights.
- Core library: vitron/
- vitron/model — builder modules for language LLM, multimodal encoder, projector, region extractor.
- vitron/train — training and fine-tuning orchestrators (deep-learning loops, monkey-patches).
- Plug-in modules: modules/GLIGEN, /SEEM, /StableVideo, /i2vgen-xl
- Each subfolder contains its own configs, data, model layers, and inference scripts.
- Support files and scripts:
- requirements.txt, pyproject.toml, style.css, scripts/zero2_offload.json.

##### Key insights from README
- Purpose: unify tasks (understand, generate, segment, edit) for static images and dynamic videos.
- Architecture: modular – core “Vitron” layers hooking into external specialized sub-modules.
- Demo: Gradio UI → core inference pipeline → conditional use of sub-modules for different vision tasks.
- Training: configurable via YAML, fine-tune with lora adapters, uses deepspeed/flash-attn.

##### Main components
a. Frontend layer (Gradio Web UI)
- app.py → input capture (text prompt, upload image/video) → display output
b. Inference Orchestrator
- app_utils or inference_image.py → request dispatcher → selects appropriate pipeline (image / video / editing)
c. Core Vitron Engine
- Language Model Builder
- Multimodal Encoder (image/video tokenizers, feature extractors)
- Pixel-level Projector & Region Extractor
- Adapter layers (LORA, δ-delta modules)
d. Plug-in Modules (GLIGEN, SEEM, StableVideo, i2vgen-xl)
- Each offers specialized dataset loaders, model extensions, postprocessors
e. Data & Checkpoints
- Raw data (data/, examples/) → preprocessing → model input
- checkpoints/ → weight loader
f. Training Pipelines
- vitron/train scripts, scripts/finetune_lora.sh → uses environment YAML, deep-learning frameworks
g. External Dependencies
- PyTorch / CUDA / FFmpeg / Decord / Detectron2 / deepspeed

##### Relationships and data flow
- User → Frontend (Gradio) → Inference Orchestrator
- Orchestrator → Core Vitron Engine → calls sub-module adapters if needed → outputs prediction
- Engine → loads data from examples/data → loads weights from checkpoints
- Training flows: Training script → Data pipeline → Vitron Engine → loss & optimizer → save to checkpoints

##### Architectural patterns and principles
- Modular plugin architecture: external modules live under modules/*
- Pipeline pattern: clear separation between data ingestion, model construction, inference, and post-processing
- Single responsibility: each vitron/model sub-folder (language, encoder, projector, region) has a dedicated role
- Extensibility: add new sub-modules by dropping into modules/ and updating config

##### Essential elements to capture
- Frontend entrypoints (app.py, inference scripts)
- Core Vitron sub-packages and their roles (language_model, multimodal_encoder, projector, extractor)
- The plugin directory structure (modules/GLIGEN, modules/SEEM, etc.)
- Data and checkpoint storage
- Training vs. Inference distinction

[User] → [Gradio App] → [Inference Orchestrator] → [Core Vitron Engine] ↔ [Plug-in Modules]
↓
[Checkpoints]
↑
[Data Pipeline]
↖ Training Scripts ↗

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User
    User(["User"]):::external

    %% Frontend Layer
    subgraph "Frontend (Gradio UI)"
        direction TB
        AppPy["app.py"]:::frontend
        AppUtilsFE["app_utils.py"]:::frontend
    end

    %% Inference Orchestrator
    subgraph "Inference Orchestrator"
        direction TB
        InferenceImage["inference_image.py"]:::orchestrator
        AppUtilsIO["app_utils.py"]:::orchestrator
    end

    %% Core Vitron Engine
    subgraph "Core Vitron Engine"
        direction TB
        subgraph "Language Model Builder"
            direction TB
            BuilderLM["builder.py"]:::core
            LLama["llava_llama.py"]:::core
            LMPT["llava_mpt.py"]:::core
            MPTDir["mpt/**"]:::core
        end
        subgraph "Multimodal Encoder"
            direction TB
            BuilderEnc["builder.py"]:::core
            CLIPEnc["clip_encoder.py"]:::core
            LangBind["languagebind/"]:::core
        end
        PixelProj["builder.py"]:::core
        RegionExt["builder.py"]:::core
        RegionLayer["layer.py"]:::core
        subgraph "Adapter Layers"
            direction TB
            ApplyDelta["apply_delta.py"]:::core
            MakeDelta["make_delta.py"]:::core
            Consolidate["consolidate.py"]:::core
        end
    end

    %% Plug-in Modules
    subgraph "Plug-in Modules"
        direction TB
        GLIGEN["GLIGEN"]:::plugin
        SEEM["SEEM"]:::plugin
        StableVideo["StableVideo"]:::plugin
        I2VGen["i2vgen-xl"]:::plugin
    end

    %% Data & Checkpoints
    subgraph "Data & Checkpoints"
        direction TB
        DataRaw["data/"]:::data
        Examples["examples/"]:::data
        Checkpoints["checkpoints/"]:::data
    end

    %% Training Pipelines
    subgraph "Training Pipelines"
        direction TB
        TrainPy["train.py"]:::training
        TrainMem["train_mem.py"]:::training
        TrainXf["train_xformers.py"]:::training
        LLATrainer["llava_trainer.py"]:::training
        FlashMPatch["llama_flash_attn_monkey_patch.py"]:::training
        XformersMPatch["llama_xformers_attn_monkey_patch.py"]:::training
        FinetuneSh["scripts/finetune_lora.sh"]:::training
    end

    %% External Dependencies & Config
    subgraph "External Dependencies & Config"
        direction TB
        Req["requirements.txt"]:::external
        PyProj["pyproject.toml"]:::external
        Zero2["zero2_offload.json"]:::external
        Style["style.css"]:::external
    end

    %% Connections
    User -->|uses| AppPy
    User -->|uses| AppUtilsFE
    AppPy -->|calls| InferenceImage
    AppPy -->|calls| AppUtilsIO
    InferenceImage -->|dispatches| BuilderLM
    InferenceImage -->|dispatches| BuilderEnc
    InferenceImage -->|dispatches| PixelProj
    InferenceImage -->|dispatches| RegionExt
    BuilderLM -->|may load| Checkpoints
    BuilderEnc -->|may load| Checkpoints
    PixelProj -->|uses| DataRaw
    RegionExt -->|uses| DataRaw
    BuilderLM ---|adapts via| ApplyDelta
    BuilderLM ---|adapts via| MakeDelta
    BuilderLM ---|adapts via| Consolidate
    BuilderLM <--> GLIGEN
    BuilderEnc <--> SEEM
    PixelProj <--> StableVideo
    RegionExt <--> I2VGen
    TrainPy -->|ingests| DataRaw
    TrainPy -->|trains| BuilderLM
    TrainPy -->|trains| BuilderEnc
    TrainPy -->|saves| Checkpoints
    FinetuneSh -->|invokes| TrainPy
    Req ---|provides deps| BuilderLM
    PyProj ---|project cfg| TrainPy

    %% Click Events
    click AppPy "https://github.com/skyworkai/vitron/blob/main/app.py"
    click AppUtilsFE "https://github.com/skyworkai/vitron/blob/main/app_utils.py"
    click InferenceImage "https://github.com/skyworkai/vitron/blob/main/inference_image.py"
    click AppUtilsIO "https://github.com/skyworkai/vitron/blob/main/app_utils.py"
    click BuilderLM "https://github.com/skyworkai/vitron/blob/main/vitron/model/builder.py"
    click LLama "https://github.com/skyworkai/vitron/blob/main/vitron/model/language_model/llava_llama.py"
    click LMPT "https://github.com/skyworkai/vitron/blob/main/vitron/model/language_model/llava_mpt.py"
    click MPTDir "https://github.com/skyworkai/vitron/tree/main/vitron/model/language_model/mpt/"
    click BuilderEnc "https://github.com/skyworkai/vitron/blob/main/vitron/model/multimodal_encoder/builder.py"
    click CLIPEnc "https://github.com/skyworkai/vitron/blob/main/vitron/model/multimodal_encoder/clip_encoder.py"
    click LangBind "https://github.com/skyworkai/vitron/tree/main/vitron/model/multimodal_encoder/languagebind/"
    click PixelProj "https://github.com/skyworkai/vitron/blob/main/vitron/model/multimodal_projector/builder.py"
    click RegionExt "https://github.com/skyworkai/vitron/blob/main/vitron/model/region_extractor/builder.py"
    click RegionLayer "https://github.com/skyworkai/vitron/blob/main/vitron/model/region_extractor/layer.py"
    click ApplyDelta "https://github.com/skyworkai/vitron/blob/main/vitron/model/apply_delta.py"
    click MakeDelta "https://github.com/skyworkai/vitron/blob/main/vitron/model/make_delta.py"
    click Consolidate "https://github.com/skyworkai/vitron/blob/main/vitron/model/consolidate.py"
    click GLIGEN "https://github.com/skyworkai/vitron/tree/main/modules/GLIGEN/"
    click SEEM "https://github.com/skyworkai/vitron/tree/main/modules/SEEM/"
    click StableVideo "https://github.com/skyworkai/vitron/tree/main/modules/StableVideo/"
    click I2VGen "https://github.com/skyworkai/vitron/tree/main/modules/i2vgen-xl/"
    click DataRaw "https://github.com/skyworkai/vitron/tree/main/data/"
    click Examples "https://github.com/skyworkai/vitron/tree/main/examples/"
    click Checkpoints "https://github.com/skyworkai/vitron/tree/main/checkpoints/"
    click TrainPy "https://github.com/skyworkai/vitron/blob/main/vitron/train/train.py"
    click TrainMem "https://github.com/skyworkai/vitron/blob/main/vitron/train/train_mem.py"
    click TrainXf "https://github.com/skyworkai/vitron/blob/main/vitron/train/train_xformers.py"
    click LLATrainer "https://github.com/skyworkai/vitron/blob/main/vitron/train/llava_trainer.py"
    click FlashMPatch "https://github.com/skyworkai/vitron/blob/main/vitron/train/llama_flash_attn_monkey_patch.py"
    click XformersMPatch "https://github.com/skyworkai/vitron/blob/main/vitron/train/llama_xformers_attn_monkey_patch.py"
    click FinetuneSh "https://github.com/skyworkai/vitron/blob/main/scripts/finetune_lora.sh"
    click Req "https://github.com/skyworkai/vitron/blob/main/requirements.txt"
    click PyProj "https://github.com/skyworkai/vitron/blob/main/pyproject.toml"
    click Zero2 "https://github.com/skyworkai/vitron/blob/main/scripts/zero2_offload.json"
    click Style "https://github.com/skyworkai/vitron/blob/main/style.css"

    %% Styles
    classDef frontend fill:#D0E6FF,stroke:#0366D6,stroke-width:1px
    classDef orchestrator fill:#E6F7FF,stroke:#1C7ED6,stroke-width:1px
    classDef core fill:#BEE3F8,stroke:#1C3D5A,stroke-width:1px
    classDef plugin fill:#D3F9D8,stroke:#2B8A3E,stroke-width:1px
    classDef data fill:#E9ECEF,stroke:#495057,stroke-width:1px,stroke-dasharray: 2 2
    classDef training fill:#FFF3BF,stroke:#F08C00,stroke-width:1px,stroke-dasharray: 5 2
    classDef external fill:#FFE3E3,stroke:#C92A2A,stroke-width:1px,stroke-dasharray: 5 2
```
