How skyworkai/vitron works
Project type and purpose
• Vitron is a research‐oriented, full-stack machine-learning toolkit and demo application for a pixel-level vision LLM (images + videos).
• It bundles: a Gradio demo (app.py), data pipelines, pre-trained checkpoints, multi-stage training/inference scripts, and plug-in modules (GLIGEN, SEEM, StableVideo, i2vgen-xl).
File structure
• Top-level entrypoints:
• app.py / app_utils.py — launches Gradio demo (frontend).
• inference_image.py, scripts/finetune_lora.sh — CLI/inference and fine-tuning wrappers.
• Data & assets:
• data/, examples/ — raw video/image samples and orchestration JSON.
• checkpoints/ — downloaded pre-trained model weights.
• Core library: vitron/
• vitron/model — builder modules for language LLM, multimodal encoder, projector, region extractor.
• vitron/train — training and fine-tuning orchestrators (deep-learning loops, monkey-patches).
• Plug-in modules: modules/GLIGEN, /SEEM, /StableVideo, /i2vgen-xl
• Each subfolder contains its own configs, data, model layers, and inference scripts.
• Support files and scripts:
• requirements.txt, pyproject.toml, style.css, scripts/zero2_offload.json.
Key insights from README
• Purpose: unify tasks (understand, generate, segment, edit) for static images and dynamic videos.
• Architecture: modular – core “Vitron” layers hooking into external specialized sub-modules.
• Demo: Gradio UI → core inference pipeline → conditional use of sub-modules for different vision tasks.
• Training: configurable via YAML, fine-tune with lora adapters, uses deepspeed/flash-attn.
Main components
a. Frontend layer (Gradio Web UI)
• app.py → input capture (text prompt, upload image/video) → display output
b. Inference Orchestrator
• app_utils or inference_image.py → request dispatcher → selects appropriate pipeline (image / video / editing)
c. Core Vitron Engine
• Language Model Builder
• Multimodal Encoder (image/video tokenizers, feature extractors)
• Pixel-level Projector & Region Extractor
• Adapter layers (LORA, δ-delta modules)
d. Plug-in Modules (GLIGEN, SEEM, StableVideo, i2vgen-xl)
• Each offers specialized dataset loaders, model extensions, postprocessors
e. Data & Checkpoints
• Raw data (data/, examples/) → preprocessing → model input
• checkpoints/ → weight loader
f. Training Pipelines
• vitron/train scripts, scripts/finetune_lora.sh → uses environment YAML, deep-learning frameworks
g. External Dependencies
• PyTorch / CUDA / FFmpeg / Decord / Detectron2 / deepspeed
Relationships and data flow
• User → Frontend (Gradio) → Inference Orchestrator
• Orchestrator → Core Vitron Engine → calls sub-module adapters if needed → outputs prediction
• Engine → loads data from examples/data → loads weights from checkpoints
• Training flows: Training script → Data pipeline → Vitron Engine → loss & optimizer → save to checkpoints
Architectural patterns and principles
• Modular plugin architecture: external modules live under modules/*
• Pipeline pattern: clear separation between data ingestion, model construction, inference, and post-processing
• Single responsibility: each vitron/model sub-folder (language, encoder, projector, region) has a dedicated role
• Extensibility: add new sub-modules by dropping into modules/ and updating config
Essential elements to capture
• Frontend entrypoints (app.py, inference scripts)
• Core Vitron sub-packages and their roles (language_model, multimodal_encoder, projector, extractor)
• The plugin directory structure (modules/GLIGEN, modules/SEEM, etc.)
• Data and checkpoint storage
• Training vs. Inference distinction
[User] → [Gradio App] → [Inference Orchestrator] → [Core Vitron Engine] ↔ [Plug-in Modules]
↓
[Checkpoints]
↑
[Data Pipeline]
↖ Training Scripts ↗