How joanrod/star-vector works
Main Components of the System
• Model Core and Architecture:
• This includes the image encoder and language model portions in starvector/model, which comprise submodules such as:
○ Image Encoder (e.g., CLIP-based image encoder and dedicated encoder code in starvector/image_encoder and starvector/model/image_encoder)
○ LLM parts adapted from StarCoder (found in starvector/model/llm/ and starvector/model/models for StarVector variants)
• Adapter modules for bridging multimodal inputs (the starvector/adapter.py and starvector/model/adapters).
• Data Pipeline:
• Data preprocessing, augmentation, and dataset management in starvector/data.
• Utility functions for processing and rasterizing SVGs in starvector/data/util.py.
• Training/Finetuning Pipeline:
• Scripts and shell files under scripts/train for both StarVector-1B and StarVector-8B.
• Training orchestration via Accelerate, Deepspeed, and FSDP as reflected in the configs/accelerate and configs/models directories.
• Evaluation and Validation:
• Validation modules under starvector/validation and evaluation scripts in scripts/validation.
• Different backend options (HuggingFace and vLLM) are supported for generating SVG code.
• Serving and Inference:
• The service logic is organized under starvector/serve. This includes:
○ A central controller (starvector/serve/controller.py)
○ Model workers (starvector/serve/model_worker.py) which actually load and run the model.
○ Gradio-based frontend demos (starvector/serve/gradio_web_server.py for both HF and vLLM backends).
○ VLLM-specific serving logic under starvector/serve/vllm_api_gradio.
• Docker & Deployment:
• Docker configurations and notebooks in the docker/ directory indicate that containerized setups for demos or production serving are supported.
• Configuration Management:
• Multiple YAML configuration files under configs for training (different GPU setups), generation parameters, and models.
• Jinja templates for chat interactions (configs/chat-template.jinja).
Relationships and Interactions Between Components
• Data flows from raw image/text input through the preprocessor (starvector/data) into the model core.
• During inference, the controller (in the serve layer) accepts requests (possibly from a UI or an API gateway), directs them to model workers and passes them onto the processing pipeline.
• The training pipeline, driven by scripts and configurations, feeds data into the model training process (combining loss calculations with evaluation metrics from starvector/metrics).
• The evaluation component (in starvector/validation) integrates with external benchmarks (as described in the README under SVG-Bench) to compute a variety of metrics.
Specify Architectural Patterns and Design Principles
• Modular design: Separate modules for data, model logic, training, validation, and serving.
• Separation of concerns: The project clearly distinguishes between training (scripts and configs), model architecture (in starvector/model and starvector/data), and inference serving (starvector/serve).
• Config-driven deployment: Use of YAML configurations (in configs/) to manage various training and inference setups.
• Multi-backend support: Incorporates both traditional HuggingFace and more performant vLLM backends.
Relevant Technologies, Frameworks, and Libraries
• PyTorch as the deep learning framework (implied by use of torchrun and FSDP).
• HuggingFace libraries for model loading and generation.
• Accelerate, Deepspeed, and FSDP for distributed training.
• Gradio for the web-based demo interface.
• Docker for containerized deployment.
• vLLM for high-performance serving.