# joanrod/star-vector architecture

> Architecture diagram of the GitHub repository joanrod/star-vector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joanrod/star-vector
- Repository: https://github.com/joanrod/star-vector

## Overview

##### Main Components of the System
- Model Core and Architecture:
- This includes the image encoder and language model portions in starvector/model, which comprise submodules such as:
○ Image Encoder (e.g., CLIP-based image encoder and dedicated encoder code in starvector/image_encoder and starvector/model/image_encoder)
○ LLM parts adapted from StarCoder (found in starvector/model/llm/ and starvector/model/models for StarVector variants)
- Adapter modules for bridging multimodal inputs (the starvector/adapter.py and starvector/model/adapters).
- Data Pipeline:
- Data preprocessing, augmentation, and dataset management in starvector/data.
- Utility functions for processing and rasterizing SVGs in starvector/data/util.py.
- Training/Finetuning Pipeline:
- Scripts and shell files under scripts/train for both StarVector-1B and StarVector-8B.
- Training orchestration via Accelerate, Deepspeed, and FSDP as reflected in the configs/accelerate and configs/models directories.
- Evaluation and Validation:
- Validation modules under starvector/validation and evaluation scripts in scripts/validation.
- Different backend options (HuggingFace and vLLM) are supported for generating SVG code.
- Serving and Inference:
- The service logic is organized under starvector/serve. This includes:
○ A central controller (starvector/serve/controller.py)
○ Model workers (starvector/serve/model_worker.py) which actually load and run the model.
○ Gradio-based frontend demos (starvector/serve/gradio_web_server.py for both HF and vLLM backends).
○ VLLM-specific serving logic under starvector/serve/vllm_api_gradio.
- Docker & Deployment:
- Docker configurations and notebooks in the docker/ directory indicate that containerized setups for demos or production serving are supported.
- Configuration Management:
- Multiple YAML configuration files under configs for training (different GPU setups), generation parameters, and models.
- Jinja templates for chat interactions (configs/chat-template.jinja).

##### Relationships and Interactions Between Components
- Data flows from raw image/text input through the preprocessor (starvector/data) into the model core.
- During inference, the controller (in the serve layer) accepts requests (possibly from a UI or an API gateway), directs them to model workers and passes them onto the processing pipeline.
- The training pipeline, driven by scripts and configurations, feeds data into the model training process (combining loss calculations with evaluation metrics from starvector/metrics).
- The evaluation component (in starvector/validation) integrates with external benchmarks (as described in the README under SVG-Bench) to compute a variety of metrics.

##### Specify Architectural Patterns and Design Principles
- Modular design: Separate modules for data, model logic, training, validation, and serving.
- Separation of concerns: The project clearly distinguishes between training (scripts and configs), model architecture (in starvector/model and starvector/data), and inference serving (starvector/serve).
- Config-driven deployment: Use of YAML configurations (in configs/) to manage various training and inference setups.
- Multi-backend support: Incorporates both traditional HuggingFace and more performant vLLM backends.

##### Relevant Technologies, Frameworks, and Libraries
- PyTorch as the deep learning framework (implied by use of torchrun and FSDP).
- HuggingFace libraries for model loading and generation.
- Accelerate, Deepspeed, and FSDP for distributed training.
- Gradio for the web-based demo interface.
- Docker for containerized deployment.
- vLLM for high-performance serving.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Tier
    subgraph "Data & Preprocessing Tier"
        A["Raw Inputs (Images & Text)"]:::external
        B["Data Ingestion & Preprocessing"]:::preprocessing
    end
    A -->|"feeds"| B

    %% Training Pipeline & Config Management
    subgraph "Training Pipeline & Config Management"
        C["Training Scripts & Orchestration"]:::training
        D["Configuration Management (YAML, Jinja)"]:::config
    end
    B -->|"preparesData"| C
    D -->|"configures"| C

    %% Model Core Tier
    subgraph "Model Core"
        H["StarVector Model Core"]:::model
        E["Image Encoder (CLIP-based)"]:::model
        F["Language Model (StarCoder-based)"]:::model
        G["Adapters for Multimodal Inputs"]:::model
    end
    C -->|"trains"| H
    H -->|"integrates"| E
    H -->|"integrates"| F
    H -->|"integrates"| G

    %% Evaluation & Validation Tier
    subgraph "Evaluation & Validation"
        I["Evaluation & Validation Modules"]:::eval
        J["Metrics Computation"]:::metrics
    end
    H -->|"outputsModel"| I
    I -->|"computes"| J

    %% Serving & Inference Tier
    subgraph "Serving & Inference"
        K["Inference Controller"]:::serve
        L["Model Worker"]:::serve
        M["Gradio UI (HF Backend)"]:::serve
        N["vLLM Serving (API & Worker)"]:::serve
    end
    I -->|"feedsInto"| K
    K -->|"dispatchesTo"| L
    K -->|"respondsTo"| M
    K -->|"alternativePath"| N
    L -->|"returnsTo"| K

    %% Docker & Deployment
    O["Docker & Container Deployment"]:::deploy
    N -->|"containerizedBy"| O
    M -->|"containerizedBy"| O

    %% Click Events
    click H "https://github.com/joanrod/star-vector/tree/main/starvector/model"
    click E "https://github.com/joanrod/star-vector/tree/main/starvector/model/image_encoder"
    click E "https://github.com/joanrod/star-vector/blob/main/starvector/image_encoder.py"
    click F "https://github.com/joanrod/star-vector/tree/main/starvector/model/llm"
    click F "https://github.com/joanrod/star-vector/tree/main/starvector/model/models"
    click G "https://github.com/joanrod/star-vector/blob/main/starvector/adapter.py"
    click G "https://github.com/joanrod/star-vector/tree/main/starvector/model/adapters"
    click B "https://github.com/joanrod/star-vector/tree/main/starvector/data"
    click B "https://github.com/joanrod/star-vector/blob/main/starvector/data/util.py"
    click C "https://github.com/joanrod/star-vector/tree/main/scripts/train"
    click C "https://github.com/joanrod/star-vector/tree/main/starvector/train"
    click D "https://github.com/joanrod/star-vector/tree/main/configs"
    click D "https://github.com/joanrod/star-vector/blob/main/configs/chat-template.jinja"
    click I "https://github.com/joanrod/star-vector/tree/main/starvector/validation"
    click I "https://github.com/joanrod/star-vector/tree/main/scripts/validation"
    click J "https://github.com/joanrod/star-vector/tree/main/starvector/metrics"
    click K "https://github.com/joanrod/star-vector/blob/main/starvector/serve/controller.py"
    click L "https://github.com/joanrod/star-vector/blob/main/starvector/serve/model_worker.py"
    click M "https://github.com/joanrod/star-vector/blob/main/starvector/serve/gradio_web_server.py"
    click M "https://github.com/joanrod/star-vector/blob/main/starvector/serve/gradio_demo_with_updated_gradio.py"
    click N "https://github.com/joanrod/star-vector/tree/main/starvector/serve/vllm_api_gradio"
    click O "https://github.com/joanrod/star-vector/tree/main/docker"

    %% Styles
    classDef external fill:#f9c,stroke:#333,stroke-width:2px;
    classDef preprocessing fill:#bbf,stroke:#333,stroke-width:2px;
    classDef training fill:#bfb,stroke:#333,stroke-width:2px;
    classDef config fill:#ffb,stroke:#333,stroke-width:2px;
    classDef model fill:#fbf,stroke:#333,stroke-width:2px;
    classDef eval fill:#dfd,stroke:#333,stroke-width:2px;
    classDef metrics fill:#fdd,stroke:#333,stroke-width:2px;
    classDef serve fill:#cdf,stroke:#333,stroke-width:2px;
    classDef deploy fill:#ffc,stroke:#333,stroke-width:2px;
```
