# leo1oel/small-vlm architecture

> Architecture diagram of the GitHub repository leo1oel/small-vlm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leo1oel/small-vlm
- Repository: https://github.com/leo1oel/small-vlm

## Overview

##### Project Type and Purpose
- This is an experimental research project centered around a “small” vision-language model (VLM). It appears to be a machine learning experiment integrating both vision and language processing, where the architecture is modular and configurable.
- The README suggests that it’s set up using the uv package manager and is built on the base of a “simple-modern-uv” template, which indicates a modern, streamlined Python development and deployment environment.

##### File Structure Analysis and Its Implications
- The main code lies within the “src/vlm” directory. This directory is subdivided into several sub-systems:
- The “config” directory contains YAML configuration files for various aspects including dataset, inference, model, and training. Notice different modes (inference and train) and various components (connector, llm, visual_encoder). This highlights that the project uses external configuration – likely through Hydra or a similar tool – to control behaviors.
- The “data” directory includes data loading utilities and datasets handling.
- The “inference” directory holds the inference engine, suggesting a dedicated pipeline for running predictions.
- The “models” directory captures the core architectural components including connectors (e.g., linear, mlp), language models, and visual encoders, indicating that these components are engineered separately and can be interchanged via configuration.
- The “train” directory contains training routines like loss computation, optimizer configuration, and the overall trainer logic.
- The “utils” folder includes utility scripts (e.g., logging and conversation template utilities) that support the system.
- Testing is supported via the “tests” directory.
- Additional files and directories (e.g., GitHub workflows, debug scripts, development/publishing instructions) indicate that the project is also set up for CI/CD and experimentation/debugging workflows.
- Overall, the structure suggests a clear separation of concerns by modularizing configuration, data management, model components, training, and inference.

##### Key Components and Their Relationships
- Configuration Module: Manages model parameters, training hyperparameters, and inference settings using YAML files. Acts as the glue that ties together the various subsystems.
- Data Module: Responsible for data loading and dataset management. This component feeds data into the training and inference pipelines.
- Models Module: Contains the neural network models segmented into different sub-components:
- Connectors: Mechanisms (e.g., linear & MLP connectors) that likely integrate the outputs and inputs between the visual and language modules.
- Language Models: Abstractions for processing language data.
- Visual Encoders: Modules such as ViT and CLIP-based architectures that process visual data.
- Training Module: Orchestrates training routines including loss calculations, optimizer routines, and learning rate scheduling.
- Inference Module: Handles prediction and evaluation using the trained models.
- Infrastructure/Utility Components: Auxiliary modules (logging, testing, development scripts) ensuring smooth operations (e.g., CI/CD, linting, and debugging).

- A “Configuration Layer” at the top:
▸ Connect YAML configs (dataset, mode, trainer, etc.) to all downstream modules.
- “Data Management” Block:
▸ Illustrate the flow from raw data to preprocessed data feeding datasets.
- “Model Components” Block split into:
▸ Visual Encoder: Side labeled with pipelines (e.g., CLIP, ViT).
▸ Language Model: Illustrated as a separate processing unit.
▸ Connectors: Positioned between visual and language modules, demonstrating the integration.
- “Training Engine” Block:
▸ Contains modules for loss computation, optimizers, schedulers, and trainer orchestration.
- “Inference Engine” Block:
▸ Representing the prediction mechanism which uses the same underlying models.
- Configuration driving the behavior of data ingestion, model instantiation, training, and inference.
- Data flow from the data module to both training and inference.
- Technical stack elements (Python, uv package manager) as peripheral components that support deployment and development workflows.
- Incorporate testing and debugging as ancillary components connecting with the main pipeline (CI/CD workflows).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    CONFIG["Configuration Module"]:::config

    %% Data Management Layer
    DATA["Data Module"]:::data

    %% Model Components Layer
    subgraph "Model Components"
        CONNECT["Connectors"]:::model
        LANG["Language Models"]:::model
        VISUAL["Visual Encoders"]:::model
    end

    %% Processing Layer
    subgraph "Processing"
        TRAIN["Training Engine"]:::train
        INF["Inference Pipeline"]:::train
    end

    %% Supporting Infrastructure
    subgraph "Supporting Infrastructure"
        UTIL["Utilities & Utils"]:::utils
        TEST["Testing Infrastructure"]:::testing
        CI["CI/CD Scripts"]:::cicd
        DEV["Dev Tools"]:::cicd
    end

    %% Main Flow: Configuration drives modules
    CONFIG -->|"drives"| DATA
    CONFIG -->|"drives"| CONNECT
    CONFIG -->|"drives"| LANG
    CONFIG -->|"drives"| VISUAL
    CONFIG -->|"drives"| TRAIN
    CONFIG -->|"drives"| INF

    %% Data flows into Processing
    DATA -->|"feeds"| TRAIN
    DATA -->|"feeds"| INF

    %% Model integration into Processing
    CONNECT -->|"used_by"| TRAIN
    LANG -->|"used_by"| TRAIN
    VISUAL -->|"used_by"| TRAIN

    CONNECT -->|"used_by"| INF
    LANG -->|"used_by"| INF
    VISUAL -->|"used_by"| INF

    %% Integration between model components
    VISUAL -->|"integrates"| CONNECT
    LANG -->|"integrates"| CONNECT

    %% Processing ancillary connections to Supporting Infrastructure
    TRAIN -.-> UTIL
    TRAIN -.-> TEST
    TRAIN -.-> CI
    TRAIN -.-> DEV

    INF -.-> UTIL
    INF -.-> TEST
    INF -.-> CI
    INF -.-> DEV

    %% Click Events
    click CONFIG "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/config"
    click DATA "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/data"
    click CONNECT "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/models/connectors"
    click LANG "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/models/language_models"
    click VISUAL "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/models/visual_encoders"
    click TRAIN "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/train"
    click INF "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/inference"
    click UTIL "https://github.com/leo1oel/small-vlm/tree/main/src/vlm/utils"
    click TEST "https://github.com/leo1oel/small-vlm/tree/main/tests"
    click CI "https://github.com/leo1oel/small-vlm/tree/main/.github/workflows"
    click DEV "https://github.com/leo1oel/small-vlm/tree/main/devtools"

    %% Styles
    classDef config fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef data fill:#AED6F1,stroke:#3498DB,stroke-width:2px;
    classDef model fill:#F5B7B1,stroke:#E74C3C,stroke-width:2px;
    classDef train fill:#A9DFBF,stroke:#27AE60,stroke-width:2px;
    classDef utils fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef testing fill:#FAD7A0,stroke:#E67E22,stroke-width:2px;
    classDef cicd fill:#D5F5E3,stroke:#28B463,stroke-width:2px;
```
