# dbiswas55/vlmhub architecture

> Architecture diagram of the GitHub repository dbiswas55/vlmhub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dbiswas55/vlmhub
- Repository: https://github.com/dbiswas55/vlmhub

## Overview

vlmhub provides one config-driven API for multimodal inference: a caller selects a registered hosting/model, Model resolves its merged settings, builds a shared text-and-image request, and dispatches it to either an API-hosted LiteLLM backend or in-process Transformers. The README also documents local-server and cloud providers, plus Transformers hardware-aware and quantized loading. A separate utility manages HuggingFace and Ollama model stores. Backend internals are only partially excerpted, so the graph omits unsupported internal call details; documented provider relationships are cited to the README.

Main entry points: [`src/vlmhub/model.py`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/model.py) (Model API), [`src/vlmhub/backends/__init__.py`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/__init__.py) (Backend Factory), [`src/vlmhub/backends/backends.py`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/backends.py) (LiteLLM Backend), [`src/vlmhub/utils/local_models.py`](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/local_models.py) (Local Model Manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Client and Registry"]
  node_model["Model API<br/>[model.py]"]
  node_config["Registry Resolver<br/>[config.py]"]
  node_registry["Model Registry<br/>[models.json]"]
end

subgraph group_inference["Inference Runtime"]
  node_request["Prompt Blocks and Request<br/>[request.py]"]
  node_factory["Backend Factory<br/>[__init__.py]"]
  node_litebackend["LiteLLM Backend<br/>[backends.py]"]
  node_transformerbackend["Transformers Backend<br/>[backends.py]"]
end

subgraph group_integrations["Model Integrations"]
  node_litellm["LiteLLM Routing"]
  node_localservers["Local Inference Servers"]
  node_cloudproviders["Cloud Model Providers"]
  node_transformersruntime["Transformers Runtime"]
  node_localmanager["Local Model Manager<br/>[local_models.py]"]
  node_hfcache[("HuggingFace Cache")]
  node_ollamastore[("Ollama Store")]
end

node_caller(("Application Caller"))

node_caller -->|"calls generate"| node_model
node_model -->|"resolves client"| node_config
node_config -->|"loads registry"| node_registry
node_model -->|"creates backend"| node_factory
node_model -->|"builds request"| node_request
node_model -.->|"dispatches request"| node_litebackend
node_factory -->|"constructs LiteLLM"| node_litebackend
node_factory -->|"constructs Transformers"| node_transformerbackend
node_litebackend -.->|"routes inference"| node_litellm
node_litellm -->|"serves local models"| node_localservers
node_litellm -->|"serves cloud models"| node_cloudproviders
node_transformerbackend -->|"loads and runs"| node_transformersruntime
node_localmanager -->|"lists downloads deletes"| node_hfcache
node_localmanager -->|"lists pulls deletes"| node_ollamastore

click node_model "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/model.py"
click node_config "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/config.py"
click node_registry "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/models.json"
click node_request "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/request.py"
click node_factory "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/__init__.py"
click node_litebackend "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/backends.py"
click node_transformerbackend "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/backends.py"
click node_localmanager "https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/local_models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_model,node_config,node_registry toneBlue
class node_request,node_factory,node_litebackend,node_transformerbackend toneAmber
class node_litellm,node_localservers,node_cloudproviders,node_transformersruntime,node_localmanager,node_hfcache,node_ollamastore toneMint
class node_caller toneIndigo
```

## Components

### Client and Registry

- [Model API](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/model.py): `src/vlmhub/model.py`
- [Registry Resolver](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/config.py): `src/vlmhub/utils/config.py`
- [Model Registry](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/models.json): `src/vlmhub/models.json`

### Inference Runtime

- [Prompt Blocks and Request](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/request.py): `src/vlmhub/backends/request.py`
- [Backend Factory](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/__init__.py): `src/vlmhub/backends/__init__.py`
- [LiteLLM Backend](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/backends.py): `src/vlmhub/backends/backends.py`
- [Transformers Backend](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/backends/backends.py): `src/vlmhub/backends/backends.py`

### Model Integrations

- **LiteLLM Routing**
- **Local Inference Servers**
- **Cloud Model Providers**
- **Transformers Runtime**
- [Local Model Manager](https://github.com/dbiswas55/vlmhub/blob/main/src/vlmhub/utils/local_models.py): `src/vlmhub/utils/local_models.py`
- **HuggingFace Cache**
- **Ollama Store**

### Other components

- **Application Caller**

## Connections

- Application Caller → Model API: calls generate (evidence: `README.md`)
- Model API → Registry Resolver: resolves client (evidence: `src/vlmhub/model.py`)
- Registry Resolver → Model Registry: loads registry (evidence: `src/vlmhub/utils/config.py`)
- Model API → Backend Factory: creates backend (evidence: `src/vlmhub/model.py`)
- Model API → Prompt Blocks and Request: builds request (evidence: `src/vlmhub/model.py`)
- Model API → LiteLLM Backend: dispatches request
- Backend Factory → LiteLLM Backend: constructs LiteLLM (evidence: `src/vlmhub/backends/__init__.py`)
- Backend Factory → Transformers Backend: constructs Transformers (evidence: `src/vlmhub/backends/__init__.py`)
- LiteLLM Backend → LiteLLM Routing: routes inference (evidence: `README.md`)
- LiteLLM Routing → Local Inference Servers: serves local models (evidence: `README.md`)
- LiteLLM Routing → Cloud Model Providers: serves cloud models (evidence: `README.md`)
- Transformers Backend → Transformers Runtime: loads and runs (evidence: `README.md`)
- Local Model Manager → HuggingFace Cache: lists downloads deletes (evidence: `src/vlmhub/utils/local_models.py`)
- Local Model Manager → Ollama Store: lists pulls deletes (evidence: `src/vlmhub/utils/local_models.py`)