How dbiswas55/vlmhub works
vlmhub provides one config-driven API for multimodal inference: a caller selects a registered hosting/model, Model resolves its merged settings, builds a shared text-and-image request, and dispatches it to either an API-hosted LiteLLM backend or in-process Transformers. The README also documents local-server and cloud providers, plus Transformers hardware-aware and quantized loading. A separate utility manages HuggingFace and Ollama model stores. Backend internals are only partially excerpted, so the graph omits unsupported internal call details; documented provider relationships are cited to the README.
Main entry points: model.py Model API · __init__.py Backend Factory · backends.py LiteLLM Backend · local_models.py Local Model Manager
Components 15
Client and Registry
- Model API
src/vlmhub/model.py - Registry Resolver
src/vlmhub/utils/config.py - Model Registry
src/vlmhub/models.json
Inference Runtime
- Prompt Blocks and Request
src/vlmhub/backends/request.py - Backend Factory
src/vlmhub/backends/__init__.py - LiteLLM Backend
src/vlmhub/backends/backends.py - Transformers Backend
src/vlmhub/backends/backends.py
Model Integrations
- LiteLLM Routing
- Local Inference Servers
- Cloud Model Providers
- Transformers Runtime
- Local Model Manager
src/vlmhub/utils/local_models.py - HuggingFace Cache
- Ollama Store
Other components
- Application Caller
Connections 14
- Application Caller → Model API: calls generate
README.md - Model API → Registry Resolver: resolves client
src/vlmhub/model.py - Registry Resolver → Model Registry: loads registry
src/vlmhub/utils/config.py - Model API → Backend Factory: creates backend
src/vlmhub/model.py - Model API → Prompt Blocks and Request: builds request
src/vlmhub/model.py - Model API → LiteLLM Backend: dispatches request
- Backend Factory → LiteLLM Backend: constructs LiteLLM
src/vlmhub/backends/__init__.py - Backend Factory → Transformers Backend: constructs Transformers
src/vlmhub/backends/__init__.py - LiteLLM Backend → LiteLLM Routing: routes inference
README.md - LiteLLM Routing → Local Inference Servers: serves local models
README.md - LiteLLM Routing → Cloud Model Providers: serves cloud models
README.md - Transformers Backend → Transformers Runtime: loads and runs
README.md - Local Model Manager → HuggingFace Cache: lists downloads deletes
src/vlmhub/utils/local_models.py - Local Model Manager → Ollama Store: lists pulls deletes
src/vlmhub/utils/local_models.py