# jasonacox/tinyllm architecture

> Architecture diagram of the GitHub repository jasonacox/tinyllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jasonacox/tinyllm
- Repository: https://github.com/jasonacox/tinyllm

## Overview

##### Project Type and Purpose
- This is a full-stack monorepo whose goal is to let you run and interact with “TinyLLM” models on consumer hardware. It bundles:
- Multiple inference engines (Ollama, vLLM, llama-cpp-python) exposing an OpenAI-compatible API.
- A FastAPI-based Chatbot web interface (with RAG, image, news, stock, weather commands).
- Vector DB modules (Qdrant, Chroma, Weaviate) for Retrieval-Augmented Generation.
- A monitoring stack and utility agents/scripts.

##### File Structure
- Top-level directories:
- agents/ (cron jobs and CLI helpers for news, movies, etc.)
- chatbot/ (FastAPI backend + templates + Docker assets + litellm local deploy)
- llmserver/ (llama-cpp-python server setup scripts and services)
- vllm/ (build, run, optional AWQ pipeline scripts)
- rag/ (scripts & Docker Compose for Qdrant, Chroma, Weaviate)
- monitoring/ (Telegraf, InfluxDB, Grafana dashboard)
- images/, lab/ (auxiliary experiments)
- root scripts (benchmark.py, chat.py, runserver.sh, loadtest.py)

##### README for Insights
- Describes three server options (Ollama, vLLM, llama-cpp), how to spin up each in Docker or as a service.
- Explains the Chatbot (port 5000) that uses the OpenAI API base to talk to your chosen LLM server.
- Details RAG features (URL summarize, `/news`, `/stock`, `/weather`, vector DB queries).

##### Main Components & Their Relationships
a. Users & Browser
- Interact via a web UI (HTML/JS/socket.io) served from the Chatbot service.
b. Chatbot Frontend
- Jinja2 templates and static JS in chatbot/app/templates (index.html, socket.io.js).
c. Chatbot Backend (FastAPI)
- Routes in chatbot/app/api/routes.py
- Core modules in chatbot/app/core (config, llm client, prompts, exceptions)
- RAG logic in chatbot/app/rag
- Image endpoint in chatbot/app/image
d. Inference Servers (OpenAI API Compatible)
- Ollama (docker run, port 11434)
- vLLM (vllm/build.sh, run.sh; GPU intensive)
- llama-cpp-python (llmserver/ with Dockerfile or systemd service)
e. RAG Modules & Vector DBs
- rag/chroma.py, rag/qdrant*.py, rag/weaviate/docker-compose.yml
- Persisted data under rag/qdrant/storage or external vector databases
f. External Agents & Features
- agents/news.py, agents/movie.py, agents/conversational.py
- Scripts for cron jobs (news-cron.sh), benchmarks, image generation
g. Monitoring Stack
- monitoring/Telegraf → InfluxDB → Grafana (dashboard.json)
h. Utility & Experimentation
- images/, lab/ for experimental servers and templates

##### Architectural Patterns & Design Principles
- Modular microservices: each concern (chat UI, inference API, RAG, monitoring) isolated.
- OpenAI-compatible interface abstracts away engine details.
- Vertical separation: UI → API → Model inference → Data layer → External services.
- Containerization (Dockerfiles, docker-compose) for portability & orchestration.

- Layers (top→bottom):
2. Chatbot Frontend (HTML/JS/socket.io)
3. Chatbot Backend (FastAPI)
4. Inference Engines cluster (Ollama, vLLM, llama-cpp-python)
5. RAG Services cluster (Chroma, Qdrant, Weaviate)
6. External Services (news API, weather API, stock API)
7. Monitoring Stack (Telegraf → InfluxDB → Grafana)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Users Layer
    User["User & Browser"]:::ui

    %% Frontend Layer
    subgraph "Chatbot Frontend"
        direction TB
        Frontend["HTML/JS (socket.io)"]:::ui
    end

    %% Backend Layer
    subgraph "Chatbot Backend" 
        direction TB
        BackendMain["main.py"]:::backend
        APIRoutes["routes.py"]:::backend
        CoreConfig["config.py"]:::backend
        CoreLLM["llm.py"]:::backend
        CorePrompts["prompts.py"]:::backend
        CoreExceptions["exceptions.py"]:::backend
        RAGDocs["documents.py"]:::backend
        RAGCore["rag.py"]:::backend
        ImageModule["image/__init__.py"]:::backend
    end

    %% Agents Layer
    subgraph "External Agents & CLI Helpers"
        direction TB
        AgentNews["news.py"]:::script
        AgentMovie["movie.py"]:::script
        AgentConv["conversational.py"]:::script
        NewsCron["news-cron.sh"]:::script
    end

    %% Inference Engines Layer
    subgraph "Inference Engines Cluster"
        direction TB
        Ollama["Ollama\n(port 11434)"]:::inference
        VLLM["vLLM"]:::inference
        LlamaCPP["llama-cpp-python"]:::inference
    end

    %% RAG Services Layer
    subgraph "RAG & Vector DBs"
        direction TB
        Chroma["Chroma"]:::vector
        Qdrant["Qdrant"]:::vector
        Weaviate["Weaviate"]:::vector
    end

    %% External APIs Layer
    subgraph "External APIs"
        direction TB
        NewsAPI["News API"]:::external
        WeatherAPI["Weather API"]:::external
        StockAPI["Stock API"]:::external
    end

    %% Monitoring Layer
    subgraph "Monitoring Stack"
        direction TB
        Telegraf["Telegraf"]:::monitoring
        InfluxDB["InfluxDB"]:::monitoring
        Grafana["Grafana"]:::monitoring
    end

    %% Connections
    User -->|"HTTP/WebSocket"| Frontend
    Frontend -->|"WS/REST"| BackendMain
    BackendMain -->|"includes"| APIRoutes
    APIRoutes --> CoreConfig
    APIRoutes --> CoreLLM
    APIRoutes --> CorePrompts
    APIRoutes --> CoreExceptions
    APIRoutes --> RAGCore
    RAGCore --> RAGDocs
    APIRoutes --> ImageModule
    BackendMain -->|"spawns/uses"| AgentNews
    BackendMain --> AgentMovie
    BackendMain --> AgentConv
    AgentNews --> NewsCron

    BackendMain -->|"HTTP POST /generate"| Ollama
    BackendMain -->|"HTTP POST /generate"| VLLM
    BackendMain -->|"HTTP POST /generate"| LlamaCPP

    BackendMain -->|"RAG Query"| Chroma
    BackendMain -->|"RAG Query"| Qdrant
    BackendMain -->|"RAG Query"| Weaviate

    BackendMain -->|"REST"| NewsAPI
    BackendMain -->|"REST"| WeatherAPI
    BackendMain -->|"REST"| StockAPI

    %% Monitoring Flow
    Telegraf --> InfluxDB
    InfluxDB --> Grafana
    Telegraf --> Ollama
    Telegraf --> VLLM
    Telegraf --> LlamaCPP
    Telegraf --> Frontend
    Telegraf --> BackendMain

    %% Click Events
    click Frontend "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/templates/index.html"
    click Frontend "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/templates/socket.io.js"
    click BackendMain "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/main.py"
    click APIRoutes "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/api/routes.py"
    click CoreConfig "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/core/config.py"
    click CoreLLM "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/core/llm.py"
    click CorePrompts "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/core/prompts.py"
    click CoreExceptions "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/core/exceptions.py"
    click RAGDocs "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/rag/documents.py"
    click RAGCore "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/rag/rag.py"
    click ImageModule "https://github.com/jasonacox/tinyllm/blob/main/chatbot/app/image/__init__.py"
    click VLLM "https://github.com/jasonacox/tinyllm/blob/main/vllm/build.sh"
    click VLLM "https://github.com/jasonacox/tinyllm/blob/main/vllm/run.sh"
    click VLLM "https://github.com/jasonacox/tinyllm/tree/main/vllm/models/"
    click VLLM "https://github.com/jasonacox/tinyllm/blob/main/vllm/compile.sh"
    click VLLM "https://github.com/jasonacox/tinyllm/blob/main/vllm/run-awq.sh"
    click LlamaCPP "https://github.com/jasonacox/tinyllm/tree/main/llmserver/Dockerfile"
    click LlamaCPP "https://github.com/jasonacox/tinyllm/blob/main/llmserver/build.sh"
    click LlamaCPP "https://github.com/jasonacox/tinyllm/blob/main/llmserver/run.sh"
    click LlamaCPP "https://github.com/jasonacox/tinyllm/blob/main/llmserver/setup-docker.sh"
    click LlamaCPP "https://github.com/jasonacox/tinyllm/blob/main/llmserver/setup-python.sh"
    click LlamaCPP "https://github.com/jasonacox/tinyllm/tree/main/llmserver/models/"
    click Chroma "https://github.com/jasonacox/tinyllm/blob/main/rag/chroma.py"
    click Qdrant "https://github.com/jasonacox/tinyllm/blob/main/rag/qdrant.py"
    click Qdrant "https://github.com/jasonacox/tinyllm/blob/main/rag/qdrant-search.py"
    click Qdrant "https://github.com/jasonacox/tinyllm/blob/main/rag/qdrant-single.py"
    click Qdrant "https://github.com/jasonacox/tinyllm/blob/main/rag/qdrant/config/config.yaml"
    click Qdrant "https://github.com/jasonacox/tinyllm/blob/main/rag/qdrant/setup.sh"
    click Qdrant "https://github.com/jasonacox/tinyllm/tree/main/rag/qdrant/storage/"
    click Weaviate "https://github.com/jasonacox/tinyllm/blob/main/rag/weaviate/docker-compose.yml"
    click Weaviate "https://github.com/jasonacox/tinyllm/blob/main/rag/weaviate/README.md"
    click AgentNews "https://github.com/jasonacox/tinyllm/blob/main/agents/news.py"
    click AgentMovie "https://github.com/jasonacox/tinyllm/blob/main/agents/movie.py"
    click AgentConv "https://github.com/jasonacox/tinyllm/blob/main/agents/conversational.py"
    click NewsCron "https://github.com/jasonacox/tinyllm/blob/main/agents/news-cron.sh"
    click Telegraf "https://github.com/jasonacox/tinyllm/blob/main/monitoring/telegraf.conf"
    click Telegraf "https://github.com/jasonacox/tinyllm/blob/main/monitoring/monitor.py"
    click Telegraf "https://github.com/jasonacox/tinyllm/tree/main/monitoring/Dockerfile"
    click Grafana "https://github.com/jasonacox/tinyllm/blob/main/monitoring/dashboard.json"
    click InfluxDB "https://github.com/jasonacox/tinyllm/tree/main/monitoring/influxdb/"
    click Grafana "https://github.com/jasonacox/tinyllm/tree/main/monitoring/grafana/"
    click Telegraf "https://github.com/jasonacox/tinyllm/blob/main/monitoring/monitor.py"
    click User "https://github.com/jasonacox/tinyllm/blob/main/runserver.sh"

    %% Styles
    classDef ui fill:#cfe2f3,stroke:#333,stroke-width:1px
    classDef backend fill:#d9ead3,stroke:#333,stroke-width:1px
    classDef inference fill:#f9cb9c,stroke:#333,stroke-width:1px
    classDef vector fill:#d9d2e9,stroke:#333,stroke-width:1px
    classDef external fill:#e7e6e6,stroke:#333,stroke-width:1px
    classDef monitoring fill:#a2d9ce,stroke:#333,stroke-width:1px
    classDef script fill:#fce5cd,stroke:#333,stroke-width:1px
```
