How jasonacox/tinyllm works
Project Type and Purpose
• This is a full-stack monorepo whose goal is to let you run and interact with “TinyLLM” models on consumer hardware. It bundles:
• Multiple inference engines (Ollama, vLLM, llama-cpp-python) exposing an OpenAI-compatible API.
• A FastAPI-based Chatbot web interface (with RAG, image, news, stock, weather commands).
• Vector DB modules (Qdrant, Chroma, Weaviate) for Retrieval-Augmented Generation.
• A monitoring stack and utility agents/scripts.
File Structure
• Top-level directories:
• agents/ (cron jobs and CLI helpers for news, movies, etc.)
• chatbot/ (FastAPI backend + templates + Docker assets + litellm local deploy)
• llmserver/ (llama-cpp-python server setup scripts and services)
• vllm/ (build, run, optional AWQ pipeline scripts)
• rag/ (scripts & Docker Compose for Qdrant, Chroma, Weaviate)
• monitoring/ (Telegraf, InfluxDB, Grafana dashboard)
• images/, lab/ (auxiliary experiments)
• root scripts (benchmark.py, chat.py, runserver.sh, loadtest.py)
README for Insights
• Describes three server options (Ollama, vLLM, llama-cpp), how to spin up each in Docker or as a service.
• Explains the Chatbot (port 5000) that uses the OpenAI API base to talk to your chosen LLM server.
• Details RAG features (URL summarize, /news, /stock, /weather, vector DB queries).
Main Components & Their Relationships
a. Users & Browser
• Interact via a web UI (HTML/JS/socket.io) served from the Chatbot service.
b. Chatbot Frontend
• Jinja2 templates and static JS in chatbot/app/templates (index.html, socket.io.js).
c. Chatbot Backend (FastAPI)
• Routes in chatbot/app/api/routes.py
• Core modules in chatbot/app/core (config, llm client, prompts, exceptions)
• RAG logic in chatbot/app/rag
• Image endpoint in chatbot/app/image
d. Inference Servers (OpenAI API Compatible)
• Ollama (docker run, port 11434)
• vLLM (vllm/build.sh, run.sh; GPU intensive)
• llama-cpp-python (llmserver/ with Dockerfile or systemd service)
e. RAG Modules & Vector DBs
• rag/chroma.py, rag/qdrant*.py, rag/weaviate/docker-compose.yml
• Persisted data under rag/qdrant/storage or external vector databases
f. External Agents & Features
• agents/news.py, agents/movie.py, agents/conversational.py
• Scripts for cron jobs (news-cron.sh), benchmarks, image generation
g. Monitoring Stack
• monitoring/Telegraf → InfluxDB → Grafana (dashboard.json)
h. Utility & Experimentation
• images/, lab/ for experimental servers and templates
Architectural Patterns & Design Principles
• Modular microservices: each concern (chat UI, inference API, RAG, monitoring) isolated.
• OpenAI-compatible interface abstracts away engine details.
• Vertical separation: UI → API → Model inference → Data layer → External services.
• Containerization (Dockerfiles, docker-compose) for portability & orchestration.
• Layers (top→bottom):
2. Chatbot Frontend (HTML/JS/socket.io)
3. Chatbot Backend (FastAPI)
4. Inference Engines cluster (Ollama, vLLM, llama-cpp-python)
5. RAG Services cluster (Chroma, Qdrant, Weaviate)
6. External Services (news API, weather API, stock API)
7. Monitoring Stack (Telegraf → InfluxDB → Grafana)