How kashinathj/local-llm-and-chatbot works
Project type + purpose (what you’re diagramming)
This is a 2-process, local-first “LLM-backed recipe assistant”:
• UI process: Streamlit chat app (src/bot.py) running in the user’s browser via Streamlit server.
• API process: FastAPI backend (src/api.py) that exposes POST /generate-recipe and orchestrates LLM calls.
• LLM providers (mode-dependent):
• Local: Ollama server running separately on the machine (Llama 3.1).
• Cloud: OpenAI API (GPT‑4o) reachable over the internet using OPENAI_API_KEY set in the API server environment.
• Shared domain model: Pydantic schema in src/models.py (Recipe JSON contract).
• Prompting assets / behavior shaping: few-shot dataset + Chef persona prompt in src/utils.py.
• LangChain glue: used in the API logic to call either ChatOllama or ChatOpenAI and to produce structured JSON output (or fallback JSON parsing for local mode).
This is not a monolith; it’s a small distributed system (even on localhost) with explicit boundaries (UI ↔ API ↔ LLM provider).
Top-level
• requirements.txt: runtime dependencies (call out FastAPI, Streamlit, LangChain, Pydantic, Uvicorn, etc.).
• run_api.*, run_bot.*, run_streamlit.ps1: “operational entrypoints” (useful for a deployment/runtime view).
src/
• api.py → Backend service boundary (FastAPI routes, request handling, LLM orchestration, env var checks, mode routing).
• bot.py → Frontend boundary (Streamlit UI, mode toggle, chat flow, calling the API, rendering recipe + nutrition sidebar).
• utils.py → Prompting / few-shot dataset (static assets influencing LLM output).
• utils.py + api.py together → “Prompt assembly + LLM invocation pipeline”.
• Two modes: mode=local vs mode=openai passed from UI → API.
• Structured output:
• OpenAI path uses with_structured_output (strong typing).
• Local path requires JSON extraction/parsing from LLM text (compatibility fallback).
• API key location: OPENAI_API_KEY is required only in API process env, not in the browser.
• Ports/process separation:
• FastAPI on 127.0.0.1:8001
• Streamlit on localhost:8501
• Healthcheck: GET /health exists; include it as an operational endpoint.
Streamlit App Container (process)
• Runs src/bot.py
• Responsible for chat UI, mode selection, rendering recipe
FastAPI Service Container (process)
• Runs src/api.py via Uvicorn
• Exposes /generate-recipe, /health
Ollama Server (Local LLM Runtime) (external container but on same machine)
• Hosts llama3.1
OpenAI API (external SaaS)
• Browser ↔ Streamlit: user messages + UI rendering
• Streamlit → FastAPI: HTTP POST /generate-recipe with {ingredients[], mode}
• FastAPI → Ollama: (local mode) LangChain ChatOllama call
• FastAPI → OpenAI: (openai mode) LangChain ChatOpenAI call (requires OPENAI_API_KEY)
• FastAPI → Streamlit: Recipe JSON response
• Streamlit → Browser: formatted recipe + nutrition sidebar
Annotate: ports (8501, 8001) and “localhost” vs “internet”.
• API Routes / Controllers (src/api.py)
• /generate-recipe handler
• /health
• Mode router: if mode=openai else local
• Error handling for missing/invalid key (OpenAI path)
• Prompt Builder (src/utils.py)
• Chef persona system prompt
• Few-shot dataset (10+ recipes)
• Schema / Validation (src/models.py)
• Recipe Pydantic model (response contract)
• Structured Output Adapter
• OpenAI: with_structured_output(Recipe)
• Local: “JSON parsing from model text” fallback
Routes → Orchestrator → Prompt Builder → LLM Client → (Structured Output Adapter) → Schema Validation → return JSON
This view is where you highlight the architectural pattern: “thin HTTP layer + orchestration pipeline + deterministic schema contract”.
Create two sequences (or one sequence with an alt block):
Sequence: Generate Recipe
alt mode=openai
• FastAPI uses ChatOpenAI + structured output
• OpenAI returns typed/structured response
else mode=local
• FastAPI uses ChatOllama
• Receives text, extracts JSON, validates to Recipe
Streamlit renders recipe (bold headers, numbered steps) + Nutrition Facts sidebar
Explicit failure branch:
• If mode=openai and API key missing/invalid → FastAPI returns error → Streamlit shows message and suggests Local mode.
Relationships & interactions to explicitly show (don’t omit)
These are the details that often get lost but matter here:
• Trust boundaries:
• Browser/UI does *not* hold secrets; API process holds OPENAI_API_KEY.
• Contract boundary:
• Provider abstraction:
• Local runtime dependency:
• Ollama must be running; call that out as a prerequisite note on the Ollama container.
• Operational endpoints:
• /health used for readiness checks and debugging.
Streamlit (src/bot.py)
• Chat UI (input, message history)
• Mode selector (Local/OpenAI)
• API Client (requests to FastAPI)
• Renderer (recipe formatting + nutrition sidebar)
FastAPI (src/api.py)
• Route: /generate-recipe
• Route: /health
• Request parsing (ingredients, mode)
• Prompt assembly (calls utils)
• LLM provider router
• OpenAI adapter (structured output)
• Ollama adapter (text → JSON parse)
• Pydantic validation (Recipe)
• Error mapper (key missing/invalid → user-friendly message)
Shared
• Recipe schema (src/models.py)
• Few-shot dataset + Chef prompt (src/utils.py)
External
• Ollama daemon + llama3.1 model
• OpenAI API