# sylvestergah/chatbotdocs architecture

> Architecture diagram of the GitHub repository sylvestergah/chatbotdocs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sylvestergah/chatbotdocs
- Repository: https://github.com/sylvestergah/chatbotdocs

## Overview

##### Project type and purpose
- This is a single-page AI chatbot application built with Streamlit for the UI and a Retrieval-Augmented Generation (RAG) backend.
- It ingests multi-format Garmin product documentation (PDF/MD/CSV/etc.), processes it into embeddings (via OpenAI), stores vectors in FAISS, and serves context-aware responses.

##### File structure
- main/: core application (Streamlit UI, RAG orchestrator, config).
- dataNsop/: local Garmin product data repository (source documents).
- linkscraper/: web scraping utilities to gather online docs.
- others/: helper scripts (testing, conversation management, alternate RAG versions).
- config/chunk_config.json: chunking parameters for document splitting.
- requirements.txt: Python dependencies including Streamlit, LangChain, FAISS, OpenAI.

##### README insights
- Tech stack: Streamlit, OpenAI GPT-4, text-embedding-ada-002 (or 3-small), FAISS, LangChain, Pickle cache.
- Features: parallel processing, semantic search, memory, error handling.

##### Main components and interactions
a. User Interface (Streamlit front end)
- Handles user input, displays chat history and source citations.
- Component: main/app.py
b. RAG Backend (Python services)
- Document loader & chunker (LangChain) reads dataNsop & scraped docs.
- Embedding service calls OpenAI APIs to generate vector representations.
- Vector store (FAISS) persists embeddings.
- Retrieval mechanism (semantic search) pulls relevant chunks.
- Generation engine calls OpenAI GPT-4 to produce answers.
- Memory & cache (Pickle) store conversation context and metadata.
- Component: main/enhanced_rag.py
c. Data Storage
- Raw docs: dataNsop/ and scraped files.
- Vector store: FAISS index files on disk.
- Cache: Pickle files for metadata & chunk configs.
d. External services
- OpenAI endpoints for embeddings & completions.
- Upstream websites for scraping (linkscraper).

##### Architectural patterns & principles
- Modular monolith: UI, orchestration, data ingestion all in Python, separated by directories.
- Pipeline pattern: ingest → chunk → embed → store → retrieve → generate.
- Parallelism: document processing uses configurable worker pools.
- Caching: avoids repeated embedding of unchanged docs.
- Configuration-driven: chunk sizes, cache TTL in JSON.

##### Detailed capture
- Data formats (PDF/MD/TXT) flowing into the loader.
- Control/config files (chunk_config.json) feeding into the chunker.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Frontend Layer
    subgraph "Frontend" 
        style Frontend fill:#D6EAF8,stroke:#000
        User["User"]:::external
        UI["Streamlit UI<br/>(main/app.py)"]:::frontend
        AltUI["No-RAG Prototype UI<br/>(others/chatbot_ui.py)"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Services"
        style Backend Services fill:#D5F5E3,stroke:#000
        Orchestrator["RAG Orchestrator<br/>(main/enhanced_rag.py)"]:::backend
        AltRAG1["Alt RAG Version 1<br/>(main/enhanced_rag1.py)"]:::backend
        AltRAG2["Alt RAG Copy<br/>(main/enhanced_rag - Copy.py)"]:::backend
        Loader["Document Loader"]:::backend
        Chunker["Chunker<br/>(others/chunk_config.py)"]:::backend
        Config["Chunk Config JSON<br/>(main/config/chunk_config.json)"]:::backend
        Embedding["Embedding Service<br/>(OpenAI API)"]:::backend
        Retriever["Semantic Retriever"]:::backend
        Generator["Generation Engine<br/>(OpenAI GPT-4)"]:::backend
        Memory["Conversation Manager<br/>(others/conversation_manager.py)"]:::backend
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        style Data Storage fill:#FCF3CF,stroke:#000
        RawDocs["Raw Documents<br/>(dataNsop/)"]:::datastore
        FAISS["Vector Store (FAISS)"]:::datastore
        Cache["Pickle Cache"]:::datastore
    end

    %% External Integrations
    subgraph "External"
        style External fill:#F2F3F4,stroke:#000
        OpenAI["OpenAI Endpoints"]:::external
        Scraper["Web Scraping Utilities"]:::external
    end

    %% Connections
    User -->|sends query| UI
    UI -->|"calls"| Orchestrator
    UI -->|"calls"| AltUI
    Orchestrator -->|loads docs| Loader
    Loader -->|chunks docs| Chunker
    Config -->|params| Chunker
    Chunker -->|requests embeddings| Embedding
    Embedding -->|stores vectors| FAISS
    Orchestrator -->|semantic search| Retriever
    Retriever -->|fetches chunks| FAISS
    Retriever -->|passes context| Generator
    Generator -->|response| UI
    Orchestrator -->|read/write| Memory
    Scraper -->|scrapes online docs| RawDocs
    RawDocs -->|ingested by| Orchestrator
    Orchestrator -->|calls| OpenAI

    %% Component Mapping Clicks
    click UI "https://github.com/sylvestergah/chatbotdocs/blob/main/main/app.py"
    click Orchestrator "https://github.com/sylvestergah/chatbotdocs/blob/main/main/enhanced_rag.py"
    click AltRAG1 "https://github.com/sylvestergah/chatbotdocs/blob/main/main/enhanced_rag1.py"
    click AltRAG2 "https://github.com/sylvestergah/chatbotdocs/blob/main/main/enhanced_rag - Copy.py"
    click Config "https://github.com/sylvestergah/chatbotdocs/blob/main/main/config/chunk_config.json"
    click RawDocs "https://github.com/sylvestergah/chatbotdocs/tree/main/dataNsop/"
    click Scraper "https://github.com/sylvestergah/chatbotdocs/blob/main/linkscraper/newscraper.py"
    click Scraper "https://github.com/sylvestergah/chatbotdocs/blob/main/linkscraper/others/pdfscaper.py"
    click Scraper "https://github.com/sylvestergah/chatbotdocs/blob/main/linkscraper/others/pdfscaper_works4htmlpdf.py"
    click Scraper "https://github.com/sylvestergah/chatbotdocs/blob/main/linkscraper/others/pdfscaper_works4pdf.py"
    click Scraper "https://github.com/sylvestergah/chatbotdocs/blob/main/linkscraper/others/scraper.py"
    click Scraper "https://github.com/sylvestergah/chatbotdocs/blob/main/linkscraper/convert.py"
    click Scraper "https://github.com/sylvestergah/chatbotdocs/blob/main/linkscraper/cleaning.py"
    click Chunker "https://github.com/sylvestergah/chatbotdocs/blob/main/others/chunk_config.py"
    click Memory "https://github.com/sylvestergah/chatbotdocs/blob/main/others/conversation_manager.py"
    click AltUI "https://github.com/sylvestergah/chatbotdocs/blob/main/others/chatbot_ui.py"
    click AltUI "https://github.com/sylvestergah/chatbotdocs/blob/main/others/chatbotwithnorag.py"
    click UI "https://github.com/sylvestergah/chatbotdocs/blob/main/main/requirements.txt"

    %% Styles
    classDef frontend fill:#D6EAF8,stroke:#000,stroke-width:1px
    classDef backend fill:#D5F5E3,stroke:#000,stroke-width:1px
    classDef datastore fill:#FCF3CF,stroke:#000,stroke-width:1px
    classDef external fill:#F2F3F4,stroke:#000,stroke-width:1px
```
