How sylvestergah/chatbotdocs works
Project type and purpose
• This is a single-page AI chatbot application built with Streamlit for the UI and a Retrieval-Augmented Generation (RAG) backend.
• It ingests multi-format Garmin product documentation (PDF/MD/CSV/etc.), processes it into embeddings (via OpenAI), stores vectors in FAISS, and serves context-aware responses.
File structure
• main/: core application (Streamlit UI, RAG orchestrator, config).
• dataNsop/: local Garmin product data repository (source documents).
• linkscraper/: web scraping utilities to gather online docs.
• others/: helper scripts (testing, conversation management, alternate RAG versions).
• config/chunk_config.json: chunking parameters for document splitting.
• requirements.txt: Python dependencies including Streamlit, LangChain, FAISS, OpenAI.
README insights
• Tech stack: Streamlit, OpenAI GPT-4, text-embedding-ada-002 (or 3-small), FAISS, LangChain, Pickle cache.
• Features: parallel processing, semantic search, memory, error handling.
Main components and interactions
a. User Interface (Streamlit front end)
• Handles user input, displays chat history and source citations.
• Component: main/app.py
b. RAG Backend (Python services)
• Document loader & chunker (LangChain) reads dataNsop & scraped docs.
• Embedding service calls OpenAI APIs to generate vector representations.
• Vector store (FAISS) persists embeddings.
• Retrieval mechanism (semantic search) pulls relevant chunks.
• Generation engine calls OpenAI GPT-4 to produce answers.
• Memory & cache (Pickle) store conversation context and metadata.
• Component: main/enhanced_rag.py
c. Data Storage
• Raw docs: dataNsop/ and scraped files.
• Vector store: FAISS index files on disk.
• Cache: Pickle files for metadata & chunk configs.
d. External services
• OpenAI endpoints for embeddings & completions.
• Upstream websites for scraping (linkscraper).
Architectural patterns & principles
• Modular monolith: UI, orchestration, data ingestion all in Python, separated by directories.
• Pipeline pattern: ingest → chunk → embed → store → retrieve → generate.
• Parallelism: document processing uses configurable worker pools.
• Caching: avoids repeated embedding of unchanged docs.
• Configuration-driven: chunk sizes, cache TTL in JSON.
Detailed capture
• Data formats (PDF/MD/TXT) flowing into the loader.
• Control/config files (chunk_config.json) feeding into the chunker.