How jatin-mehra119/crawlgpt works
Project type and purpose
• This is a Python-based full-stack application combining:
• A data-intensive backend (web crawler, summarizer, vector database)
• A lightweight frontend (Streamlit chat interface)
• Embedded persistence (SQLite) and in-memory similarity search (FAISS)
• Purpose: Crawl web content asynchronously, chunk and embed it, store in FAISS/SQLite, and expose a chat UI for Retrieval-Augmented Generation (RAG).
File structure
• Top-level:
• Dockerfile, .github/workflows → containerization and CI/CD
• setup_env.py, pyproject.toml → environment and build metadata
• crawlgpt.db → persistent SQLite store
• src/crawlgpt/core:
• LLMBasedCrawler.py → orchestrates Playwright crawling, rate limiting, validation
• SummaryGenerator.py → text chunking, summary LLM calls
• DatabaseHandler.py → FAISS vector index management
• database.py → SQL schema and user/chat history
• src/crawlgpt/ui:
• chat_app.py → main Streamlit app (entry point)
• login.py → authentication flow (user management)
• chat_ui.py, run.py → UI helpers and launch script
• src/crawlgpt/utils:
• content_validator, data_manager, monitoring, progress, helper_functions → ancillary services
• tests/ → unit and integration coverage for core components
README for additional insights
• Confirms use of:
• Playwright for async crawling
• FAISS for vector database
• Groq/Ollama LLM APIs
• Streamlit for UI
• SQLite for user & chat persistence
• Mentions RAG, rate limiting, content validation, monitoring, progress tracking
Main components & their interactions
a. User Interface (Streamlit chat_app)
• Handles login → routes to chat_ui
• Sends queries to backend service layer
b. Authentication & Session Management
• database.py + login.py → SQLite stores users, hashed passwords, chat history
• session tokens or Streamlit st.session_state
c. Crawler Service (LLMBasedCrawler)
• Receives URLs or user queries → uses Playwright to fetch pages
• content_validator ensures valid URLs/content
d. Summary Service (SummaryGenerator)
• Chunks text, calls LLM (Groq/Ollama) for summarization
• Returns embeddings (sentence-transformers)
e. Vector Store (DatabaseHandler + FAISS)
• Stores/retrieves embeddings
• Persistent to disk (frozen FAISS index)
f. SQL Database (database.py)
• Stores user/chat metadata and raw text for audit
g. Utility Services
• monitoring.py collects metrics
• progress.py reports crawl/summarization progress
• data_manager handles import/export
Architectural patterns & principles
• Modular separation: core vs. ui vs. utils
• Single Responsibility: each core class does one job
• Dependency inversion: high-level services depend on abstract interfaces (e.g., DatabaseHandler)
• Event-driven/asynchronous: crawling and summarization run asynchronously
• RAG pattern: retrieval layer (FAISS) + generation layer (LLM)
Detail
• Layers: UI, auth, crawling, summarization, storage, utils
• Technology stacks next to each component (e.g., “FAISS (Python)”)
• How Docker/GitHub Actions fit:
• Deployment container includes all services
• CI/CD pipeline pushes to Hugging Face Spaces