# jatin-mehra119/crawlgpt architecture

> Architecture diagram of the GitHub repository jatin-mehra119/crawlgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jatin-mehra119/crawlgpt
- Repository: https://github.com/jatin-mehra119/crawlgpt

## Overview

##### Project type and purpose
- This is a Python-based full-stack application combining:
- A data-intensive backend (web crawler, summarizer, vector database)
- A lightweight frontend (Streamlit chat interface)
- Embedded persistence (SQLite) and in-memory similarity search (FAISS)
- Purpose: Crawl web content asynchronously, chunk and embed it, store in FAISS/SQLite, and expose a chat UI for Retrieval-Augmented Generation (RAG).

##### File structure
- Top-level:
- Dockerfile, .github/workflows → containerization and CI/CD
- setup_env.py, pyproject.toml → environment and build metadata
- crawlgpt.db → persistent SQLite store
- src/crawlgpt/core:
- LLMBasedCrawler.py → orchestrates Playwright crawling, rate limiting, validation
- SummaryGenerator.py → text chunking, summary LLM calls
- DatabaseHandler.py → FAISS vector index management
- database.py → SQL schema and user/chat history
- src/crawlgpt/ui:
- chat_app.py → main Streamlit app (entry point)
- login.py → authentication flow (user management)
- chat_ui.py, run.py → UI helpers and launch script
- src/crawlgpt/utils:
- content_validator, data_manager, monitoring, progress, helper_functions → ancillary services
- tests/ → unit and integration coverage for core components

##### README for additional insights
- Confirms use of:
- Playwright for async crawling
- FAISS for vector database
- Groq/Ollama LLM APIs
- Streamlit for UI
- SQLite for user & chat persistence
- Mentions RAG, rate limiting, content validation, monitoring, progress tracking

##### Main components & their interactions
a. User Interface (Streamlit chat_app)
- Handles login → routes to chat_ui
- Sends queries to backend service layer
b. Authentication & Session Management
- database.py + login.py → SQLite stores users, hashed passwords, chat history
- session tokens or Streamlit st.session_state
c. Crawler Service (LLMBasedCrawler)
- Receives URLs or user queries → uses Playwright to fetch pages
- content_validator ensures valid URLs/content
d. Summary Service (SummaryGenerator)
- Chunks text, calls LLM (Groq/Ollama) for summarization
- Returns embeddings (sentence-transformers)
e. Vector Store (DatabaseHandler + FAISS)
- Stores/retrieves embeddings
- Persistent to disk (frozen FAISS index)
f. SQL Database (database.py)
- Stores user/chat metadata and raw text for audit
g. Utility Services
- monitoring.py collects metrics
- progress.py reports crawl/summarization progress
- data_manager handles import/export

##### Architectural patterns & principles
- Modular separation: core vs. ui vs. utils
- Single Responsibility: each core class does one job
- Dependency inversion: high-level services depend on abstract interfaces (e.g., DatabaseHandler)
- Event-driven/asynchronous: crawling and summarization run asynchronously
- RAG pattern: retrieval layer (FAISS) + generation layer (LLM)

##### Detail
- Layers: UI, auth, crawling, summarization, storage, utils
- Technology stacks next to each component (e.g., “FAISS (Python)”)
- How Docker/GitHub Actions fit:
- Deployment container includes all services
- CI/CD pipeline pushes to Hugging Face Spaces

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    subgraph "UI Layer"
        direction TB
        ChatApp["Streamlit Main App"]:::frontend
        Login["UI Login Flow"]:::frontend
        ChatUI["Chat UI Components"]:::frontend
        AppEntrypoint["App Entrypoint Wrapper"]:::frontend
        RunScript["Run Script"]:::frontend
    end

    %% Authentication & Session
    AuthDB["Auth & Session (SQLite Schema)"]:::database

    %% Service Layer
    subgraph "Service Layer"
        direction TB
        Crawler["LLMBasedCrawler\n(Playwright, Rate Limiting)"]:::service
        Summary["SummaryGenerator\n(Chunking, LLM)"]:::service
        VectorStore["DatabaseHandler\n(FAISS Index)"]:::database
        SQLStorage["SQLite File\n(crawlgpt.db)"]:::database
    end

    %% Cross-cutting Concerns
    subgraph "Cross-cutting Concerns"
        direction TB
        DataManager["Data Import/Export"]:::util
        Monitoring["Monitoring & Metrics"]:::util
        Progress["Progress Tracking"]:::util
        Helpers["Helper Functions"]:::util
        Validator["Content Validator"]:::util
    end

    %% External APIs
    subgraph "External Services"
        direction TB
        GroqAPI["Groq API"]:::external
        OllamaAPI["Ollama API"]:::external
    end

    %% Config & CI/CD
    subgraph "Config & CI/CD"
        direction TB
        EnvSetup["Environment Setup Script"]:::config
        Metadata["Project Metadata"]:::config
        ConfigVars["Config Variables"]:::config
        DockerSpec["Dockerfile"]:::config
        CIWorkflow["GitHub Actions Workflow"]:::config
    end

    %% Interactions
    ChatApp --> Login
    Login --> AuthDB
    ChatApp --> ChatUI
    ChatUI --> Crawler
    Crawler -->|fetch raw text| Summary
    Summary -->|embeddings| VectorStore
    Summary -->|raw text| SQLStorage
    Summary --> GroqAPI
    Summary --> OllamaAPI
    ChatUI -->|query retrieval| VectorStore
    VectorStore --> ChatUI

    %% Utilities usage
    Crawler -.->|validate content| Validator
    Crawler -.-> Progress
    Summary -.-> Progress
    Summary -.-> Monitoring
    DataManager --- SQLStorage
    Helpers --- Crawler
    Helpers --- Summary
    Monitoring --- Crawler
    Monitoring --- Summary

    %% Config connections
    EnvSetup --- Crawler
    Metadata --- Summary
    ConfigVars --- AuthDB
    DockerSpec --- Crawler
    DockerSpec --- Summary
    CIWorkflow --- DockerSpec

    %% Click Events
    click ChatApp "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/ui/chat_app.py"
    click Login "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/ui/login.py"
    click ChatUI "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/ui/chat_ui.py"
    click AppEntrypoint "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/ui/app.py"
    click RunScript "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/ui/run.py"
    click AuthDB "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/core/database.py"
    click Crawler "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/core/LLMBasedCrawler.py"
    click Validator "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/utils/content_validator.py"
    click Summary "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/core/SummaryGenerator.py"
    click VectorStore "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/core/DatabaseHandler.py"
    click SQLStorage "https://github.com/jatin-mehra119/crawlgpt/blob/main/crawlgpt.db"
    click DataManager "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/utils/data_manager.py"
    click Monitoring "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/utils/monitoring.py"
    click Progress "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/utils/progress.py"
    click Helpers "https://github.com/jatin-mehra119/crawlgpt/blob/main/src/crawlgpt/utils/helper_functions.py"
    click EnvSetup "https://github.com/jatin-mehra119/crawlgpt/blob/main/setup_env.py"
    click Metadata "https://github.com/jatin-mehra119/crawlgpt/blob/main/pyproject.toml"
    click ConfigVars "https://github.com/jatin-mehra119/crawlgpt/blob/main/config.py"
    click DockerSpec "https://github.com/jatin-mehra119/crawlgpt/tree/main/Dockerfile"
    click CIWorkflow "https://github.com/jatin-mehra119/crawlgpt/blob/main/.github/workflows/Push_to_hf.yaml"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#0288D1,color:#000
    classDef service fill:#E8F5E9,stroke:#2E7D32,color:#000
    classDef database fill:#FFF3E0,stroke:#EF6C00,color:#000
    classDef util fill:#ECEFF1,stroke:#607D8B,color:#000
    classDef external fill:#F3E5F5,stroke:#8E24AA,color:#000
    classDef config fill:#FFEBEE,stroke:#C62828,color:#000
```
