# rithwikthirumalaraju14/agenticai-webscraper architecture

> Architecture diagram of the GitHub repository rithwikthirumalaraju14/agenticai-webscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rithwikthirumalaraju14/agenticai-webscraper
- Repository: https://github.com/rithwikthirumalaraju14/agenticai-webscraper

## Overview

##### Project type and purpose
- This is a full-stack Flask web application serving as an AI-powered web crawler, multimedia processor, query engine, voice interface, and data visualization tool.
- Its core use case is:
- Crawl arbitrary websites (text, images, audio, PDF, Word)
- Store and index content in a Chroma vector DB
- Answer natural-language queries via a Groq AI agent (optionally augmented by Searxng web search)
- Support voice input/output (ElevenLabs or gTTS + speech_recognition)
- Generate interactive Plotly visualizations

##### File structure
- Top level:
- app.py – Flask entrypoint, orchestrates routing, form/voice I/O, invokes crawler/agent/visualizer modules
- requirements.txt – Python dependencies
- static/ – CSS and generated Plotly assets
- templates/ – Jinja HTML (index.html)
- (Per README) Additional runtime directories:
- media/ (audio, video, pdf, word) – raw downloads
- tmp/ – agent caches (SQLite)
- store/ – Chroma vector DB files
- web_content.json – persisted crawl snapshot
- Configuration:
- .env – GROQ_API_KEY
- External dependencies: Poppler, Tesseract

##### Key architectural insights from README
- Monolithic Flask backend exposing web UI + REST endpoints
- Modular subcomponents: Crawler, Storage, AI Agent, Voice I/O, Visualization
- External services/APIs: Groq API, Searxng, ElevenLabs/gTTS, Tesseract, Poppler

1. User Interface (Browser)
– HTML/CSS/JS delivered by Flask (templates/, static/)
2. Flask Backend (app.py)
3. Web Crawler Module
– Uses requests + BeautifulSoup
– Downloads to media/
4. Storage Layer
– Chroma Vector DB (store/)
– SQLite (tmp/) for agent state
– Flat JSON (web_content.json)
5. AI Agent Module
– Groq API integration
– Optional Searxng web search
6. Voice Interface Module
– SpeechRecognition (input), ElevenLabs/gTTS (output), gTTS fallback
7. PDF/Image Processing Module
– poppler-utils, pdf2image, pytesseract
8. Visualization Module
– Plotly graph generation, assets in static/
9. External Services
– Groq AI Service
– Searxng Search API
– ElevenLabs TTS API

- User → Browser UI → “Crawl” request → Flask endpoint → Crawler → Storage
- User → “Query” or “Voice” request → Flask →
– Voice Module (if audio) ↔ speech_recognition / TTS → Flask
– AI Agent Module ↔ Groq API / Searxng → Storage lookup
- AI Agent → Storage (Chroma) for embeddings / retrieval
- Visualization Module → Storage or Agent output → Plotly → static assets → Browser
- PDF/Image Module invoked by Crawler → media/ + Storage

c. Architectural patterns & design principles
- Layered architecture: Presentation (templates/static) → Application (Flask+modules) → Data (Chroma/SQLite/files)
- Modular separation: each functional concern in its own sub-module
- External integration: clearly delineate third-party APIs vs. local processing

d. Technologies
- Flask (web framework), Jinja2 (templating)
- requests, BeautifulSoup4 (crawling)
- chromadb, langchain (vector storage)
- groq SDK, optional Searxng REST
- speech_recognition, pyttsx3/ElevenLabs/gTTS (voice)
- pdf2image, pytesseract (OCR)
- plotly (visualization)

##### Detail
- Don’t just show “Backend” – break it into Crawler, AI Agent, Voice, Visualization
- Both synchronous (HTTP) and asynchronous (file I/O) flows
- Startup/config steps (loading .env, instantiating DBs) if relevant

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Documentation
    Doc["README.md"]:::doc
    Req["requirements.txt"]:::doc

    %% Frontend Layer
    subgraph "Frontend Layer"
        Browser["Browser UI"]:::frontend
        Templates["Jinja2 Templates"]:::frontend
        Static["Static Assets (CSS & Plotly)"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        Flask["Flask Backend"]:::backend
        Crawler["Crawler Module"]:::module
        PDFProc["PDF/Image Processing Module"]:::module
        Agent["AI Agent Module"]:::module
        Voice["Voice Interface Module"]:::module
        Viz["Visualization Module"]:::module
    end

    %% Data Layer
    subgraph "Data Layer"
        MediaStore["Media Storage (media/)"]:::data
        WebContent["Web Content JSON"]:::data
        ChromaDB["Chroma Vector DB"]:::data
        SQLite["SQLite Cache (tmp/)"]:::data
    end

    %% External Services
    subgraph "External Services"
        Groq["Groq AI Service"]:::external
        Searx["Searxng Search API"]:::external
        ElevenLabs["ElevenLabs TTS API"]:::external
        SpeechRec["SpeechRecognition Engine"]:::external
        OCR["Poppler/Tesseract OCR"]:::external
    end

    %% Connections
    Doc --> Req
    Browser -->|HTTP Requests| Flask
    Flask -->|POST /crawl| Crawler
    Crawler -->|downloads files| MediaStore
    Crawler -->|invokes| PDFProc
    PDFProc -->|stores processed| MediaStore
    Crawler -->|writes| WebContent
    WebContent -->|ingest embeddings| ChromaDB

    Flask -->|POST /query| Agent
    Agent -->|vector search| ChromaDB
    Agent -->|optional web search| Searx
    Agent -->|calls AI| Groq

    Flask -->|voice I/O| Voice
    Voice -->|speech-to-text| SpeechRec
    Voice -->|text-to-speech| ElevenLabs

    Flask -->|generate visualization| Viz
    Viz -->|reads embeddings| ChromaDB
    Viz -->|outputs assets| Static
    Static --> Browser

    %% Click Events
    click Flask "https://github.com/rithwikthirumalaraju14/agenticai-webscraper/blob/main/app.py"
    click Doc "https://github.com/rithwikthirumalaraju14/agenticai-webscraper/blob/main/README.md"
    click Req "https://github.com/rithwikthirumalaraju14/agenticai-webscraper/blob/main/requirements.txt"
    click Templates "https://github.com/rithwikthirumalaraju14/agenticai-webscraper/blob/main/templates/index.html"
    click Static "https://github.com/rithwikthirumalaraju14/agenticai-webscraper/blob/main/static/styles.css"

    %% Styles
    classDef doc fill:#f9f,stroke:#333,stroke-width:1px
    classDef frontend fill:#bbf,stroke:#333,stroke-width:1px
    classDef backend fill:#bfb,stroke:#333,stroke-width:1px
    classDef module fill:#ffd,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef data fill:#fcf,stroke:#333,stroke-width:1px,shape:cylinder
    classDef external fill:#fee,stroke:#333,stroke-width:1px,shape:cloud
```
