How rithwikthirumalaraju14/agenticai-webscraper works
Project type and purpose
• This is a full-stack Flask web application serving as an AI-powered web crawler, multimedia processor, query engine, voice interface, and data visualization tool.
• Its core use case is:
• Crawl arbitrary websites (text, images, audio, PDF, Word)
• Store and index content in a Chroma vector DB
• Answer natural-language queries via a Groq AI agent (optionally augmented by Searxng web search)
• Support voice input/output (ElevenLabs or gTTS + speech_recognition)
• Generate interactive Plotly visualizations
File structure
• Top level:
• app.py – Flask entrypoint, orchestrates routing, form/voice I/O, invokes crawler/agent/visualizer modules
• requirements.txt – Python dependencies
• static/ – CSS and generated Plotly assets
• templates/ – Jinja HTML (index.html)
• (Per README) Additional runtime directories:
• media/ (audio, video, pdf, word) – raw downloads
• tmp/ – agent caches (SQLite)
• store/ – Chroma vector DB files
• web_content.json – persisted crawl snapshot
• Configuration:
• .env – GROQ_API_KEY
• External dependencies: Poppler, Tesseract
Key architectural insights from README
• Monolithic Flask backend exposing web UI + REST endpoints
• Modular subcomponents: Crawler, Storage, AI Agent, Voice I/O, Visualization
• External services/APIs: Groq API, Searxng, ElevenLabs/gTTS, Tesseract, Poppler
1. User Interface (Browser)
– HTML/CSS/JS delivered by Flask (templates/, static/)
2. Flask Backend (app.py)
3. Web Crawler Module
– Uses requests + BeautifulSoup
– Downloads to media/
4. Storage Layer
– Chroma Vector DB (store/)
– SQLite (tmp/) for agent state
– Flat JSON (web_content.json)
5. AI Agent Module
– Groq API integration
– Optional Searxng web search
6. Voice Interface Module
– SpeechRecognition (input), ElevenLabs/gTTS (output), gTTS fallback
7. PDF/Image Processing Module
– poppler-utils, pdf2image, pytesseract
8. Visualization Module
– Plotly graph generation, assets in static/
9. External Services
– Groq AI Service
– Searxng Search API
– ElevenLabs TTS API
• User → Browser UI → “Crawl” request → Flask endpoint → Crawler → Storage
• User → “Query” or “Voice” request → Flask →
– Voice Module (if audio) ↔ speech_recognition / TTS → Flask
– AI Agent Module ↔ Groq API / Searxng → Storage lookup
• AI Agent → Storage (Chroma) for embeddings / retrieval
• Visualization Module → Storage or Agent output → Plotly → static assets → Browser
• PDF/Image Module invoked by Crawler → media/ + Storage
c. Architectural patterns & design principles
• Layered architecture: Presentation (templates/static) → Application (Flask+modules) → Data (Chroma/SQLite/files)
• Modular separation: each functional concern in its own sub-module
• External integration: clearly delineate third-party APIs vs. local processing
d. Technologies
• Flask (web framework), Jinja2 (templating)
• requests, BeautifulSoup4 (crawling)
• chromadb, langchain (vector storage)
• groq SDK, optional Searxng REST
• speech_recognition, pyttsx3/ElevenLabs/gTTS (voice)
• pdf2image, pytesseract (OCR)
• plotly (visualization)
Detail
• Don’t just show “Backend” – break it into Crawler, AI Agent, Voice, Visualization
• Both synchronous (HTTP) and asynchronous (file I/O) flows
• Startup/config steps (loading .env, instantiating DBs) if relevant