How ayanb123/deep-web-research works
Project type and purpose
• From the file tree and minimal README, this is a full-stack data analytics/discovery web application built in Python.
• It uses Streamlit for the frontend UI (streamlit_app.py, streamlit_components/, streamlit_pages/) and a collection of Python modules for crawling, search, analytics, storage, and websockets.
• Its core use case is to crawl both clearnet and onion services, index and store the results (via a Chroma-based SQLite vector store), support advanced search and visualization (network graphs, metrics), and expose it all through a Streamlit web interface with real-time notifications.
File structure
• Top-level “streamlit_app.py” is the entry point for the UI, while “app.py”, “app_adapter.py”, “app_components.py” and “app_state.py” form the server/API layer behind the UI.
• “streamlit_components/” and “streamlit_pages/” hold custom widgets and page layouts.
• Crawling and ingestion live in “crawler.py”, “enhanced_crawler.py”, “crawler_pool.py”, “seed_data.py”, “clearnet_search.py”, “onion_database.py”, with retry and connection management utilities.
• Search and query modules include “query_builder.py”, “query_executor.py”, “search_service.py”.
• Data persistence uses “database_utils.py” for traditional storage and “chroma_db/chroma.sqlite3” as a vector store.
• Analytics and visualization are in “advanced_analytics.py” and “network_visualization.py”.
• Cross-cutting concerns: “config.py”, “security_utils.py”, “content_safety.py”, “notification_system.py”, “export_manager.py”, plus websockets (“websocket_manager.py”, “websocket_client.py”, “websocket_auth.py”).
Elements
• Clouds for external dependencies (clearnet search engine, onion network)
• Blue for UI components
• Green for core application services
• Orange for data stores
• Gray for utilities/security
Detail
• Grouping (e.g., wrap streamlit_components/ and streamlit_pages/ in a “Streamlit UI” boundary)
• Cross-cutting utilities (config, security, retry) as a shared side-bar or overlay