# ayanb123/deep-web-research architecture

> Architecture diagram of the GitHub repository ayanb123/deep-web-research, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayanb123/deep-web-research
- Repository: https://github.com/ayanb123/deep-web-research

## Overview

##### Project type and purpose
- From the file tree and minimal README, this is a full-stack data analytics/discovery web application built in Python.
- It uses Streamlit for the frontend UI (streamlit_app.py, streamlit_components/, streamlit_pages/) and a collection of Python modules for crawling, search, analytics, storage, and websockets.
- Its core use case is to crawl both clearnet and onion services, index and store the results (via a Chroma-based SQLite vector store), support advanced search and visualization (network graphs, metrics), and expose it all through a Streamlit web interface with real-time notifications.

##### File structure
- Top-level “streamlit_app.py” is the entry point for the UI, while “app.py”, “app_adapter.py”, “app_components.py” and “app_state.py” form the server/API layer behind the UI.
- “streamlit_components/” and “streamlit_pages/” hold custom widgets and page layouts.
- Crawling and ingestion live in “crawler.py”, “enhanced_crawler.py”, “crawler_pool.py”, “seed_data.py”, “clearnet_search.py”, “onion_database.py”, with retry and connection management utilities.
- Search and query modules include “query_builder.py”, “query_executor.py”, “search_service.py”.
- Data persistence uses “database_utils.py” for traditional storage and “chroma_db/chroma.sqlite3” as a vector store.
- Analytics and visualization are in “advanced_analytics.py” and “network_visualization.py”.
- Cross-cutting concerns: “config.py”, “security_utils.py”, “content_safety.py”, “notification_system.py”, “export_manager.py”, plus websockets (“websocket_manager.py”, “websocket_client.py”, “websocket_auth.py”).

##### Elements
- Clouds for external dependencies (clearnet search engine, onion network)
- Blue for UI components
- Green for core application services
- Orange for data stores
- Gray for utilities/security

##### Detail
- Grouping (e.g., wrap streamlit_components/ and streamlit_pages/ in a “Streamlit UI” boundary)
- Cross-cutting utilities (config, security, retry) as a shared side-bar or overlay

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    Cloud_Clearnet("Clearnet Search Engine"):::external
    Cloud_OnionNetwork("Onion Network"):::external

    %% Frontend Tier
    subgraph "Frontend Tier"
        style FrontendTier fill: #E8F4FC, stroke:#0366D6
        streamlit_app["streamlit_app.py"]:::ui
        streamlit_components["streamlit_components/"]:::ui
        streamlit_pages["streamlit_pages/"]:::ui
        streamlit_websocket_component["streamlit_websocket_component.py"]:::ui
    end

    %% Application Tier
    subgraph "Application Tier"
        style AppTier fill: #E8F9E4, stroke:#28A745
        app_py["app.py"]:::core
        app_adapter["app_adapter.py"]:::core
        app_components["app_components.py"]:::core
        app_state["app_state.py"]:::core

        subgraph "Crawling & Ingestion"
            style CrawlingTier fill:#E8F9E4, stroke:#28A745
            crawler["crawler.py"]:::core
            enhanced_crawler["enhanced_crawler.py"]:::core
            crawler_pool["crawler_pool.py"]:::core
            clearnet_search["clearnet_search.py"]:::core
            onion_database["onion_database.py"]:::core
            seed_data["seed_data.py"]:::core
        end

        subgraph "Search & Query"
            style SearchTier fill:#E8F9E4, stroke:#28A745
            query_builder["query_builder.py"]:::core
            query_executor["query_executor.py"]:::core
            search_service["search_service.py"]:::core
        end

        subgraph "Analytics & Visualization"
            style AnalyticsTier fill:#E8F9E4, stroke:#28A745
            advanced_analytics["advanced_analytics.py"]:::core
            network_visualization["network_visualization.py"]:::core
        end

        subgraph "Notifications & Export"
            style NotifyTier fill:#E8F9E4, stroke:#28A745
            notification_system["notification_system.py"]:::core
            export_manager["export_manager.py"]:::core
        end

        subgraph "Websocket Layer"
            style WebsocketTier fill:#E8F9E4, stroke:#28A745
            websocket_manager["websocket_manager.py"]:::core
            websocket_client["websocket_client.py"]:::core
            websocket_auth["websocket_auth.py"]:::core
        end

        subgraph "Utilities & Config"
            style UtilsTier fill:#F0F0F0, stroke:#6A737D
            config["config.py"]:::util
            security_utils["security_utils.py"]:::util
            retry_utils["retry_utils.py"]:::util
            connection_manager["connection_manager.py"]:::util
            content_safety["content_safety.py"]:::util
        end
    end

    %% Data Tier
    subgraph "Data Tier"
        style DataTier fill:#FFF4E5, stroke:#D97706
        chroma_db["chroma_db/chroma.sqlite3"]:::data
        database_utils["database_utils.py"]:::data
    end

    %% Relationships
    User["User"]:::external -->|"interacts with"| streamlit_app
    streamlit_app -->|"calls API"| app_py
    streamlit_app -->|"loads pages"| streamlit_pages
    streamlit_app -->|"uses components"| streamlit_components
    streamlit_websocket_component -->|"WS connect"| websocket_manager
    websocket_manager -->|"auth via"| websocket_auth
    websocket_manager -->|"notify client"| streamlit_websocket_component

    app_py -->|"adapts to UI"| app_adapter
    app_py -->|"uses UI components"| app_components
    app_py -->|"manages state"| app_state

    app_py -->|"triggers crawl"| crawler_pool
    crawler_pool --> crawler
    crawler --> clearnet_search
    clearnet_search --> Cloud_Clearnet
    crawler --> onion_database
    onion_database --> Cloud_OnionNetwork
    crawler_pool --> enhanced_crawler
    seed_data --> crawler_pool
    crawler --> database_utils

    app_py -->|"executes search"| search_service
    search_service --> query_builder
    search_service --> query_executor
    query_executor --> database_utils

    database_utils --> chroma_db
    search_service --> chroma_db

    advanced_analytics --> database_utils
    network_visualization --> database_utils

    notification_system --> database_utils
    notification_system --> websocket_manager

    export_manager --> database_utils
    export_manager --> app_state

    %% Utilities
    app_py --> config
    app_py --> security_utils
    crawler --> retry_utils
    crawler_pool --> connection_manager
    content_safety --> crawler
    app_py --> content_safety

    %% Click Events
    click streamlit_app "https://github.com/ayanb123/deep-web-research/blob/main/streamlit_app.py"
    click streamlit_components "https://github.com/ayanb123/deep-web-research/tree/main/streamlit_components/"
    click streamlit_pages "https://github.com/ayanb123/deep-web-research/tree/main/streamlit_pages/"
    click streamlit_websocket_component "https://github.com/ayanb123/deep-web-research/blob/main/streamlit_websocket_component.py"
    click app_py "https://github.com/ayanb123/deep-web-research/blob/main/app.py"
    click app_adapter "https://github.com/ayanb123/deep-web-research/blob/main/app_adapter.py"
    click app_components "https://github.com/ayanb123/deep-web-research/blob/main/app_components.py"
    click app_state "https://github.com/ayanb123/deep-web-research/blob/main/app_state.py"
    click crawler "https://github.com/ayanb123/deep-web-research/blob/main/crawler.py"
    click enhanced_crawler "https://github.com/ayanb123/deep-web-research/blob/main/enhanced_crawler.py"
    click crawler_pool "https://github.com/ayanb123/deep-web-research/blob/main/crawler_pool.py"
    click clearnet_search "https://github.com/ayanb123/deep-web-research/blob/main/clearnet_search.py"
    click onion_database "https://github.com/ayanb123/deep-web-research/blob/main/onion_database.py"
    click seed_data "https://github.com/ayanb123/deep-web-research/blob/main/seed_data.py"
    click query_builder "https://github.com/ayanb123/deep-web-research/blob/main/query_builder.py"
    click query_executor "https://github.com/ayanb123/deep-web-research/blob/main/query_executor.py"
    click search_service "https://github.com/ayanb123/deep-web-research/blob/main/search_service.py"
    click chroma_db "https://github.com/ayanb123/deep-web-research/blob/main/chroma_db/chroma.sqlite3"
    click database_utils "https://github.com/ayanb123/deep-web-research/blob/main/database_utils.py"
    click advanced_analytics "https://github.com/ayanb123/deep-web-research/blob/main/advanced_analytics.py"
    click network_visualization "https://github.com/ayanb123/deep-web-research/blob/main/network_visualization.py"
    click notification_system "https://github.com/ayanb123/deep-web-research/blob/main/notification_system.py"
    click export_manager "https://github.com/ayanb123/deep-web-research/blob/main/export_manager.py"
    click websocket_manager "https://github.com/ayanb123/deep-web-research/blob/main/websocket_manager.py"
    click websocket_client "https://github.com/ayanb123/deep-web-research/blob/main/websocket_client.py"
    click websocket_auth "https://github.com/ayanb123/deep-web-research/blob/main/websocket_auth.py"
    click config "https://github.com/ayanb123/deep-web-research/blob/main/config.py"
    click security_utils "https://github.com/ayanb123/deep-web-research/blob/main/security_utils.py"
    click retry_utils "https://github.com/ayanb123/deep-web-research/blob/main/retry_utils.py"
    click connection_manager "https://github.com/ayanb123/deep-web-research/blob/main/connection_manager.py"
    click content_safety "https://github.com/ayanb123/deep-web-research/blob/main/content_safety.py"

    %% Styles
    classDef ui fill:#BEE3F8,stroke:#0366D6,color:#0366D6
    classDef core fill:#C6F6D5,stroke:#28A745,color:#1F581F
    classDef data fill:#FEEBC8,stroke:#D97706,color:#B45309
    classDef util fill:#E2E8F0,stroke:#6A737D,color:#6A737D
    classDef external fill:#F5F5F5,stroke:#A0A0A0,stroke-dasharray: 5, color:#333
```
