# kangvcar/infospider architecture

> Architecture diagram of the GitHub repository kangvcar/infospider, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kangvcar/infospider
- Repository: https://github.com/kangvcar/infospider

## Overview

##### Project Type and Purpose
- InfoSpider is a specialized data retrieval tool that aggregates personal data from multiple online sources (e.g., email providers, social media, e-commerce, blogs, etc.).
- Its purpose is to “take back your personal information” by crawling these disparate sources, storing the data uniformly (in JSON), and even providing basic visualization/analysis.
- The design must capture not only the different data source “spider” modules but also the supporting components for GUI, data analysis, and possibly future web-based operations.

##### Analyzing the File Structure
- The main business logic is housed under the “Spiders” directory. Inside, you have separate subdirectories (like A12306, JdSpider, alipay, bilibili, etc.) each containing a specific spider script—this indicates highly modular design for each data source.
- The “tools” directory contains a main script that likely serves as the orchestrator to start crawls.
- The “extension” directory (with HTML, CSS, JavaScript files) suggests a browser-based or desktop GUI extension component.
- The “tests” directory is organized by functionality (e.g., DeepAnalysis, blog_analyse, etc.), demonstrating separation between spider crawling and subsequent data analysis.
- Documentation is maintained in “docs”, and project configuration/dependency files are in the root (LICENSE, requirements.txt, etc.).

##### Key Components
a. Spider Modules:
◦ Represent each independent spider as a modular component responsible for a specific data source.
◦ Group them under a “Web Crawlers” or “Data Source Spiders” block.

b. Data Aggregation & Storage:
◦ A module that collects and stores all fetched data, noting that each spider outputs data in JSON format.

c. Data Analysis/Visualization:
◦ Include a component for data analysis, which is reflected by the “tests/DeepAnalysis” scripts and the “pyecharts” output (HTML charts).

d. User Interface & Orchestration:
◦ Show the orchestration layer (from “tools/main.py”, “uitest”, or even GUI components from the “extension”) that triggers spider execution.
◦ A GUI component interacts with steering functions—for example, letting the user choose data sources and save paths.

e. External Dependencies:
◦ Include external dependencies such as the Chrome browser and matching Chrome driver, which are vital for browser-based scraping.
◦ Note Python and any key libraries (like pyecharts and selenium if applicable).

##### Modeling Relationships and Interactions
- That each Spider feeds its output (JSON data) into a Data Aggregator module.
- Aggregator to the Data Analysis/Visualization component, where processed data is converted into analytical charts.
- Interaction paths that indicate dependency on external tools (e.g., the Chrome browser and drivers) for some spider modules.

##### Architectural Patterns and Technologies
- The design is modular with a plug-in like pattern, as every spider functions independently, satisfying separation of concerns.
- The project is built using Python 3 (per README and file tree) and adheres to a file structure that can easily scale with new spider modules.
- The architecture embraces a separation between data retrieval (crawlers) and data post-processing (analysis/visualization).
- Highlighting how the project could evolve into a full-stack solution in v2.0, with added web interface components (as mentioned in the plans).

## Diagram (Mermaid)

```mermaid
graph TD
    %% User Interface & Orchestration
    subgraph "User Interface & Orchestration"
        UI_Orch["Main Orchestrator (tools/main.py)"]:::orchestrator
        Extension["GUI Extension (extension)"]:::ui
        UI_Test["UI Test (uitest/main.py)"]:::ui
    end

    %% Spider Modules
    subgraph "Spider Modules"
        Spider_A12306["Spider: A12306"]:::spider
        Spider_JdSpider["Spider: JdSpider"]:::spider
        Spider_alipay["Spider: alipay"]:::spider
        Spider_bilibili["Spider: bilibili"]:::spider
        OtherSpiders["Other Spiders"]:::spider
    end

    %% Data Post-Processing
    subgraph "Data Post-Processing"
        Aggregator["Data Aggregator (JSON Storage)"]:::aggregator
        Analysis_DA["Deep Analysis (tests/DeepAnalysis)"]:::analysis
        Analysis_Blog["Blog Analysis (tests/blog_analyse)"]:::analysis
    end

    %% External Dependencies
    subgraph "External Dependencies"
        External["External Dependencies (Chrome, Python, Libraries)"]:::external
    end

    %% Connections from UI to Spider Modules
    UI_Orch -->|"triggers"| Spider_A12306
    UI_Orch -->|"triggers"| Spider_JdSpider
    UI_Orch -->|"triggers"| Spider_alipay
    UI_Orch -->|"triggers"| Spider_bilibili
    UI_Orch -->|"triggers"| OtherSpiders

    %% Connections from Spider Modules to Data Aggregator
    Spider_A12306 -->|"outputs"| Aggregator
    Spider_JdSpider -->|"outputs"| Aggregator
    Spider_alipay -->|"outputs"| Aggregator
    Spider_bilibili -->|"outputs"| Aggregator
    OtherSpiders -->|"outputs"| Aggregator

    %% Connections from Data Aggregator to Analysis Modules
    Aggregator -->|"feeds"| Analysis_DA
    Aggregator -->|"feeds"| Analysis_Blog

    %% UI internal interactions
    Extension -->|"interacts"| UI_Orch
    UI_Test -->|"tests"| UI_Orch

    %% External dependencies usage
    UI_Orch -->|"uses"| External
    Spider_A12306 -->|"requires"| External
    Spider_JdSpider -->|"requires"| External
    Spider_alipay -->|"requires"| External
    Spider_bilibili -->|"requires"| External
    OtherSpiders -->|"requires"| External

    %% Click Events
    click UI_Orch "https://github.com/kangvcar/infospider/blob/master/tools/main.py"
    click Extension "https://github.com/kangvcar/infospider/tree/master/extension"
    click UI_Test "https://github.com/kangvcar/infospider/blob/master/uitest/main.py"
    click Spider_A12306 "https://github.com/kangvcar/infospider/tree/master/Spiders/A12306"
    click Spider_JdSpider "https://github.com/kangvcar/infospider/tree/master/Spiders/JdSpider"
    click Spider_alipay "https://github.com/kangvcar/infospider/tree/master/Spiders/alipay"
    click Spider_bilibili "https://github.com/kangvcar/infospider/tree/master/Spiders/bilibili"
    click Aggregator "https://github.com/kangvcar/infospider/blob/master/Spiders/taobao/taobao_cookies.json"
    click Analysis_DA "https://github.com/kangvcar/infospider/tree/master/tests/DeepAnalysis"
    click Analysis_Blog "https://github.com/kangvcar/infospider/tree/master/tests/blog_analyse"

    %% Styles
    classDef orchestrator fill:#FFCCCC,stroke:#990000,stroke-width:2px;
    classDef ui fill:#FFE6CC,stroke:#CC6600,stroke-width:2px;
    classDef spider fill:#CCFFCC,stroke:#009900,stroke-width:2px;
    classDef aggregator fill:#CCCCFF,stroke:#000099,stroke-width:2px;
    classDef analysis fill:#FFFFCC,stroke:#999900,stroke-width:2px;
    classDef external fill:#E6E6E6,stroke:#666666,stroke-width:2px;
```
