How kangvcar/infospider works
Project Type and Purpose
• InfoSpider is a specialized data retrieval tool that aggregates personal data from multiple online sources (e.g., email providers, social media, e-commerce, blogs, etc.).
• Its purpose is to “take back your personal information” by crawling these disparate sources, storing the data uniformly (in JSON), and even providing basic visualization/analysis.
• The design must capture not only the different data source “spider” modules but also the supporting components for GUI, data analysis, and possibly future web-based operations.
Analyzing the File Structure
• The main business logic is housed under the “Spiders” directory. Inside, you have separate subdirectories (like A12306, JdSpider, alipay, bilibili, etc.) each containing a specific spider script—this indicates highly modular design for each data source.
• The “tools” directory contains a main script that likely serves as the orchestrator to start crawls.
• The “extension” directory (with HTML, CSS, JavaScript files) suggests a browser-based or desktop GUI extension component.
• The “tests” directory is organized by functionality (e.g., DeepAnalysis, blog_analyse, etc.), demonstrating separation between spider crawling and subsequent data analysis.
• Documentation is maintained in “docs”, and project configuration/dependency files are in the root (LICENSE, requirements.txt, etc.).
Key Components
a. Spider Modules:
◦ Represent each independent spider as a modular component responsible for a specific data source.
◦ Group them under a “Web Crawlers” or “Data Source Spiders” block.
b. Data Aggregation & Storage:
◦ A module that collects and stores all fetched data, noting that each spider outputs data in JSON format.
c. Data Analysis/Visualization:
◦ Include a component for data analysis, which is reflected by the “tests/DeepAnalysis” scripts and the “pyecharts” output (HTML charts).
d. User Interface & Orchestration:
◦ Show the orchestration layer (from “tools/main.py”, “uitest”, or even GUI components from the “extension”) that triggers spider execution.
◦ A GUI component interacts with steering functions—for example, letting the user choose data sources and save paths.
e. External Dependencies:
◦ Include external dependencies such as the Chrome browser and matching Chrome driver, which are vital for browser-based scraping.
◦ Note Python and any key libraries (like pyecharts and selenium if applicable).
Modeling Relationships and Interactions
• That each Spider feeds its output (JSON data) into a Data Aggregator module.
• Aggregator to the Data Analysis/Visualization component, where processed data is converted into analytical charts.
• Interaction paths that indicate dependency on external tools (e.g., the Chrome browser and drivers) for some spider modules.
Architectural Patterns and Technologies
• The design is modular with a plug-in like pattern, as every spider functions independently, satisfying separation of concerns.
• The project is built using Python 3 (per README and file tree) and adheres to a file structure that can easily scale with new spider modules.
• The architecture embraces a separation between data retrieval (crawlers) and data post-processing (analysis/visualization).
• Highlighting how the project could evolve into a full-stack solution in v2.0, with added web interface components (as mentioned in the plans).