How yulezhang/jdcomment_spider works
Project Type and Purpose
• This is a Python-based web scraping tool – specifically a “JD爬虫” (JD Spider) – designed to extract user comments and associated metadata (like user ID, comment text, star rating, etc.) from the JD website.
• In addition, the project includes data analysis components: the JDComment_Processing.ipynb notebook shows data visualization (using bar charts and word clouds) to display patterns in purchase times, comment keywords, and member levels.
• There’s a secondary notebook (Taobao_Spider.ipynb) that indicates a similar scraping functionality for Taobao, even though the README emphasizes JD.
File Structure
• Source Files:
• SpiderScript.py: The primary Python script for JD web scraping.
• Taobao_Spider.ipynb: Notebook showing scraping or experimentation for Taobao.
• JDComment_Processing.ipynb: Notebook for processing and visualizing the scraped JD comments.
• Data Storage:
• The “data” directory holds CSV files (JDComment_data.csv, taobaoComment_data.csv) and text files with URL collections.
• Visual Assets & Resources:
• “picture” directory contains images used for demonstration or reporting (screenshots, figures for analysis, etc.).
• “front” directory contains the simsun.ttc font file likely used for rendering Chinese characters correctly in visualizations.
• Configuration & Requirements:
• requirements.txt lists the required Python libraries (requests, fake_useragent, BeautifulSoup).
• Git files (.gitignore, .gitattributes) and documentation files (README, database report files) indicate version control and further project documentation.
Key Components and Their Roles
a. Web Scraping Component:
• SpiderScript.py (and the Taobao_Spider.ipynb notebook) is the extraction engine.
• Utilizes core libraries (requests, fake_useragent, BeautifulSoup) to make HTTP requests, simulate browser requests, and parse JSON/HTML.
• Connects to external JD (and Taobao) servers, fetching comment data.
b. Data Storage:
• Scraped data is saved into CSV files in the “data” directory.
• The structure suggests an ETL-like pipeline where raw JSON data is processed into tabular (CSV) format.
c. Data Processing and Visualization:
• JDComment_Processing.ipynb is used for extracting insights from the scraped data.
• It generates visual outputs such as bar charts and word clouds, making use of plotting libraries (e.g., matplotlib, wordcloud – not explicitly listed but implied).
d. Configuration & Assets:
• requirements.txt ensures all dependencies are managed.
• The “front” folder assets like fonts and the “picture” folder provide visual support for UI or reports.
• Main Components to Include:
• External Websites: JD (and Taobao) comment interfaces.
• Web Scraper Module:
‣ SpiderScript.py (and Taobao_Spider.ipynb) — mark these as “Scraping Engine”.
‣ Indicate the use of libraries (requests, fake_useragent, BeautifulSoup).
• Data Storage:
‣ CSV files and text files in the “data” folder.
• Data Processing/Visualization:
‣ JDComment_Processing.ipynb, representing the analysis and report-generation part.
• Assets/Supporting Resources:
‣ “front” and “picture” directories for fonts and images.
• External Dependencies/Environment:
‣ Represent the Python runtime (Python 3.5.2) and possibly the Anaconda environment.
• Relationships and Interaction Flow:
1. The Scraper component makes HTTP requests to JD (and Taobao) servers.
2. Data is retrieved in JSON format, parsed and stored in CSV files.
3. The Data Processing module reads these CSV files to generate visual analytics.
4. Visual components (charts, word clouds) may use assets from “front” for proper text rendering.
• Architectural Patterns:
• Clearly, the project follows a data pipeline (Extract → Transform → Load) structure.
• Separation of concerns is evident: one set of scripts performs data extraction while another handles data analysis and visualization.