# py-bin/dianping_textmining architecture

> Architecture diagram of the GitHub repository py-bin/dianping_textmining, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/py-bin/dianping_textmining
- Repository: https://github.com/py-bin/dianping_textmining

## Overview

This repository is a Chinese Dianping review-mining project: it collects shop reviews, stores them, explores and preprocesses review text, then trains and evaluates sentiment classifiers. The README also describes word-cloud analysis, TF-IDF, imbalance handling, and sample prediction. The crawler-to-database call is visible in source; notebook implementations are not excerpted, so notebook relationships below reflect README-described workflow rather than inspected calls.

Main entry points: [`文本分析挖掘/文本挖掘&情感分析.ipynb`](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb) (Text preprocessing), [`大众点评爬虫/main.py`](https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/main.py) (Review crawler), [`大众点评爬虫/CRAW_IP.py`](https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/CRAW_IP.py) (Proxy collector), [`文本分析挖掘/data.csv`](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/data.csv) (Review dataset)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Review collection"]
  node_crawler["Review crawler<br/>[main.py]"]
  node_proxy_crawler["Proxy collector<br/>[CRAW_IP.py]"]
  node_mysql[("MySQL persistence<br/>[mysqls.py]")]
  node_resume["Crawl progress<br/>[main.py]"]
end

subgraph group_analysis["Text analysis"]
  node_eda["Exploratory analysis<br/>[探索性数据分析.ipynb]"]
  node_preprocessing["Text preprocessing<br/>[文本挖掘&amp;情感分析.ipynb]"]
  node_segmentation["Chinese segmentation<br/>[文本挖掘&amp;情感分析.ipynb]"]
  node_stopwords["Stopword resource<br/>[stopwords.txt]"]
  node_wordcloud["Word-cloud analysis<br/>[文本挖掘&amp;情感分析.ipynb]"]
end

subgraph group_modeling["Sentiment modeling"]
  node_tfidf["TF-IDF features<br/>[文本挖掘&amp;情感分析.ipynb]"]
  node_imbalance["Class balancing<br/>[文本挖掘&amp;情感分析.ipynb]"]
  node_classifier["Naive Bayes model<br/>[文本挖掘&amp;情感分析.ipynb]"]
  node_evaluation["Model evaluation<br/>[文本挖掘&amp;情感分析.ipynb]"]
  node_prediction["Sentiment prediction<br/>[文本挖掘&amp;情感分析.ipynb]"]
end

subgraph group_inputs["Data and services"]
  node_dianping["Dianping reviews"]
  node_proxy_source["Xici proxy listings"]
  node_proxy_file["Proxy list<br/>[proxies.txt]"]
  node_review_data["Review dataset<br/>[data.csv]"]
end

node_operator(("Project operator"))

node_operator -->|"runs"| node_crawler
node_crawler -->|"requests reviews"| node_dianping
node_crawler -->|"saves reviews"| node_mysql
node_crawler -->|"reads and writes"| node_resume
node_operator -.->|"runs optionally"| node_proxy_crawler
node_proxy_crawler -->|"scrapes listings"| node_proxy_source
node_proxy_crawler -->|"writes proxies"| node_proxy_file
node_review_data -.->|"explores"| node_eda
node_review_data -.->|"preprocesses"| node_preprocessing
node_preprocessing -->|"segments text"| node_segmentation
node_preprocessing -.->|"uses"| node_stopwords
node_preprocessing -.->|"feeds analysis"| node_wordcloud
node_preprocessing -->|"builds features"| node_tfidf
node_tfidf -.->|"provides features"| node_imbalance
node_imbalance -.->|"trains with samples"| node_classifier
node_tfidf -->|"vectorizes input"| node_classifier
node_classifier -->|"evaluated by"| node_evaluation
node_classifier -->|"scores comments"| node_prediction

click node_crawler "https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/main.py"
click node_proxy_crawler "https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/CRAW_IP.py"
click node_mysql "https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/mysqls.py"
click node_resume "https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/main.py"
click node_proxy_file "https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/proxies.txt"
click node_review_data "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/data.csv"
click node_eda "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%8E%A2%E7%B4%A2%E6%80%A7%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90.ipynb"
click node_preprocessing "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"
click node_segmentation "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"
click node_stopwords "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/stopwords.txt"
click node_wordcloud "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"
click node_tfidf "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"
click node_imbalance "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"
click node_classifier "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"
click node_evaluation "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"
click node_prediction "https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawler,node_proxy_crawler,node_mysql,node_resume toneBlue
class node_eda,node_preprocessing,node_segmentation,node_stopwords,node_wordcloud toneAmber
class node_tfidf,node_imbalance,node_classifier,node_evaluation,node_prediction toneMint
class node_dianping,node_proxy_source,node_proxy_file,node_review_data toneRose
class node_operator toneIndigo
```

## Components

### Review collection

- [Review crawler](https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/main.py): `大众点评爬虫/main.py`
- [Proxy collector](https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/CRAW_IP.py): `大众点评爬虫/CRAW_IP.py`
- [MySQL persistence](https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/mysqls.py): `大众点评爬虫/mysqls.py`
- [Crawl progress](https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/main.py): `大众点评爬虫/main.py`

### Text analysis

- [Exploratory analysis](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%8E%A2%E7%B4%A2%E6%80%A7%E6%95%B0%E6%8D%AE%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/探索性数据分析.ipynb`
- [Text preprocessing](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`
- [Chinese segmentation](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`
- [Stopword resource](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/stopwords.txt): `文本分析挖掘/stopwords.txt`
- [Word-cloud analysis](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`

### Sentiment modeling

- [TF-IDF features](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`
- [Class balancing](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`
- [Naive Bayes model](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`
- [Model evaluation](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`
- [Sentiment prediction](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/%E6%96%87%E6%9C%AC%E6%8C%96%E6%8E%98%26%E6%83%85%E6%84%9F%E5%88%86%E6%9E%90.ipynb): `文本分析挖掘/文本挖掘&情感分析.ipynb`

### Data and services

- **Dianping reviews**
- **Xici proxy listings**
- [Proxy list](https://github.com/py-bin/dianping_textmining/blob/master/%E5%A4%A7%E4%BC%97%E7%82%B9%E8%AF%84%E7%88%AC%E8%99%AB/proxies.txt): `大众点评爬虫/proxies.txt`
- [Review dataset](https://github.com/py-bin/dianping_textmining/blob/master/%E6%96%87%E6%9C%AC%E5%88%86%E6%9E%90%E6%8C%96%E6%8E%98/data.csv): `文本分析挖掘/data.csv`

### Other components

- **Project operator**

## Connections

- Project operator → Review crawler: runs (evidence: `README.md`)
- Review crawler → Dianping reviews: requests reviews (evidence: `README.md`)
- Review crawler → MySQL persistence: saves reviews (evidence: `大众点评爬虫/main.py`)
- Review crawler → Crawl progress: reads and writes (evidence: `大众点评爬虫/main.py`)
- Project operator → Proxy collector: runs optionally (evidence: `README.md`)
- Proxy collector → Xici proxy listings: scrapes listings (evidence: `大众点评爬虫/CRAW_IP.py`)
- Proxy collector → Proxy list: writes proxies (evidence: `大众点评爬虫/CRAW_IP.py`)
- Review dataset → Exploratory analysis: explores (evidence: `README.md`)
- Review dataset → Text preprocessing: preprocesses (evidence: `README.md`)
- Text preprocessing → Chinese segmentation: segments text (evidence: `README.md`)
- Text preprocessing → Stopword resource: uses (evidence: `README.md`)
- Text preprocessing → Word-cloud analysis: feeds analysis (evidence: `README.md`)
- Text preprocessing → TF-IDF features: builds features (evidence: `README.md`)
- TF-IDF features → Class balancing: provides features (evidence: `README.md`)
- Class balancing → Naive Bayes model: trains with samples (evidence: `README.md`)
- TF-IDF features → Naive Bayes model: vectorizes input (evidence: `README.md`)
- Naive Bayes model → Model evaluation: evaluated by (evidence: `README.md`)
- Naive Bayes model → Sentiment prediction: scores comments (evidence: `README.md`)