How py-bin/dianping_textmining works
This repository is a Chinese Dianping review-mining project: it collects shop reviews, stores them, explores and preprocesses review text, then trains and evaluates sentiment classifiers. The README also describes word-cloud analysis, TF-IDF, imbalance handling, and sample prediction. The crawler-to-database call is visible in source; notebook implementations are not excerpted, so notebook relationships below reflect README-described workflow rather than inspected calls.
Main entry points: 文本挖掘&情感分析.ipynb Text preprocessing · main.py Review crawler · CRAW_IP.py Proxy collector · data.csv Review dataset
Components 19
Review collection
- Review crawler
大众点评爬虫/main.py - Proxy collector
大众点评爬虫/CRAW_IP.py - MySQL persistence
大众点评爬虫/mysqls.py - Crawl progress
大众点评爬虫/main.py
Text analysis
- Exploratory analysis
文本分析挖掘/探索性数据分析.ipynb - Text preprocessing
文本分析挖掘/文本挖掘&情感分析.ipynb - Chinese segmentation
文本分析挖掘/文本挖掘&情感分析.ipynb - Stopword resource
文本分析挖掘/stopwords.txt - Word-cloud analysis
文本分析挖掘/文本挖掘&情感分析.ipynb
Sentiment modeling
- TF-IDF features
文本分析挖掘/文本挖掘&情感分析.ipynb - Class balancing
文本分析挖掘/文本挖掘&情感分析.ipynb - Naive Bayes model
文本分析挖掘/文本挖掘&情感分析.ipynb - Model evaluation
文本分析挖掘/文本挖掘&情感分析.ipynb - Sentiment prediction
文本分析挖掘/文本挖掘&情感分析.ipynb
Data and services
- Dianping reviews
- Xici proxy listings
- Proxy list
大众点评爬虫/proxies.txt - Review dataset
文本分析挖掘/data.csv
Other components
- Project operator
Connections 18
- Project operator → Review crawler: runs
README.md - Review crawler → Dianping reviews: requests reviews
README.md - Review crawler → MySQL persistence: saves reviews
大众点评爬虫/main.py - Review crawler → Crawl progress: reads and writes
大众点评爬虫/main.py - Project operator → Proxy collector: runs optionally
README.md - Proxy collector → Xici proxy listings: scrapes listings
大众点评爬虫/CRAW_IP.py - Proxy collector → Proxy list: writes proxies
大众点评爬虫/CRAW_IP.py - Review dataset → Exploratory analysis: explores
README.md - Review dataset → Text preprocessing: preprocesses
README.md - Text preprocessing → Chinese segmentation: segments text
README.md - Text preprocessing → Stopword resource: uses
README.md - Text preprocessing → Word-cloud analysis: feeds analysis
README.md - Text preprocessing → TF-IDF features: builds features
README.md - TF-IDF features → Class balancing: provides features
README.md - Class balancing → Naive Bayes model: trains with samples
README.md - TF-IDF features → Naive Bayes model: vectorizes input
README.md - Naive Bayes model → Model evaluation: evaluated by
README.md - Naive Bayes model → Sentiment prediction: scores comments
README.md