How iphysresearch/top250movie_douban works
This repository is a Douban Top-250 data collection and analysis project. Its main workflow crawls movie records, then movie comments and commenter profiles; saved JSON is cleaned and used for descriptive statistics, visualizations, and sentiment/classification experiments. The README documents these analytical capabilities, but the notebook source was not sampled, so the analysis is represented at subsystem level rather than as inferred notebook internals. The crawler relationship to item definitions is visible in source; collection and analysis handoffs are documented in the README.
Main entry points: spiders Comment crawlers · movie_item_spider.py Movie crawler · Filting.ipynb Data cleaning · pipelines.py Output pipelines
Components 14
Data collection
- Movie crawler
douban_movie/douban_movie/spiders/movie_item_spider.py - Comment crawlers
douban_movie/douban_movie/spiders - Profile crawlers
douban_movie/douban_movie/spiders - Scraped item schemas
douban_movie/douban_movie/items.py - JSON datasets
douban_movie/data
Data preparation
- Data cleaning
data_cleaning&feature_engineering/Filting.ipynb
Analysis and modeling
- Movie statistics
data_cleaning&feature_engineering - Comment analysis
data_cleaning&feature_engineering - Profile analysis
data_cleaning&feature_engineering - Cross-dataset analysis
data_cleaning&feature_engineering
Crawler runtime
- DNS cache
douban_movie/douban_movie/dns_cache.py - Output pipelines
douban_movie/douban_movie/pipelines.py
Other components
- Project user
- Douban site
Connections 23
- Project user → Movie crawler: starts crawl
readme.md - Movie crawler → Douban site: requests pages
douban_movie/douban_movie/spiders/movie_item_spider.py - Comment crawlers → Douban site: requests comments
douban_movie/douban_movie/spiders/movie_comment_spider20.py - Profile crawlers → Douban site: requests profiles
readme.md - Movie crawler → Scraped item schemas: creates items
douban_movie/douban_movie/spiders/movie_item_spider.py - Comment crawlers → Scraped item schemas: creates items
douban_movie/douban_movie/spiders/movie_comment_spider20.py - Profile crawlers → Scraped item schemas: creates items
douban_movie/douban_movie/spiders/movie_Xpeople.py - Movie crawler → Output pipelines: writes through
readme.md - Comment crawlers → Output pipelines: writes through
douban_movie/douban_movie/spiders/movie_comment_spider20.py - Profile crawlers → Output pipelines: writes through
readme.md - Output pipelines → JSON datasets: writes JSON
readme.md - Movie crawler → Comment crawlers: provides movie IDs
readme.md - Comment crawlers → Profile crawlers: provides profile URLs
readme.md - JSON datasets → Data cleaning: loads datasets
readme.md - Data cleaning → Movie statistics: prepares features
readme.md - Data cleaning → Comment analysis: prepares features
readme.md - Data cleaning → Profile analysis: prepares features
readme.md - Movie statistics → Cross-dataset analysis: contributes data
readme.md - Comment analysis → Cross-dataset analysis: contributes data
readme.md - Profile analysis → Cross-dataset analysis: contributes data
readme.md - Project user → Movie statistics: reviews results
readme.md - Project user → Comment analysis: reviews results
readme.md - Comment crawlers → DNS cache: uses cache
douban_movie/douban_movie/spiders/movie_comment_spider20.py