# iphysresearch/top250movie_douban architecture

> Architecture diagram of the GitHub repository iphysresearch/top250movie_douban, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iphysresearch/top250movie_douban
- Repository: https://github.com/iphysresearch/top250movie_douban

## Overview

This repository is a Douban Top-250 data collection and analysis project. Its main workflow crawls movie records, then movie comments and commenter profiles; saved JSON is cleaned and used for descriptive statistics, visualizations, and sentiment/classification experiments. The README documents these analytical capabilities, but the notebook source was not sampled, so the analysis is represented at subsystem level rather than as inferred notebook internals. The crawler relationship to item definitions is visible in source; collection and analysis handoffs are documented in the README.

Main entry points: [`douban_movie/douban_movie/spiders`](https://github.com/iphysresearch/top250movie_douban/tree/master/douban_movie/douban_movie/spiders) (Comment crawlers), [`douban_movie/douban_movie/spiders/movie_item_spider.py`](https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/spiders/movie_item_spider.py) (Movie crawler), [`data_cleaning&feature_engineering/Filting.ipynb`](https://github.com/iphysresearch/top250movie_douban/blob/master/data_cleaning%26feature_engineering/Filting.ipynb) (Data cleaning), [`douban_movie/douban_movie/pipelines.py`](https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/pipelines.py) (Output pipelines)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collect["Data collection"]
  node_movie_spider["Movie crawler"]
  node_comment_spiders["Comment crawlers"]
  node_people_spiders["Profile crawlers"]
  node_item_schema["Scraped item schemas<br/>[items.py]"]
  node_json_outputs[("JSON datasets")]
end

subgraph group_prepare["Data preparation"]
  node_cleaning["Data cleaning<br/>[Filting.ipynb]"]
end

subgraph group_analyze["Analysis and modeling"]
  node_movie_stats["Movie statistics"]
  node_comment_analysis["Comment analysis"]
  node_profile_analysis["Profile analysis"]
  node_joint_analysis["Cross-dataset analysis"]
end

subgraph group_runtime["Crawler runtime"]
  node_dns_cache["DNS cache<br/>[dns_cache.py]"]
  node_scrapy_pipeline["Output pipelines<br/>[pipelines.py]"]
end

node_analyst(("Project user"))
node_douban{{"Douban site"}}

node_analyst -->|"starts crawl"| node_movie_spider
node_movie_spider -->|"requests pages"| node_douban
node_comment_spiders -->|"requests comments"| node_douban
node_people_spiders -->|"requests profiles"| node_douban
node_movie_spider -->|"creates items"| node_item_schema
node_comment_spiders -->|"creates items"| node_item_schema
node_people_spiders -->|"creates items"| node_item_schema
node_movie_spider -.->|"writes through"| node_scrapy_pipeline
node_comment_spiders -->|"writes through"| node_scrapy_pipeline
node_people_spiders -.->|"writes through"| node_scrapy_pipeline
node_scrapy_pipeline -->|"writes JSON"| node_json_outputs
node_movie_spider -->|"provides movie IDs"| node_comment_spiders
node_comment_spiders -->|"provides profile URLs"| node_people_spiders
node_json_outputs -->|"loads datasets"| node_cleaning
node_cleaning -->|"prepares features"| node_movie_stats
node_cleaning -->|"prepares features"| node_comment_analysis
node_cleaning -->|"prepares features"| node_profile_analysis
node_movie_stats -->|"contributes data"| node_joint_analysis
node_comment_analysis -->|"contributes data"| node_joint_analysis
node_profile_analysis -->|"contributes data"| node_joint_analysis
node_analyst -->|"reviews results"| node_movie_stats
node_analyst -->|"reviews results"| node_comment_analysis
node_comment_spiders -->|"uses cache"| node_dns_cache

click node_movie_spider "https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/spiders/movie_item_spider.py"
click node_comment_spiders "https://github.com/iphysresearch/top250movie_douban/tree/master/douban_movie/douban_movie/spiders"
click node_people_spiders "https://github.com/iphysresearch/top250movie_douban/tree/master/douban_movie/douban_movie/spiders"
click node_item_schema "https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/items.py"
click node_json_outputs "https://github.com/iphysresearch/top250movie_douban/tree/master/douban_movie/data"
click node_cleaning "https://github.com/iphysresearch/top250movie_douban/blob/master/data_cleaning%26feature_engineering/Filting.ipynb"
click node_movie_stats "https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering"
click node_comment_analysis "https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering"
click node_profile_analysis "https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering"
click node_joint_analysis "https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering"
click node_dns_cache "https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/dns_cache.py"
click node_scrapy_pipeline "https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/pipelines.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_movie_spider,node_comment_spiders,node_people_spiders,node_item_schema,node_json_outputs,node_analyst toneBlue
class node_cleaning toneAmber
class node_movie_stats,node_comment_analysis,node_profile_analysis,node_joint_analysis toneMint
class node_dns_cache,node_scrapy_pipeline toneRose
class node_douban toneIndigo
```

## Components

### Data collection

- [Movie crawler](https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/spiders/movie_item_spider.py): `douban_movie/douban_movie/spiders/movie_item_spider.py`
- [Comment crawlers](https://github.com/iphysresearch/top250movie_douban/tree/master/douban_movie/douban_movie/spiders): `douban_movie/douban_movie/spiders`
- [Profile crawlers](https://github.com/iphysresearch/top250movie_douban/tree/master/douban_movie/douban_movie/spiders): `douban_movie/douban_movie/spiders`
- [Scraped item schemas](https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/items.py): `douban_movie/douban_movie/items.py`
- [JSON datasets](https://github.com/iphysresearch/top250movie_douban/tree/master/douban_movie/data): `douban_movie/data`

### Data preparation

- [Data cleaning](https://github.com/iphysresearch/top250movie_douban/blob/master/data_cleaning%26feature_engineering/Filting.ipynb): `data_cleaning&feature_engineering/Filting.ipynb`

### Analysis and modeling

- [Movie statistics](https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering): `data_cleaning&feature_engineering`
- [Comment analysis](https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering): `data_cleaning&feature_engineering`
- [Profile analysis](https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering): `data_cleaning&feature_engineering`
- [Cross-dataset analysis](https://github.com/iphysresearch/top250movie_douban/tree/master/data_cleaning%26feature_engineering): `data_cleaning&feature_engineering`

### Crawler runtime

- [DNS cache](https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/dns_cache.py): `douban_movie/douban_movie/dns_cache.py`
- [Output pipelines](https://github.com/iphysresearch/top250movie_douban/blob/master/douban_movie/douban_movie/pipelines.py): `douban_movie/douban_movie/pipelines.py`

### Other components

- **Project user**
- **Douban site**

## Connections

- Project user → Movie crawler: starts crawl (evidence: `readme.md`)
- Movie crawler → Douban site: requests pages (evidence: `douban_movie/douban_movie/spiders/movie_item_spider.py`)
- Comment crawlers → Douban site: requests comments (evidence: `douban_movie/douban_movie/spiders/movie_comment_spider20.py`)
- Profile crawlers → Douban site: requests profiles (evidence: `readme.md`)
- Movie crawler → Scraped item schemas: creates items (evidence: `douban_movie/douban_movie/spiders/movie_item_spider.py`)
- Comment crawlers → Scraped item schemas: creates items (evidence: `douban_movie/douban_movie/spiders/movie_comment_spider20.py`)
- Profile crawlers → Scraped item schemas: creates items (evidence: `douban_movie/douban_movie/spiders/movie_Xpeople.py`)
- Movie crawler → Output pipelines: writes through (evidence: `readme.md`)
- Comment crawlers → Output pipelines: writes through (evidence: `douban_movie/douban_movie/spiders/movie_comment_spider20.py`)
- Profile crawlers → Output pipelines: writes through (evidence: `readme.md`)
- Output pipelines → JSON datasets: writes JSON (evidence: `readme.md`)
- Movie crawler → Comment crawlers: provides movie IDs (evidence: `readme.md`)
- Comment crawlers → Profile crawlers: provides profile URLs (evidence: `readme.md`)
- JSON datasets → Data cleaning: loads datasets (evidence: `readme.md`)
- Data cleaning → Movie statistics: prepares features (evidence: `readme.md`)
- Data cleaning → Comment analysis: prepares features (evidence: `readme.md`)
- Data cleaning → Profile analysis: prepares features (evidence: `readme.md`)
- Movie statistics → Cross-dataset analysis: contributes data (evidence: `readme.md`)
- Comment analysis → Cross-dataset analysis: contributes data (evidence: `readme.md`)
- Profile analysis → Cross-dataset analysis: contributes data (evidence: `readme.md`)
- Project user → Movie statistics: reviews results (evidence: `readme.md`)
- Project user → Comment analysis: reviews results (evidence: `readme.md`)
- Comment crawlers → DNS cache: uses cache (evidence: `douban_movie/douban_movie/spiders/movie_comment_spider20.py`)
