# xingclover/douban_comments_tracker architecture

> Architecture diagram of the GitHub repository xingclover/douban_comments_tracker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xingclover/douban_comments_tracker
- Repository: https://github.com/xingclover/douban_comments_tracker

## Overview

This is a drama-monitoring and opinion-analysis pipeline: scheduled and operator-triggered collectors gather Douban comments, full discussion threads, user/community data, and iQIYI/Weibo signals; PostgreSQL stores the results; LLM and streaming stages enrich or flag them; and a Streamlit console exposes task execution and analytical exploration. The graph preserves the README’s documented behavioral profiling, trend analysis, and community-analysis capabilities, though their complete implementations were not sampled. Edges use README statements or shown source references; unsampled subsystems are represented at their exact enclosing directories.

Main entry points: [`streamlit_console/app.py`](https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/app.py) (Streamlit Console), [`scraper/douban_post_scraper.py`](https://github.com/xingclover/douban_comments_tracker/blob/main/scraper/douban_post_scraper.py) (Thread Scraper), [`streamlit_console/pages/3_Query.py`](https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/pages/3_Query.py) (Query Console), [`db.py`](https://github.com/xingclover/douban_comments_tracker/blob/main/db.py) (DB Connection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection &amp; Scheduling"]
  node_airflow["Airflow DAGs"]
  node_comment_scraper["Comment Scraper"]
  node_thread_scraper["Thread Scraper"]
  node_topic_discovery["Topic Discovery"]
  node_user_profiling["User &amp; Groups"]
  node_iqiyi_collection["iQIYI Signals"]
  node_weibo_collection["Weibo Signals<br/>[weibo_scraper.py]"]
end

subgraph group_storage["Storage &amp; Enrichment"]
  node_postgres[("PostgreSQL")]
  node_db_access["DB Connection<br/>[db.py]"]
  node_llm_labeling["Post Labeling"]
  node_topic_classification["Comment Classification"]
  node_ollama{{"Ollama"}}
end

subgraph group_signals["Detection &amp; Analysis"]
  node_kafka_producer["Comment Producer"]
  node_kafka_consumer["Low-Rating Consumer"]
  node_analysis["Trend &amp; Behavior Analysis"]
end

subgraph group_console["Operations &amp; Exploration"]
  node_console_app["Streamlit Console<br/>[app.py]"]
  node_console_tasks["Task Registry &amp; Runner<br/>[tasks.py]"]
  node_console_queries["Query Console<br/>[3_Query.py]"]
  node_query_registry["SQL Query Registry<br/>[query_registry.py]"]
  node_console_browse["Browse &amp; Views<br/>[2_Browse.py]"]
end

node_operator(("Analyst / Operator"))

node_airflow -->|"schedules collection"| node_comment_scraper
node_airflow -->|"schedules collection"| node_thread_scraper
node_operator -->|"operates"| node_console_app
node_console_app -->|"offers task execution"| node_console_tasks
node_console_tasks -->|"dispatches task"| node_thread_scraper
node_comment_scraper -->|"stores comments"| node_postgres
node_thread_scraper -->|"stores posts and replies"| node_postgres
node_topic_discovery -.->|"selects discussion topics"| node_thread_scraper
node_user_profiling -->|"stores profile data"| node_postgres
node_iqiyi_collection -->|"stores heat signals"| node_postgres
node_weibo_collection -->|"stores platform data"| node_postgres
node_db_access -->|"opens connection"| node_postgres
node_llm_labeling -->|"reads and writes"| node_db_access
node_llm_labeling -->|"requests labels"| node_ollama
node_topic_classification -->|"classifies comments"| node_ollama
node_kafka_producer -->|"streams low ratings"| node_kafka_consumer
node_kafka_producer -->|"reads comments"| node_postgres
node_kafka_consumer -->|"records detections"| node_postgres
node_postgres -->|"supplies analytical data"| node_analysis
node_console_app -->|"exposes SQL querying"| node_console_queries
node_console_queries -->|"loads query definitions"| node_query_registry
node_console_queries -->|"executes queries"| node_db_access
node_console_app -->|"offers data exploration"| node_console_browse
node_console_browse -->|"browses analytical views"| node_postgres

click node_airflow "https://github.com/xingclover/douban_comments_tracker/tree/main/airflow/dags"
click node_comment_scraper "https://github.com/xingclover/douban_comments_tracker/blob/main/scraper/douban_comments_scraper.py"
click node_thread_scraper "https://github.com/xingclover/douban_comments_tracker/blob/main/scraper/douban_post_scraper.py"
click node_topic_discovery "https://github.com/xingclover/douban_comments_tracker/blob/main/scraper/douban_topics_statistic_scraper.py"
click node_user_profiling "https://github.com/xingclover/douban_comments_tracker/tree/main/scraper"
click node_iqiyi_collection "https://github.com/xingclover/douban_comments_tracker/tree/main/iqiyi"
click node_weibo_collection "https://github.com/xingclover/douban_comments_tracker/blob/main/multi_platform_scraper/weibo_scraper.py"
click node_db_access "https://github.com/xingclover/douban_comments_tracker/blob/main/db.py"
click node_llm_labeling "https://github.com/xingclover/douban_comments_tracker/blob/main/analysisLLM/post_labeling/label_posts_with_llm.py"
click node_topic_classification "https://github.com/xingclover/douban_comments_tracker/blob/main/analysisLLM/topics/classify_comment_topic.py"
click node_kafka_producer "https://github.com/xingclover/douban_comments_tracker/blob/main/kafka_pipeline/producer_send_comments.py"
click node_kafka_consumer "https://github.com/xingclover/douban_comments_tracker/blob/main/kafka_pipeline/consumer_low_rating.py"
click node_analysis "https://github.com/xingclover/douban_comments_tracker/tree/main/analysis"
click node_console_app "https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/app.py"
click node_console_tasks "https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/core/tasks.py"
click node_console_queries "https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/pages/3_Query.py"
click node_query_registry "https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/core/query_registry.py"
click node_console_browse "https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/pages/2_Browse.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow,node_comment_scraper,node_thread_scraper,node_topic_discovery,node_user_profiling,node_iqiyi_collection,node_weibo_collection toneBlue
class node_postgres,node_db_access,node_llm_labeling,node_topic_classification,node_ollama toneAmber
class node_kafka_producer,node_kafka_consumer,node_analysis toneMint
class node_console_app,node_console_tasks,node_console_queries,node_query_registry,node_console_browse toneRose
class node_operator toneIndigo
```

## Components

### Collection & Scheduling

- [Airflow DAGs](https://github.com/xingclover/douban_comments_tracker/tree/main/airflow/dags): `airflow/dags`
- [Comment Scraper](https://github.com/xingclover/douban_comments_tracker/blob/main/scraper/douban_comments_scraper.py): `scraper/douban_comments_scraper.py`
- [Thread Scraper](https://github.com/xingclover/douban_comments_tracker/blob/main/scraper/douban_post_scraper.py): `scraper/douban_post_scraper.py`
- [Topic Discovery](https://github.com/xingclover/douban_comments_tracker/blob/main/scraper/douban_topics_statistic_scraper.py): `scraper/douban_topics_statistic_scraper.py`
- [User & Groups](https://github.com/xingclover/douban_comments_tracker/tree/main/scraper): `scraper`
- [iQIYI Signals](https://github.com/xingclover/douban_comments_tracker/tree/main/iqiyi): `iqiyi`
- [Weibo Signals](https://github.com/xingclover/douban_comments_tracker/blob/main/multi_platform_scraper/weibo_scraper.py): `multi_platform_scraper/weibo_scraper.py`

### Storage & Enrichment

- **PostgreSQL**
- [DB Connection](https://github.com/xingclover/douban_comments_tracker/blob/main/db.py): `db.py`
- [Post Labeling](https://github.com/xingclover/douban_comments_tracker/blob/main/analysisLLM/post_labeling/label_posts_with_llm.py): `analysisLLM/post_labeling/label_posts_with_llm.py`
- [Comment Classification](https://github.com/xingclover/douban_comments_tracker/blob/main/analysisLLM/topics/classify_comment_topic.py): `analysisLLM/topics/classify_comment_topic.py`
- **Ollama**

### Detection & Analysis

- [Comment Producer](https://github.com/xingclover/douban_comments_tracker/blob/main/kafka_pipeline/producer_send_comments.py): `kafka_pipeline/producer_send_comments.py`
- [Low-Rating Consumer](https://github.com/xingclover/douban_comments_tracker/blob/main/kafka_pipeline/consumer_low_rating.py): `kafka_pipeline/consumer_low_rating.py`
- [Trend & Behavior Analysis](https://github.com/xingclover/douban_comments_tracker/tree/main/analysis): `analysis`

### Operations & Exploration

- [Streamlit Console](https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/app.py): `streamlit_console/app.py`
- [Task Registry & Runner](https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/core/tasks.py): `streamlit_console/core/tasks.py`
- [Query Console](https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/pages/3_Query.py): `streamlit_console/pages/3_Query.py`
- [SQL Query Registry](https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/core/query_registry.py): `streamlit_console/core/query_registry.py`
- [Browse & Views](https://github.com/xingclover/douban_comments_tracker/blob/main/streamlit_console/pages/2_Browse.py): `streamlit_console/pages/2_Browse.py`

### Other components

- **Analyst / Operator**

## Connections

- Airflow DAGs → Comment Scraper: schedules collection (evidence: `README.md`)
- Airflow DAGs → Thread Scraper: schedules collection (evidence: `README.md`)
- Analyst / Operator → Streamlit Console: operates (evidence: `README.md`)
- Streamlit Console → Task Registry & Runner: offers task execution
- Task Registry & Runner → Thread Scraper: dispatches task (evidence: `streamlit_console/core/tasks.py`)
- Comment Scraper → PostgreSQL: stores comments (evidence: `README.md`)
- Thread Scraper → PostgreSQL: stores posts and replies (evidence: `scraper/douban_post_scraper.py`)
- Topic Discovery → Thread Scraper: selects discussion topics
- User & Groups → PostgreSQL: stores profile data (evidence: `README.md`)
- iQIYI Signals → PostgreSQL: stores heat signals
- Weibo Signals → PostgreSQL: stores platform data
- DB Connection → PostgreSQL: opens connection (evidence: `db.py`)
- Post Labeling → DB Connection: reads and writes (evidence: `analysisLLM/post_labeling/label_posts_with_llm.py`)
- Post Labeling → Ollama: requests labels (evidence: `analysisLLM/post_labeling/label_posts_with_llm.py`)
- Comment Classification → Ollama: classifies comments
- Comment Producer → Low-Rating Consumer: streams low ratings (evidence: `README.md`)
- Comment Producer → PostgreSQL: reads comments
- Low-Rating Consumer → PostgreSQL: records detections
- PostgreSQL → Trend & Behavior Analysis: supplies analytical data (evidence: `README.md`)
- Streamlit Console → Query Console: exposes SQL querying (evidence: `README.md`)
- Query Console → SQL Query Registry: loads query definitions (evidence: `streamlit_console/pages/3_Query.py`)
- Query Console → DB Connection: executes queries (evidence: `streamlit_console/pages/3_Query.py`)
- Streamlit Console → Browse & Views: offers data exploration (evidence: `streamlit_console/pages/2_Browse.py`)
- Browse & Views → PostgreSQL: browses analytical views (evidence: `README.md`)
