How xingclover/douban_comments_tracker works
This is a drama-monitoring and opinion-analysis pipeline: scheduled and operator-triggered collectors gather Douban comments, full discussion threads, user/community data, and iQIYI/Weibo signals; PostgreSQL stores the results; LLM and streaming stages enrich or flag them; and a Streamlit console exposes task execution and analytical exploration. The graph preserves the README’s documented behavioral profiling, trend analysis, and community-analysis capabilities, though their complete implementations were not sampled. Edges use README statements or shown source references; unsampled subsystems are represented at their exact enclosing directories.
Main entry points: app.py Streamlit Console · douban_post_scraper.py Thread Scraper · 3_Query.py Query Console · db.py DB Connection
Components 21
Collection & Scheduling
- Airflow DAGs
airflow/dags - Comment Scraper
scraper/douban_comments_scraper.py - Thread Scraper
scraper/douban_post_scraper.py - Topic Discovery
scraper/douban_topics_statistic_scraper.py - User & Groups
scraper - iQIYI Signals
iqiyi - Weibo Signals
multi_platform_scraper/weibo_scraper.py
Storage & Enrichment
- PostgreSQL
- DB Connection
db.py - Post Labeling
analysisLLM/post_labeling/label_posts_with_llm.py - Comment Classification
analysisLLM/topics/classify_comment_topic.py - Ollama
Detection & Analysis
- Comment Producer
kafka_pipeline/producer_send_comments.py - Low-Rating Consumer
kafka_pipeline/consumer_low_rating.py - Trend & Behavior Analysis
analysis
Operations & Exploration
- Streamlit Console
streamlit_console/app.py - Task Registry & Runner
streamlit_console/core/tasks.py - Query Console
streamlit_console/pages/3_Query.py - SQL Query Registry
streamlit_console/core/query_registry.py - Browse & Views
streamlit_console/pages/2_Browse.py
Other components
- Analyst / Operator
Connections 24
- Airflow DAGs → Comment Scraper: schedules collection
README.md - Airflow DAGs → Thread Scraper: schedules collection
README.md - Analyst / Operator → Streamlit Console: operates
README.md - Streamlit Console → Task Registry & Runner: offers task execution
- Task Registry & Runner → Thread Scraper: dispatches task
streamlit_console/core/tasks.py - Comment Scraper → PostgreSQL: stores comments
README.md - Thread Scraper → PostgreSQL: stores posts and replies
scraper/douban_post_scraper.py - Topic Discovery → Thread Scraper: selects discussion topics
- User & Groups → PostgreSQL: stores profile data
README.md - iQIYI Signals → PostgreSQL: stores heat signals
- Weibo Signals → PostgreSQL: stores platform data
- DB Connection → PostgreSQL: opens connection
db.py - Post Labeling → DB Connection: reads and writes
analysisLLM/post_labeling/label_posts_with_llm.py - Post Labeling → Ollama: requests labels
analysisLLM/post_labeling/label_posts_with_llm.py - Comment Classification → Ollama: classifies comments
- Comment Producer → Low-Rating Consumer: streams low ratings
README.md - Comment Producer → PostgreSQL: reads comments
- Low-Rating Consumer → PostgreSQL: records detections
- PostgreSQL → Trend & Behavior Analysis: supplies analytical data
README.md - Streamlit Console → Query Console: exposes SQL querying
README.md - Query Console → SQL Query Registry: loads query definitions
streamlit_console/pages/3_Query.py - Query Console → DB Connection: executes queries
streamlit_console/pages/3_Query.py - Streamlit Console → Browse & Views: offers data exploration
streamlit_console/pages/2_Browse.py - Browse & Views → PostgreSQL: browses analytical views
README.md