How hquan07/news_flow works
NewsPulse collects Vietnamese news and social posts, enriches them through Kafka and Spark streaming, and serves analytics through a FastAPI-backed dashboard. The graph preserves distinct crawl, raw persistence, stream processing, analytical storage, and serving stages, alongside documented quality, alerting, and backfill capabilities. Paths for some documented subsystems are enclosing directories because their detailed implementation was not sampled; those connections follow the README architecture rather than inspected call sites.
Main entry points: newspulse_crawler Scrapy Crawlers · warehouse ClickHouse Warehouse · streaming_job.py Spark Streaming Job · dags Airflow Schedules
Components 25
Collection
- Scrapy Crawlers
crawlers/newspulse_crawler - Raw Object Storage
- MongoDB Raw Data
Stream Processing
- Kafka
infrastructure/kafka_utils - Spark Streaming Job
spark/streaming/streaming_job.py - Kafka Consumer
spark/streaming/kafka_consumer.py - Vietnamese NLP
spark/processing - NLP Writers
spark/streaming/nlp_writers.py - Stream Sink Writers
spark/streaming/sink_writers.py
Analytics Serving
- ClickHouse Warehouse
data_platform/warehouse - FastAPI Application
api/main.py - Analytics Services
api/services/analytics.py - Analytics and SSE Routes
api/routers - Authentication Routes
api/routers/auth.py - User Data Store
api/database.py
Quality Operations
- Airflow Schedules
data_platform/dags - Data Quality Checks
data_platform/data_quality - Anomaly and Crisis Alerts
data_platform/dags/anomaly_alerting_dag.py - Telegram Reports
infrastructure/monitoring/telegram_alert.py - Historical Backfills
data_platform/warehouse - Health Monitoring
infrastructure/monitoring
Dashboard
- Next.js Dashboard
frontend/src/app/page.tsx - API Client
frontend/src/lib/api.ts
Other components
- Dashboard User
- News and Social Sources
Connections 28
- News and Social Sources → Scrapy Crawlers: crawl content
- Airflow Schedules → Scrapy Crawlers: schedule crawls
- Scrapy Crawlers → Raw Object Storage: store raw HTML
- Scrapy Crawlers → MongoDB Raw Data: persist items
- Scrapy Crawlers → Kafka: publish articles
- Kafka → Kafka Consumer: stream records
- Spark Streaming Job → Kafka Consumer: run consumer
- Spark Streaming Job → Vietnamese NLP: apply enrichment
- Spark Streaming Job → NLP Writers: write NLP results
- Spark Streaming Job → Stream Sink Writers: write stream data
- NLP Writers → ClickHouse Warehouse: store enrichments
- Stream Sink Writers → ClickHouse Warehouse: store processed data
- Airflow Schedules → Data Quality Checks: schedule validation
- Airflow Schedules → Anomaly and Crisis Alerts: schedule alerting
- Anomaly and Crisis Alerts → ClickHouse Warehouse: query metrics
- Anomaly and Crisis Alerts → Telegram Reports: send notifications
- Airflow Schedules → Telegram Reports: send daily reports
- Historical Backfills → ClickHouse Warehouse: load history
- Next.js Dashboard → API Client: request API data
- API Client → FastAPI Application: call endpoints
- FastAPI Application → Analytics and SSE Routes: register routes
- Analytics and SSE Routes → Analytics Services: request analytics
- Analytics Services → ClickHouse Warehouse: query warehouse
- Analytics and SSE Routes → Authentication Routes: dispatch auth
- Authentication Routes → User Data Store: read and write users
- Dashboard User → Next.js Dashboard: use dashboard
- Health Monitoring → MongoDB Raw Data: check health
- Health Monitoring → Kafka: check health