# hquan07/news_flow architecture

> Architecture diagram of the GitHub repository hquan07/news_flow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hquan07/news_flow
- Repository: https://github.com/hquan07/news_flow

## Overview

NewsPulse collects Vietnamese news and social posts, enriches them through Kafka and Spark streaming, and serves analytics through a FastAPI-backed dashboard. The graph preserves distinct crawl, raw persistence, stream processing, analytical storage, and serving stages, alongside documented quality, alerting, and backfill capabilities. Paths for some documented subsystems are enclosing directories because their detailed implementation was not sampled; those connections follow the README architecture rather than inspected call sites.

Main entry points: [`crawlers/newspulse_crawler`](https://github.com/hquan07/news_flow/tree/main/crawlers/newspulse_crawler) (Scrapy Crawlers), [`data_platform/warehouse`](https://github.com/hquan07/news_flow/tree/main/data_platform/warehouse) (ClickHouse Warehouse), [`spark/streaming/streaming_job.py`](https://github.com/hquan07/news_flow/blob/main/spark/streaming/streaming_job.py) (Spark Streaming Job), [`data_platform/dags`](https://github.com/hquan07/news_flow/tree/main/data_platform/dags) (Airflow Schedules)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection"]
  node_crawlers["Scrapy Crawlers"]
  node_rawstore[("Raw Object Storage")]
  node_mongodb[("MongoDB Raw Data")]
end

subgraph group_processing["Stream Processing"]
  node_kafka["Kafka"]
  node_sparkjob["Spark Streaming Job<br/>[streaming_job.py]"]
  node_kafkaconsumer["Kafka Consumer<br/>[kafka_consumer.py]"]
  node_nlp["Vietnamese NLP"]
  node_nlpwriters["NLP Writers<br/>[nlp_writers.py]"]
  node_sinkwriters["Stream Sink Writers<br/>[sink_writers.py]"]
end

subgraph group_serving["Analytics Serving"]
  node_clickhouse[("ClickHouse Warehouse")]
  node_api["FastAPI Application<br/>[main.py]"]
  node_analytics["Analytics Services<br/>[analytics.py]"]
  node_routes["Analytics and SSE Routes"]
  node_auth["Authentication Routes<br/>[auth.py]"]
  node_mongoauth[("User Data Store<br/>[database.py]")]
end

subgraph group_operations["Quality Operations"]
  node_orchestrator["Airflow Schedules"]
  node_quality["Data Quality Checks"]
  node_alerts["Anomaly and Crisis Alerts"]
  node_telegram["Telegram Reports<br/>[telegram_alert.py]"]
  node_backfill["Historical Backfills"]
  node_monitoring["Health Monitoring"]
end

subgraph group_dashboard["Dashboard"]
  node_frontend["Next.js Dashboard<br/>[page.tsx]"]
  node_frontendapi["API Client<br/>[api.ts]"]
end

node_user(("Dashboard User"))
node_sources(("News and Social Sources"))

node_sources -->|"crawl content"| node_crawlers
node_orchestrator -.->|"schedule crawls"| node_crawlers
node_crawlers -->|"store raw HTML"| node_rawstore
node_crawlers -->|"persist items"| node_mongodb
node_crawlers -->|"publish articles"| node_kafka
node_kafka -->|"stream records"| node_kafkaconsumer
node_sparkjob -->|"run consumer"| node_kafkaconsumer
node_sparkjob -->|"apply enrichment"| node_nlp
node_sparkjob -->|"write NLP results"| node_nlpwriters
node_sparkjob -->|"write stream data"| node_sinkwriters
node_nlpwriters -->|"store enrichments"| node_clickhouse
node_sinkwriters -->|"store processed data"| node_clickhouse
node_orchestrator -.->|"schedule validation"| node_quality
node_orchestrator -.->|"schedule alerting"| node_alerts
node_alerts -->|"query metrics"| node_clickhouse
node_alerts -->|"send notifications"| node_telegram
node_orchestrator -.->|"send daily reports"| node_telegram
node_backfill -->|"load history"| node_clickhouse
node_frontend -->|"request API data"| node_frontendapi
node_frontendapi -->|"call endpoints"| node_api
node_api -->|"register routes"| node_routes
node_routes -->|"request analytics"| node_analytics
node_analytics -->|"query warehouse"| node_clickhouse
node_routes -->|"dispatch auth"| node_auth
node_auth -->|"read and write users"| node_mongoauth
node_user -->|"use dashboard"| node_frontend
node_monitoring -.->|"check health"| node_mongodb
node_monitoring -.->|"check health"| node_kafka

click node_crawlers "https://github.com/hquan07/news_flow/tree/main/crawlers/newspulse_crawler"
click node_orchestrator "https://github.com/hquan07/news_flow/tree/main/data_platform/dags"
click node_kafka "https://github.com/hquan07/news_flow/tree/main/infrastructure/kafka_utils"
click node_sparkjob "https://github.com/hquan07/news_flow/blob/main/spark/streaming/streaming_job.py"
click node_kafkaconsumer "https://github.com/hquan07/news_flow/blob/main/spark/streaming/kafka_consumer.py"
click node_nlp "https://github.com/hquan07/news_flow/tree/main/spark/processing"
click node_nlpwriters "https://github.com/hquan07/news_flow/blob/main/spark/streaming/nlp_writers.py"
click node_sinkwriters "https://github.com/hquan07/news_flow/blob/main/spark/streaming/sink_writers.py"
click node_clickhouse "https://github.com/hquan07/news_flow/tree/main/data_platform/warehouse"
click node_api "https://github.com/hquan07/news_flow/blob/main/api/main.py"
click node_analytics "https://github.com/hquan07/news_flow/blob/main/api/services/analytics.py"
click node_routes "https://github.com/hquan07/news_flow/tree/main/api/routers"
click node_auth "https://github.com/hquan07/news_flow/blob/main/api/routers/auth.py"
click node_mongoauth "https://github.com/hquan07/news_flow/blob/main/api/database.py"
click node_frontend "https://github.com/hquan07/news_flow/blob/main/frontend/src/app/page.tsx"
click node_frontendapi "https://github.com/hquan07/news_flow/blob/main/frontend/src/lib/api.ts"
click node_quality "https://github.com/hquan07/news_flow/tree/main/data_platform/data_quality"
click node_alerts "https://github.com/hquan07/news_flow/blob/main/data_platform/dags/anomaly_alerting_dag.py"
click node_telegram "https://github.com/hquan07/news_flow/blob/main/infrastructure/monitoring/telegram_alert.py"
click node_backfill "https://github.com/hquan07/news_flow/tree/main/data_platform/warehouse"
click node_monitoring "https://github.com/hquan07/news_flow/tree/main/infrastructure/monitoring"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawlers,node_rawstore,node_mongodb,node_user toneBlue
class node_kafka,node_sparkjob,node_kafkaconsumer,node_nlp,node_nlpwriters,node_sinkwriters toneAmber
class node_clickhouse,node_api,node_analytics,node_routes,node_auth,node_mongoauth toneMint
class node_orchestrator,node_quality,node_alerts,node_telegram,node_backfill,node_monitoring toneRose
class node_frontend,node_frontendapi,node_sources toneIndigo
```

## Components

### Collection

- [Scrapy Crawlers](https://github.com/hquan07/news_flow/tree/main/crawlers/newspulse_crawler): `crawlers/newspulse_crawler`
- **Raw Object Storage**
- **MongoDB Raw Data**

### Stream Processing

- [Kafka](https://github.com/hquan07/news_flow/tree/main/infrastructure/kafka_utils): `infrastructure/kafka_utils`
- [Spark Streaming Job](https://github.com/hquan07/news_flow/blob/main/spark/streaming/streaming_job.py): `spark/streaming/streaming_job.py`
- [Kafka Consumer](https://github.com/hquan07/news_flow/blob/main/spark/streaming/kafka_consumer.py): `spark/streaming/kafka_consumer.py`
- [Vietnamese NLP](https://github.com/hquan07/news_flow/tree/main/spark/processing): `spark/processing`
- [NLP Writers](https://github.com/hquan07/news_flow/blob/main/spark/streaming/nlp_writers.py): `spark/streaming/nlp_writers.py`
- [Stream Sink Writers](https://github.com/hquan07/news_flow/blob/main/spark/streaming/sink_writers.py): `spark/streaming/sink_writers.py`

### Analytics Serving

- [ClickHouse Warehouse](https://github.com/hquan07/news_flow/tree/main/data_platform/warehouse): `data_platform/warehouse`
- [FastAPI Application](https://github.com/hquan07/news_flow/blob/main/api/main.py): `api/main.py`
- [Analytics Services](https://github.com/hquan07/news_flow/blob/main/api/services/analytics.py): `api/services/analytics.py`
- [Analytics and SSE Routes](https://github.com/hquan07/news_flow/tree/main/api/routers): `api/routers`
- [Authentication Routes](https://github.com/hquan07/news_flow/blob/main/api/routers/auth.py): `api/routers/auth.py`
- [User Data Store](https://github.com/hquan07/news_flow/blob/main/api/database.py): `api/database.py`

### Quality Operations

- [Airflow Schedules](https://github.com/hquan07/news_flow/tree/main/data_platform/dags): `data_platform/dags`
- [Data Quality Checks](https://github.com/hquan07/news_flow/tree/main/data_platform/data_quality): `data_platform/data_quality`
- [Anomaly and Crisis Alerts](https://github.com/hquan07/news_flow/blob/main/data_platform/dags/anomaly_alerting_dag.py): `data_platform/dags/anomaly_alerting_dag.py`
- [Telegram Reports](https://github.com/hquan07/news_flow/blob/main/infrastructure/monitoring/telegram_alert.py): `infrastructure/monitoring/telegram_alert.py`
- [Historical Backfills](https://github.com/hquan07/news_flow/tree/main/data_platform/warehouse): `data_platform/warehouse`
- [Health Monitoring](https://github.com/hquan07/news_flow/tree/main/infrastructure/monitoring): `infrastructure/monitoring`

### Dashboard

- [Next.js Dashboard](https://github.com/hquan07/news_flow/blob/main/frontend/src/app/page.tsx): `frontend/src/app/page.tsx`
- [API Client](https://github.com/hquan07/news_flow/blob/main/frontend/src/lib/api.ts): `frontend/src/lib/api.ts`

### Other components

- **Dashboard User**
- **News and Social Sources**

## Connections

- News and Social Sources → Scrapy Crawlers: crawl content
- Airflow Schedules → Scrapy Crawlers: schedule crawls
- Scrapy Crawlers → Raw Object Storage: store raw HTML
- Scrapy Crawlers → MongoDB Raw Data: persist items
- Scrapy Crawlers → Kafka: publish articles
- Kafka → Kafka Consumer: stream records
- Spark Streaming Job → Kafka Consumer: run consumer
- Spark Streaming Job → Vietnamese NLP: apply enrichment
- Spark Streaming Job → NLP Writers: write NLP results
- Spark Streaming Job → Stream Sink Writers: write stream data
- NLP Writers → ClickHouse Warehouse: store enrichments
- Stream Sink Writers → ClickHouse Warehouse: store processed data
- Airflow Schedules → Data Quality Checks: schedule validation
- Airflow Schedules → Anomaly and Crisis Alerts: schedule alerting
- Anomaly and Crisis Alerts → ClickHouse Warehouse: query metrics
- Anomaly and Crisis Alerts → Telegram Reports: send notifications
- Airflow Schedules → Telegram Reports: send daily reports
- Historical Backfills → ClickHouse Warehouse: load history
- Next.js Dashboard → API Client: request API data
- API Client → FastAPI Application: call endpoints
- FastAPI Application → Analytics and SSE Routes: register routes
- Analytics and SSE Routes → Analytics Services: request analytics
- Analytics Services → ClickHouse Warehouse: query warehouse
- Analytics and SSE Routes → Authentication Routes: dispatch auth
- Authentication Routes → User Data Store: read and write users
- Dashboard User → Next.js Dashboard: use dashboard
- Health Monitoring → MongoDB Raw Data: check health
- Health Monitoring → Kafka: check health