# ysoseri1224/visual-data-analysis-of-popular-video-platforms architecture

> Architecture diagram of the GitHub repository ysoseri1224/visual-data-analysis-of-popular-video-platforms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms
- Repository: https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms

## Overview

MediaCrawler collects platform content for structured storage; the FastAPI backend provides conversational NL2SQL analysis and visualization services, surfaced through a Next.js interface. The graph preserves the documented platform, query, connection, account, and monitoring capabilities. Several crawler and backend subsystems are represented at their source paths because their implementation was not sampled; those links are included only where the README documents the relationship.

Main entry points: [`backend/app/api/endpoints`](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/backend/app/api/endpoints) (API routes), [`MediaCrawler/main.py`](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/main.py) (Crawler entry), [`MediaCrawler/media_platform`](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/MediaCrawler/media_platform) (Platform crawlers), [`backend/app/db/mongodb.py`](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/db/mongodb.py) (MongoDB state)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Data collection"]
  node_crawler_entry["Crawler entry<br/>[main.py]"]
  node_crawler_base["Crawler framework<br/>[base_crawler.py]"]
  node_crawler_platforms["Platform crawlers"]
  node_crawler_clients["Platform clients"]
  node_crawler_proxy["Proxy pool<br/>[proxy_ip_pool.py]"]
  node_crawler_utils["Crawler utilities<br/>[crawler_util.py]"]
end

subgraph group_analysis["Analysis backend"]
  node_api["API routes"]
  node_query["SQL execution<br/>[query.py]"]
  node_nl2sql["NL2SQL service<br/>[nl2sql_service.py]"]
  node_deepseek["DeepSeek service"]
  node_conversation["Conversation services"]
  node_connections["Data connections"]
  node_visualization["Visualization routes<br/>[visualization.py]"]
  node_accounts["Accounts and access<br/>[auth.py]"]
  node_monitoring["Logs and monitoring<br/>[monitoring.py]"]
end

subgraph group_persistence["Data and state"]
  node_crawler_storage["Crawler stores"]
  node_mysql[("MySQL data<br/>[async_db.py]")]
  node_mongo[("MongoDB state<br/>[mongodb.py]")]
end

subgraph group_presentation["Web application"]
  node_web_app["Analysis interface<br/>[AIchat.tsx]"]
  node_web_services["Frontend services<br/>[apiService.ts]"]
end

node_crawler_user(("Crawler operator"))
node_web_user(("Analysis user"))
node_external_mysql[("External MySQL")]
node_deepseek_api["DeepSeek API"]

node_crawler_user -->|"starts crawl"| node_crawler_entry
node_crawler_entry -->|"starts crawler"| node_crawler_base
node_crawler_entry -->|"selects platform"| node_crawler_platforms
node_crawler_platforms -.->|"fetches content"| node_crawler_clients
node_crawler_clients -.->|"uses proxies"| node_crawler_proxy
node_crawler_clients -.->|"uses utilities"| node_crawler_utils
node_crawler_platforms -.->|"stores results"| node_crawler_storage
node_crawler_storage -.->|"writes data"| node_mysql
node_web_user -->|"asks questions"| node_web_app
node_web_app -.->|"uses services"| node_web_services
node_web_services -.->|"calls API"| node_api
node_api -.->|"routes queries"| node_nl2sql
node_nl2sql -->|"generates SQL"| node_deepseek
node_deepseek -.->|"requests completion"| node_deepseek_api
node_api -.->|"executes SQL"| node_query
node_query -->|"queries data"| node_external_mysql
node_api -.->|"manages history"| node_conversation
node_api -.->|"manages connections"| node_connections
node_connections -->|"connects database"| node_external_mysql
node_api -.->|"generates charts"| node_visualization
node_api -.->|"handles access"| node_accounts
node_api -.->|"serves monitoring"| node_monitoring
node_conversation -.->|"stores history"| node_mongo
node_accounts -.->|"stores accounts"| node_mongo
node_crawler_entry -->|"initializes database"| node_mysql
node_api -->|"connects at startup"| node_mongo

click node_crawler_entry "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/main.py"
click node_crawler_base "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/base/base_crawler.py"
click node_crawler_platforms "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/MediaCrawler/media_platform"
click node_crawler_clients "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/MediaCrawler/media_platform"
click node_crawler_proxy "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/proxy/proxy_ip_pool.py"
click node_crawler_utils "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/tools/crawler_util.py"
click node_crawler_storage "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/MediaCrawler/store"
click node_mysql "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/async_db.py"
click node_web_app "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/frontend/src/pages/conversations/AIchat.tsx"
click node_web_services "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/frontend/src/services/apiService.ts"
click node_api "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/backend/app/api/endpoints"
click node_query "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/query.py"
click node_nl2sql "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/nl2sql_service.py"
click node_deepseek "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/deepseek_service.py"
click node_conversation "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/conversation_service.py"
click node_connections "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/connection_service.py"
click node_visualization "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/visualization.py"
click node_accounts "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/auth.py"
click node_monitoring "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/monitoring.py"
click node_mongo "https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/db/mongodb.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawler_entry,node_crawler_base,node_crawler_platforms,node_crawler_clients,node_crawler_proxy,node_crawler_utils,node_web_user toneBlue
class node_api,node_query,node_nl2sql,node_deepseek,node_conversation,node_connections,node_visualization,node_accounts,node_monitoring,node_external_mysql toneAmber
class node_crawler_storage,node_mysql,node_mongo,node_deepseek_api toneMint
class node_web_app,node_web_services toneRose
class node_crawler_user toneIndigo
```

## Components

### Data collection

- [Crawler entry](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/main.py): `MediaCrawler/main.py`
- [Crawler framework](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/base/base_crawler.py): `MediaCrawler/base/base_crawler.py`
- [Platform crawlers](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/MediaCrawler/media_platform): `MediaCrawler/media_platform`
- [Platform clients](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/MediaCrawler/media_platform): `MediaCrawler/media_platform`
- [Proxy pool](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/proxy/proxy_ip_pool.py): `MediaCrawler/proxy/proxy_ip_pool.py`
- [Crawler utilities](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/tools/crawler_util.py): `MediaCrawler/tools/crawler_util.py`

### Analysis backend

- [API routes](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/backend/app/api/endpoints): `backend/app/api/endpoints`
- [SQL execution](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/query.py): `backend/app/api/endpoints/query.py`
- [NL2SQL service](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/nl2sql_service.py): `backend/app/services/nl2sql_service.py`
- [DeepSeek service](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/deepseek_service.py): `backend/app/services/deepseek_service.py`
- [Conversation services](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/conversation_service.py): `backend/app/services/conversation_service.py`
- [Data connections](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/services/connection_service.py): `backend/app/services/connection_service.py`
- [Visualization routes](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/visualization.py): `backend/app/api/endpoints/visualization.py`
- [Accounts and access](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/auth.py): `backend/app/api/endpoints/auth.py`
- [Logs and monitoring](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/api/endpoints/monitoring.py): `backend/app/api/endpoints/monitoring.py`

### Data and state

- [Crawler stores](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/tree/main/MediaCrawler/store): `MediaCrawler/store`
- [MySQL data](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/MediaCrawler/async_db.py): `MediaCrawler/async_db.py`
- [MongoDB state](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/backend/app/db/mongodb.py): `backend/app/db/mongodb.py`

### Web application

- [Analysis interface](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/frontend/src/pages/conversations/AIchat.tsx): `frontend/src/pages/conversations/AIchat.tsx`
- [Frontend services](https://github.com/ysoseri1224/visual-data-analysis-of-popular-video-platforms/blob/main/frontend/src/services/apiService.ts): `frontend/src/services/apiService.ts`

### Other components

- **Crawler operator**
- **Analysis user**
- **External MySQL**
- **DeepSeek API**

## Connections

- Crawler operator → Crawler entry: starts crawl (evidence: `MediaCrawler/main.py`)
- Crawler entry → Crawler framework: starts crawler (evidence: `MediaCrawler/main.py`)
- Crawler entry → Platform crawlers: selects platform (evidence: `MediaCrawler/main.py`)
- Platform crawlers → Platform clients: fetches content
- Platform clients → Proxy pool: uses proxies (evidence: `MediaCrawler/media_platform/tieba/client.py`)
- Platform clients → Crawler utilities: uses utilities
- Platform crawlers → Crawler stores: stores results (evidence: `README.md`)
- Crawler stores → MySQL data: writes data (evidence: `README.md`)
- Analysis user → Analysis interface: asks questions (evidence: `README.md`)
- Analysis interface → Frontend services: uses services
- Frontend services → API routes: calls API (evidence: `README.md`)
- API routes → NL2SQL service: routes queries (evidence: `README.md`)
- NL2SQL service → DeepSeek service: generates SQL (evidence: `README.md`)
- DeepSeek service → DeepSeek API: requests completion (evidence: `README.md`)
- API routes → SQL execution: executes SQL (evidence: `README.md`)
- SQL execution → External MySQL: queries data (evidence: `backend/app/api/endpoints/query.py`)
- API routes → Conversation services: manages history (evidence: `README.md`)
- API routes → Data connections: manages connections (evidence: `README.md`)
- Data connections → External MySQL: connects database (evidence: `README.md`)
- API routes → Visualization routes: generates charts (evidence: `README.md`)
- API routes → Accounts and access: handles access (evidence: `README.md`)
- API routes → Logs and monitoring: serves monitoring (evidence: `README.md`)
- Conversation services → MongoDB state: stores history (evidence: `README.md`)
- Accounts and access → MongoDB state: stores accounts (evidence: `README.md`)
- Crawler entry → MySQL data: initializes database (evidence: `MediaCrawler/main.py`)
- API routes → MongoDB state: connects at startup
