How ysoseri1224/visual-data-analysis-of-popular-video-platforms works
MediaCrawler collects platform content for structured storage; the FastAPI backend provides conversational NL2SQL analysis and visualization services, surfaced through a Next.js interface. The graph preserves the documented platform, query, connection, account, and monitoring capabilities. Several crawler and backend subsystems are represented at their source paths because their implementation was not sampled; those links are included only where the README documents the relationship.
Main entry points: endpoints API routes · main.py Crawler entry · media_platform Platform crawlers · mongodb.py MongoDB state
Components 24
Data collection
- Crawler entry
MediaCrawler/main.py - Crawler framework
MediaCrawler/base/base_crawler.py - Platform crawlers
MediaCrawler/media_platform - Platform clients
MediaCrawler/media_platform - Proxy pool
MediaCrawler/proxy/proxy_ip_pool.py - Crawler utilities
MediaCrawler/tools/crawler_util.py
Analysis backend
- API routes
backend/app/api/endpoints - SQL execution
backend/app/api/endpoints/query.py - NL2SQL service
backend/app/services/nl2sql_service.py - DeepSeek service
backend/app/services/deepseek_service.py - Conversation services
backend/app/services/conversation_service.py - Data connections
backend/app/services/connection_service.py - Visualization routes
backend/app/api/endpoints/visualization.py - Accounts and access
backend/app/api/endpoints/auth.py - Logs and monitoring
backend/app/api/endpoints/monitoring.py
Data and state
- Crawler stores
MediaCrawler/store - MySQL data
MediaCrawler/async_db.py - MongoDB state
backend/app/db/mongodb.py
Web application
- Analysis interface
frontend/src/pages/conversations/AIchat.tsx - Frontend services
frontend/src/services/apiService.ts
Other components
- Crawler operator
- Analysis user
- External MySQL
- DeepSeek API
Connections 26
- Crawler operator → Crawler entry: starts crawl
MediaCrawler/main.py - Crawler entry → Crawler framework: starts crawler
MediaCrawler/main.py - Crawler entry → Platform crawlers: selects platform
MediaCrawler/main.py - Platform crawlers → Platform clients: fetches content
- Platform clients → Proxy pool: uses proxies
MediaCrawler/media_platform/tieba/client.py - Platform clients → Crawler utilities: uses utilities
- Platform crawlers → Crawler stores: stores results
README.md - Crawler stores → MySQL data: writes data
README.md - Analysis user → Analysis interface: asks questions
README.md - Analysis interface → Frontend services: uses services
- Frontend services → API routes: calls API
README.md - API routes → NL2SQL service: routes queries
README.md - NL2SQL service → DeepSeek service: generates SQL
README.md - DeepSeek service → DeepSeek API: requests completion
README.md - API routes → SQL execution: executes SQL
README.md - SQL execution → External MySQL: queries data
backend/app/api/endpoints/query.py - API routes → Conversation services: manages history
README.md - API routes → Data connections: manages connections
README.md - Data connections → External MySQL: connects database
README.md - API routes → Visualization routes: generates charts
README.md - API routes → Accounts and access: handles access
README.md - API routes → Logs and monitoring: serves monitoring
README.md - Conversation services → MongoDB state: stores history
README.md - Accounts and access → MongoDB state: stores accounts
README.md - Crawler entry → MySQL data: initializes database
MediaCrawler/main.py - API routes → MongoDB state: connects at startup