How mnwato/tradingview-scraper works
Main Components (Top Level)
• It to three main subsystems:
• Data Scraping Engine
• Real-Time Streaming Engine
• Export System
b) Data Sources (Left Side):
• TradingView WebSocket Connection
• TradingView Web Pages
c) Output Handlers (Right Side):
• JSON Export Handler
• CSV Export Handler
• In-Memory Results Handler
Detailed Component Breakdown
• Ideas Scraper Module
• News Scraper Module
• Technical Indicators Module
• Real-Time Price Module
b) Data Storage (Bottom):
• Configuration Storage
• Areas (areas.json)
• Exchanges (exchanges.txt)
• Indicators (indicators.txt)
• Languages (languages.json)
• Timeframes (timeframes.json)
Data Flow Representation
• WebSocket connections for real-time data
• HTTP requests for web scraping
• Internal data processing flows
• Export data flows
Component Details
For each major component, include:
• Input/Output formats
• Key methods/functions
• Data transformation steps
• Blue: Core components
• Green: Data input sources
• Orange: Processing modules
• Purple: Export/Output handlers
• Gray: Configuration/Static data
Specific Implementation Guidelines:
[TradingView WebSocket] → [Real-Time Stream Handler]
• ↓
• [Price Stream Module]
• ↓
• [Data Formatter]
Scraping Pipeline
[TradingView Web Pages] → [Scraping Engine]
• ↓
• [Ideas Module] ← → [Central Coordinator] → [News Module]
• ↓
• [Technical Indicators Module]
Configuration Management
[Configuration Store]
• ↓
[Data Validators] → [Service Initializers]
Export System Flow
[Data Processor] → [Export Manager]
• ↙ ↘
• [JSON Writer] [CSV Writer]
Important Considerations:
Modular nature of the system
• Clear boundaries between different scraping modules
• Extension points for future modules
Real-time capabilities
• Separate WebSocket handling from traditional scraping
• Buffering/streaming mechanisms
Error handling and retry mechanisms
• Error handling pathways
• Validation checkpoints
Configuration management
• How static data (JSONs and TXTs) is used
• Configuration loading paths
Remember to include version information and note that this is an actively developed project with ongoing feature additions as indicated in the README's To-Do list.