# mnwato/tradingview-scraper architecture

> Architecture diagram of the GitHub repository mnwato/tradingview-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mnwato/tradingview-scraper
- Repository: https://github.com/mnwato/tradingview-scraper

## Overview

##### Main Components (Top Level)

- It to three main subsystems:
- Data Scraping Engine
- Real-Time Streaming Engine
- Export System

b) Data Sources (Left Side):
- TradingView WebSocket Connection
- TradingView Web Pages

c) Output Handlers (Right Side):
- JSON Export Handler
- CSV Export Handler
- In-Memory Results Handler

##### Detailed Component Breakdown

- Ideas Scraper Module
- News Scraper Module
- Technical Indicators Module
- Real-Time Price Module

b) Data Storage (Bottom):
- Configuration Storage
- Areas (areas.json)
- Exchanges (exchanges.txt)
- Indicators (indicators.txt)
- Languages (languages.json)
- Timeframes (timeframes.json)

##### Data Flow Representation

- WebSocket connections for real-time data
- HTTP requests for web scraping
- Internal data processing flows
- Export data flows

##### Component Details

For each major component, include:
- Input/Output formats
- Key methods/functions
- Data transformation steps

- Blue: Core components
- Green: Data input sources
- Orange: Processing modules
- Purple: Export/Output handlers
- Gray: Configuration/Static data

Specific Implementation Guidelines:

**[TradingView WebSocket] → [Real-Time Stream Handler]**
- ↓
- [Price Stream Module]
- ↓
- [Data Formatter]

##### Scraping Pipeline
**[TradingView Web Pages] → [Scraping Engine]**
- ↓
- [Ideas Module] ← → [Central Coordinator] → [News Module]
- ↓
- [Technical Indicators Module]

##### Configuration Management
**[Configuration Store]**
- ↓
**[Data Validators] → [Service Initializers]**

##### Export System Flow
**[Data Processor] → [Export Manager]**
- ↙           ↘
- [JSON Writer]    [CSV Writer]

Important Considerations:

##### Modular nature of the system
- Clear boundaries between different scraping modules
- Extension points for future modules

##### Real-time capabilities
- Separate WebSocket handling from traditional scraping
- Buffering/streaming mechanisms

##### Error handling and retry mechanisms
- Error handling pathways
- Validation checkpoints

##### Configuration management
- How static data (JSONs and TXTs) is used
- Configuration loading paths

Remember to include version information and note that this is an actively developed project with ongoing feature additions as indicated in the README's To-Do list.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Core Components
    Core["TradingView Scraper Core"]:::core
    click Core "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/tradingview_scraper.py"

    %% Data Sources
    TVWeb["TradingView Web Pages"]:::external
    TVSocket["TradingView WebSocket"]:::external

    %% Main Processing Components
    ScrapeEngine["Data Scraping Engine"]:::processor
    StreamEngine["Real-Time Streaming Engine"]:::processor
    click StreamEngine "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/symbols/stream/__init__.py"
    ExportSystem["Export System"]:::processor

    %% Scraping Modules
    IdeasMod["Ideas Scraper"]:::module
    click IdeasMod "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/symbols/ideas.py"
    NewsMod["News Scraper"]:::module
    click NewsMod "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/symbols/news.py"
    TechMod["Technical Indicators"]:::module
    click TechMod "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/symbols/technicals.py"
    UtilsMod["Utilities"]:::module
    click UtilsMod "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/symbols/utils.py"

    %% Streaming Components
    PriceStream["Price Stream Module"]:::stream
    click PriceStream "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/symbols/stream/price.py"
    DataFormat["Data Formatter"]:::processor

    %% Symbol Management
    SymbolCore["Symbol Management"]:::core
    click SymbolCore "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/symbols/__init__.py"

    %% Export Handlers
    JsonEx["JSON Export Handler"]:::output
    CsvEx["CSV Export Handler"]:::output
    MemEx["In-Memory Results"]:::output

    %% Configuration Storage
    subgraph Config ["Configuration Storage"]
        Areas["Areas Config"]:::config
        click Areas "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/data/areas.json"
        Exchanges["Exchanges Config"]:::config
        click Exchanges "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/data/exchanges.txt"
        Indicators["Indicators Config"]:::config
        click Indicators "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/data/indicators.txt"
        Languages["Languages Config"]:::config
        click Languages "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/data/languages.json"
        Timeframes["Timeframes Config"]:::config
        click Timeframes "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/data/timeframes.json"
        NewsProviders["News Providers Config"]:::config
        click NewsProviders "https://github.com/mnwato/tradingview-scraper/blob/main/tradingview_scraper/data/news_providers.txt"
    end

    %% Connections
    Core --> ScrapeEngine
    Core --> StreamEngine
    Core --> ExportSystem
    Core --> SymbolCore

    TVWeb --> ScrapeEngine
    TVSocket --> StreamEngine

    ScrapeEngine --> IdeasMod
    ScrapeEngine --> NewsMod
    ScrapeEngine --> TechMod
    ScrapeEngine --> UtilsMod

    StreamEngine --> PriceStream
    PriceStream --> DataFormat

    ExportSystem --> JsonEx
    ExportSystem --> CsvEx
    ExportSystem --> MemEx

    Config --> Core

    %% Styles
    classDef core fill:#2374ab,stroke:#000,stroke-width:2px,color:#fff
    classDef external fill:#95a5a6,stroke:#000,stroke-width:2px
    classDef processor fill:#e67e22,stroke:#000,stroke-width:2px
    classDef module fill:#27ae60,stroke:#000,stroke-width:2px
    classDef stream fill:#8e44ad,stroke:#000,stroke-width:2px
    classDef output fill:#e74c3c,stroke:#000,stroke-width:2px
    classDef config fill:#95a5a6,stroke:#000,stroke-width:1px

    %% Legend
    subgraph Legend
        L1["Core Components"]:::core
        L2["External Sources"]:::external
        L3["Processors"]:::processor
        L4["Modules"]:::module
        L5["Stream Components"]:::stream
        L6["Output Handlers"]:::output
        L7["Configuration"]:::config
    end
```
