# hasnocool/tradingview-script-downloader architecture

> Architecture diagram of the GitHub repository hasnocool/tradingview-script-downloader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hasnocool/tradingview-script-downloader
- Repository: https://github.com/hasnocool/tradingview-script-downloader

## Overview

##### Main Components

a) Input Layer:
- Multiple fetch scripts (fetch.py, fetch_editors-picks.py, fetch_indicators.py, etc.)
- Configuration files (requirements.txt, docker-compose.yaml)

b) Processing Layer:
- Concurrent Download Manager
- Selenium WebDriver Controller
- Aiohttp Client Manager
- Data Processing Components (clean-data.py)

c) Storage Layer:
- File System Storage
- Backup System (backup-data.sh)

##### Component Relationships

**[External Systems]**

- v
**[Input Layer] --> [Processing Layer] --> [Storage Layer]**

##### Specific Implementation Details

a) External Systems (Top Layer):
- Both HTTP/HTTPS connections

b) Input Layer (Upper Middle):
- Configuration management
- Docker container boundaries (reference Dockerfile and docker-compose.yaml)

c) Processing Layer (Middle):
- Two parallel processing paths:
- Selenium WebDriver (Headless Firefox)
- Aiohttp Client Pool
- BeautifulSoup4 parsing component

d) Storage Layer (Bottom):
- File system storage component
- Backup system component
- Data flow for cleaned and raw data

##### Important Flows to Highlight

- Concurrent request handling
- Data cleaning pipeline
- Backup processes
- Docker containerization boundaries

##### Technical Details

Annotations for:
- Python version
- Key dependencies (Selenium, aiohttp, BeautifulSoup4)
- Concurrent processing limits
- File storage patterns

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    TV[TradingView Website]:::external

    %% Input Layer
    subgraph Input
        direction TB
        FetchScripts{"Fetch Scripts"}
        FetchEditors["fetch_editors-picks.py"]
        FetchIndicators["fetch_indicators.py"]
        FetchMore["fetch_more.py"]
        FetchScriptsFile["fetch_scripts.py"]
        FetchStrategies["fetch_strategies.py"]
        FetchMain["fetch.py"]
        
        Config{"Configuration"}
        Docker["Dockerfile"]
        DockerCompose["docker-compose.yaml"]
        Requirements["requirements.txt"]
    end

    %% Processing Layer
    subgraph Processing
        direction TB
        ConcurrentMgr["Concurrent Download Manager"]
        Selenium["Selenium WebDriver<br/>(Headless Firefox)"]
        Aiohttp["Aiohttp Client Pool"]
        DataProcess["Data Processing"]
        CleanData["clean-data.py"]
        CountData["count.py"]
    end

    %% Storage Layer
    subgraph Storage
        direction TB
        FileSystem["File System Storage"]
        FileContents[".file_contents_output"]
        RepoSummary[".repo_summary"]
        LatestContents["latest_file_contents.txt"]
        ConsolidatedSummary["consolidated_summary.txt"]
        Backup["backup-data.sh"]
    end

    %% Docker Infrastructure
    subgraph Docker
        direction TB
        DockerBuild["docker-build.sh"]
        DockerRun["docker-run.sh"]
    end

    %% Relationships
    TV --> FetchScripts
    FetchScripts --> |Data Flow| ConcurrentMgr
    ConcurrentMgr --> Selenium
    ConcurrentMgr --> Aiohttp
    Selenium --> DataProcess
    Aiohttp --> DataProcess
    DataProcess --> FileSystem
    FileSystem --> Backup

    %% Component Details
    FetchScripts --> FetchEditors
    FetchScripts --> FetchIndicators
    FetchScripts --> FetchMore
    FetchScripts --> FetchScriptsFile
    FetchScripts --> FetchStrategies
    FetchScripts --> FetchMain
    
    Config --> Docker
    Config --> DockerCompose
    Config --> Requirements
    
    DataProcess --> CleanData
    DataProcess --> CountData
    
    FileSystem --> FileContents
    FileSystem --> RepoSummary
    FileContents --> LatestContents
    RepoSummary --> ConsolidatedSummary

    Docker --> DockerBuild
    Docker --> DockerRun

    %% Click Events
    click FetchEditors "https://github.com/hasnocool/tradingview-script-downloader/blob/main/fetch_editors-picks.py"
    click FetchIndicators "https://github.com/hasnocool/tradingview-script-downloader/blob/main/fetch_indicators.py"
    click FetchMore "https://github.com/hasnocool/tradingview-script-downloader/blob/main/fetch_more.py"
    click FetchScriptsFile "https://github.com/hasnocool/tradingview-script-downloader/blob/main/fetch_scripts.py"
    click FetchStrategies "https://github.com/hasnocool/tradingview-script-downloader/blob/main/fetch_strategies.py"
    click FetchMain "https://github.com/hasnocool/tradingview-script-downloader/blob/main/fetch.py"
    click Docker "https://github.com/hasnocool/tradingview-script-downloader/tree/main/Dockerfile"
    click DockerCompose "https://github.com/hasnocool/tradingview-script-downloader/blob/main/docker-compose.yaml"
    click Requirements "https://github.com/hasnocool/tradingview-script-downloader/blob/main/requirements.txt"
    click CleanData "https://github.com/hasnocool/tradingview-script-downloader/blob/main/clean-data.py"
    click CountData "https://github.com/hasnocool/tradingview-script-downloader/blob/main/count.py"
    click FileContents "https://github.com/hasnocool/tradingview-script-downloader/blob/main/.file_contents_output"
    click RepoSummary "https://github.com/hasnocool/tradingview-script-downloader/blob/main/.repo_summary"
    click Backup "https://github.com/hasnocool/tradingview-script-downloader/blob/main/backup-data.sh"
    click DockerBuild "https://github.com/hasnocool/tradingview-script-downloader/blob/main/docker-build.sh"
    click DockerRun "https://github.com/hasnocool/tradingview-script-downloader/blob/main/docker-run.sh"
    click LatestContents "https://github.com/hasnocool/tradingview-script-downloader/blob/main/.file_contents_output/latest_file_contents.txt"
    click ConsolidatedSummary "https://github.com/hasnocool/tradingview-script-downloader/blob/main/.repo_summary/consolidated_summary.txt"

    %% Styling
    classDef external fill:#68B5E1
    classDef input fill:#95D891
    classDef processing fill:#FFE873
    classDef storage fill:#B888F8
    classDef docker fill:#D3D3D3
    
    class TV external
    class FetchScripts,FetchEditors,FetchIndicators,FetchMore,FetchScriptsFile,FetchStrategies,FetchMain,Config,Docker,DockerCompose,Requirements input
    class ConcurrentMgr,Selenium,Aiohttp,DataProcess,CleanData,CountData processing
    class FileSystem,FileContents,RepoSummary,LatestContents,ConsolidatedSummary,Backup storage
    class DockerBuild,DockerRun docker
```
