# hellotherematey/tedata architecture

> Architecture diagram of the GitHub repository hellotherematey/tedata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hellotherematey/tedata
- Repository: https://github.com/hellotherematey/tedata

## Overview

##### And Define the Main Components
- Client/Entry Points:
- Command Line Interface (CLI): Triggered when running “python -m tedata …” (see __main__.py).
- Interactive/API Usage: Importing the “tedata” package in scripts or notebooks.
- Scraping Engine:
- TE_Scraper Class (in src/tedata/scraper.py and scrape_chart.py): Contains the scraping workflow (loading pages, scraping x-axis and y-axis information, scaling data, and extracting metadata).
- Search Module (src/tedata/search.py): Provides “search_TE” functionality to search Trading Economics and fetch a table of possible datasets.
- Utility and Support Modules:
- Base Functions (src/tedata/base.py)
- Logger Setup (src/tedata/logger_setup.py) for debugging and error reporting.
- Additional JS and utility scripts (e.g., custom_datespan.js, check_highcharts.js) that supplement the scraping process.
- Data Processing and Export:
- Integration with Pandas for handling the scraped data.
- Plotting (via Plotly and optionally Kaleido for export).
- Export functions (saving to Excel, CSV, or HDF5) within the TE_Scraper object.
- Documentation & Testing Components:
- Docs directory: Contains HTML documentation (including guides and screenshots).
- Tests directory: Includes notebooks and automated tests to validate functionality.

##### External and Dependency Interactions
- External Dependencies:
- Selenium Webdriver (specifically configured with Firefox).
- BeautifulSoup4 for HTML parsing.
- Plotly for interactive charting.
- Pandas for data structures and exporting formats.
- Optional libraries such as openpyxl, kaleido, and nbformat.
- External Systems:
- Trading Economics website from which data is scraped.
- Web Browser (Firefox) used and controlled by Selenium.

##### Lay Out the Data Flow and Component Interactions
- User invokes either the CLI or imports tedata in a notebook/script.
- In the Core API:
- When a URL or search query is provided, the application routes it to the Search Module or directly to the Scraper Module.
- From “search_TE”: The search query is sent, results (dataframes of potential indicators) are returned.
- When scraping is initiated (either via “scrape_chart” or direct TE_Scraper calls), the following steps occur:
a. The Web Driver is initialized (or reused) and navigates to the Trading Economics URL.
b. Selenium operates the browser (Firefox) to capture the chart element.
○ Data scraping via SVG path extraction, scaling pixel values using y-axis values.
○ Alternative methods (tooltips, mixed, highcharts_api) indicated as options.
d. Data Processing: The raw pixel/index data is converted to a Pandas Series along with metadata.
e. Plotting & Export: The processed data may be plotted (with Plotly) and exported to Excel/CSV/HDF5.
- Error handling and logging:
- Logger Setup module receives events and potential error messages, routed back to the central logger.

##### Architectural Patterns and Design Principles
- Modularity: Each functional block (search, scraping, logging, utilities) is separated in the src/tedata folder.
- Single Responsibility: Each module tackles a specific aspect – data fetching, parsing, transforming, and plotting.
- Extensibility: The design (with multiple scraping methods as options) suggests an extensible architecture for adding more methods (e.g., support for Chrome in the future).
- Dependency Injection: Options for reusing or reinitializing drivers via the TE_Scraper’s parameters help maintain flexibility.

##### Final Checklist
- Clearly label every major component with its purpose.
- Distinguish between different layers (API layer, scraping engine layer, data processing/visualization layer, external dependencies).
- Incorporate notes on how browser automation is critical to initializing and driving the scraping engine.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client/Entry Points
    subgraph "Client/Entry Points"
        CLI("CLI / Entry Point"):::client
        API("Interactive/API Usage"):::client
    end

    %% User Input
    Input("Input Query/URL"):::core

    %% Arrows from Client/Entry Points to Input
    CLI -->|"triggers"| Input
    API -->|"sends request"| Input

    %% Core Processing
    subgraph "Core Processing"
        Search("Search Module (search_TE)"):::core
        TE("TE_Scraper"):::core
        SC("Scrape Chart Process"):::core
    end

    %% Routing from Input to Core Processing
    Input -->|"search"| Search
    Input -->|"scrape"| TE
    TE -->|"chartExtraction"| SC

    %% Data Processing & Export
    subgraph "Data Processing & Export"
        PD("Pandas Data Processing"):::data
        PC("Plotly Charting & Export"):::data
    end

    TE -->|"processData"| PD
    TE -->|"plotExport"| PC

    %% Utility and Support Modules
    subgraph "Utility and Support Modules"
        Base("Base Functions"):::utility
        Logger("Logger Setup Module"):::utility
        JS1("custom_datespan.js"):::utility
        JS2("check_highcharts.js"):::utility
        JS3("firstLastDates.js"):::utility
        JS4("init_tooltips.js"):::utility
        JS5("latest_points.js"):::utility
    end

    %% Logging relations
    Search -->|"logs"| Logger
    TE -->|"logs"| Logger
    SC -->|"logs"| Logger
    %% Utility usage (dotted line)
    TE ---|"utilizes"| Base
    TE ---|"utilizes"| JS1
    TE ---|"utilizes"| JS2
    TE ---|"utilizes"| JS3
    TE ---|"utilizes"| JS4
    TE ---|"utilizes"| JS5

    %% External Dependencies & Systems
    subgraph "External Dependencies & Systems"
        SD("Selenium Webdriver (Firefox)"):::external
        BS("BeautifulSoup4"):::external
        FB("Firefox Browser"):::external
        TEW("Trading Economics Website"):::external
    end

    %% External interactions
    TE -->|"initDriver"| SD
    TE ---|"parsesHTML"| BS
    SD -->|"controls"| FB
    FB -->|"accesses"| TEW

    %% Project Support
    subgraph "Project Support"
        Docs("Documentation"):::support
        Tests("Testing Suite"):::support
    end

    %% Styles
    classDef client fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px;
    classDef core fill:#bbdefb,stroke:#1565c0,stroke-width:2px;
    classDef data fill:#fff9c4,stroke:#f9a825,stroke-width:2px;
    classDef utility fill:#d1c4e9,stroke:#6a1b9a,stroke-width:2px;
    classDef external fill:#ffe082,stroke:#ff6f00,stroke-width:2px;
    classDef support fill:#ffccbc,stroke:#d84315,stroke-width:2px;

    %% Click Events
    click CLI "https://github.com/hellotherematey/tedata/blob/main/src/tedata/__main__.py"
    click API "https://github.com/hellotherematey/tedata/blob/main/src/tedata/__init__.py"
    click TE "https://github.com/hellotherematey/tedata/blob/main/src/tedata/scraper.py"
    click SC "https://github.com/hellotherematey/tedata/blob/main/src/tedata/scrape_chart.py"
    click Search "https://github.com/hellotherematey/tedata/blob/main/src/tedata/search.py"
    click Base "https://github.com/hellotherematey/tedata/blob/main/src/tedata/base.py"
    click Logger "https://github.com/hellotherematey/tedata/blob/main/src/tedata/logger_setup.py"
    click JS1 "https://github.com/hellotherematey/tedata/blob/main/src/tedata/custom_datespan.js"
    click JS2 "https://github.com/hellotherematey/tedata/blob/main/src/tedata/check_highcharts.js"
    click JS3 "https://github.com/hellotherematey/tedata/blob/main/src/tedata/firstLastDates.js"
    click JS4 "https://github.com/hellotherematey/tedata/blob/main/src/tedata/init_tooltips.js"
    click JS5 "https://github.com/hellotherematey/tedata/blob/main/src/tedata/latest_points.js"
    click Docs "https://github.com/hellotherematey/tedata/tree/main/docs"
    click Tests "https://github.com/hellotherematey/tedata/tree/main/tests"
```
