How hellotherematey/tedata works
And Define the Main Components
• Client/Entry Points:
• Command Line Interface (CLI): Triggered when running “python -m tedata …” (see __main__.py).
• Interactive/API Usage: Importing the “tedata” package in scripts or notebooks.
• Scraping Engine:
• TE_Scraper Class (in src/tedata/scraper.py and scrape_chart.py): Contains the scraping workflow (loading pages, scraping x-axis and y-axis information, scaling data, and extracting metadata).
• Search Module (src/tedata/search.py): Provides “search_TE” functionality to search Trading Economics and fetch a table of possible datasets.
• Utility and Support Modules:
• Base Functions (src/tedata/base.py)
• Logger Setup (src/tedata/logger_setup.py) for debugging and error reporting.
• Additional JS and utility scripts (e.g., custom_datespan.js, check_highcharts.js) that supplement the scraping process.
• Data Processing and Export:
• Integration with Pandas for handling the scraped data.
• Plotting (via Plotly and optionally Kaleido for export).
• Export functions (saving to Excel, CSV, or HDF5) within the TE_Scraper object.
• Documentation & Testing Components:
• Docs directory: Contains HTML documentation (including guides and screenshots).
• Tests directory: Includes notebooks and automated tests to validate functionality.
External and Dependency Interactions
• External Dependencies:
• Selenium Webdriver (specifically configured with Firefox).
• BeautifulSoup4 for HTML parsing.
• Plotly for interactive charting.
• Pandas for data structures and exporting formats.
• Optional libraries such as openpyxl, kaleido, and nbformat.
• External Systems:
• Trading Economics website from which data is scraped.
• Web Browser (Firefox) used and controlled by Selenium.
Lay Out the Data Flow and Component Interactions
• User invokes either the CLI or imports tedata in a notebook/script.
• In the Core API:
• When a URL or search query is provided, the application routes it to the Search Module or directly to the Scraper Module.
• From “search_TE”: The search query is sent, results (dataframes of potential indicators) are returned.
• When scraping is initiated (either via “scrape_chart” or direct TE_Scraper calls), the following steps occur:
a. The Web Driver is initialized (or reused) and navigates to the Trading Economics URL.
b. Selenium operates the browser (Firefox) to capture the chart element.
○ Data scraping via SVG path extraction, scaling pixel values using y-axis values.
○ Alternative methods (tooltips, mixed, highcharts_api) indicated as options.
d. Data Processing: The raw pixel/index data is converted to a Pandas Series along with metadata.
e. Plotting & Export: The processed data may be plotted (with Plotly) and exported to Excel/CSV/HDF5.
• Error handling and logging:
• Logger Setup module receives events and potential error messages, routed back to the central logger.
Architectural Patterns and Design Principles
• Modularity: Each functional block (search, scraping, logging, utilities) is separated in the src/tedata folder.
• Single Responsibility: Each module tackles a specific aspect – data fetching, parsing, transforming, and plotting.
• Extensibility: The design (with multiple scraping methods as options) suggests an extensible architecture for adding more methods (e.g., support for Chrome in the future).
• Dependency Injection: Options for reusing or reinitializing drivers via the TE_Scraper’s parameters help maintain flexibility.
Final Checklist
• Clearly label every major component with its purpose.
• Distinguish between different layers (API layer, scraping engine layer, data processing/visualization layer, external dependencies).
• Incorporate notes on how browser automation is critical to initializing and driving the scraping engine.