# longngd/stockmarketetlpipeline architecture

> Architecture diagram of the GitHub repository longngd/stockmarketetlpipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/longngd/stockmarketetlpipeline
- Repository: https://github.com/longngd/stockmarketetlpipeline

## Overview

##### Project type and purpose
- This is an ETL pipeline project (“Stock Market ETL Pipeline”) built to collect, process, and load stock market and currency data into Google BigQuery.
- It uses Apache Airflow for scheduling and orchestration, Python libraries (pandas, yfinance, BeautifulSoup) for data processing, and pandas-gbq to write to BigQuery.

##### File structure
- Top-level files only:
- dag.py – defines the Airflow DAG and tasks
- ETL.py – contains data-extraction, transformation, and loading functions
- requirements.txt – declares Python dependencies
- README.md – project overview and instructions
- No nested directories; functionality is split between orchestration (dag.py) and processing modules (ETL.py).

##### README for additional insights
- Data sources: Yahoo Finance (via yfinance) and currency rates (via BeautifulSoup).
- Transformations: compute RSI, SMA, EMA, Bollinger Bands, clean & normalize.
- Load target: Google BigQuery.
- Orchestration: Airflow DAG, scheduled daily, manual trigger available.

##### Elements
- Clear labels on every component (e.g., “Airflow Scheduler,” “ETL.extract_stock_data()”).
- Rectangles for code modules/tasks
- Parallelograms for external systems/APIs

## Diagram (Mermaid)

```mermaid
flowchart TD
    Trigger["⏰ Daily Trigger"]:::orchestration
    subgraph "Airflow Orchestrator"
        Scheduler["Airflow Scheduler"]:::orchestration
        DAG["DAG Definition"]:::orchestration
    end
    subgraph "External Data Sources"
        Yahoo["Yahoo Finance API"]:::external
        FX["FX Rates Web Scraper"]:::external
    end
    subgraph "ETL Processing Module"
        ExtractStock["Extract Stock Data"]:::processing
        ExtractFX["Extract FX Rates"]:::processing
        Compute["Compute Indicators (RSI, SMA, EMA, Bollinger)"]:::processing
        Load["Load to BigQuery"]:::processing
    end
    BigQuery["Google BigQuery"]:::storage
    subgraph "Configuration & Documentation"
        Requirements["requirements.txt"]:::configuration
        README["README.md"]:::configuration
    end

    Trigger --> Scheduler --> DAG
    DAG --> Yahoo --> ExtractStock
    DAG --> FX --> ExtractFX
    ExtractStock --> Compute
    ExtractFX --> Compute
    Compute --> Load --> BigQuery
    DAG --> Requirements
    DAG --> README

    click DAG "https://github.com/longngd/stockmarketetlpipeline/blob/master/dag.py"
    click ExtractStock "https://github.com/longngd/stockmarketetlpipeline/blob/master/ETL.py"
    click ExtractFX "https://github.com/longngd/stockmarketetlpipeline/blob/master/ETL.py"
    click Compute "https://github.com/longngd/stockmarketetlpipeline/blob/master/ETL.py"
    click Load "https://github.com/longngd/stockmarketetlpipeline/blob/master/ETL.py"
    click Requirements "https://github.com/longngd/stockmarketetlpipeline/blob/master/requirements.txt"
    click README "https://github.com/longngd/stockmarketetlpipeline/blob/master/README.md"

    classDef orchestration fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#FFA500,stroke:#000,stroke-width:1px
    classDef storage fill:#DDA0DD,stroke:#000,stroke-width:1px
    classDef configuration fill:#F0E68C,stroke:#000,stroke-width:1px
```
