How longngd/stockmarketetlpipeline works
Project type and purpose
• This is an ETL pipeline project (“Stock Market ETL Pipeline”) built to collect, process, and load stock market and currency data into Google BigQuery.
• It uses Apache Airflow for scheduling and orchestration, Python libraries (pandas, yfinance, BeautifulSoup) for data processing, and pandas-gbq to write to BigQuery.
File structure
• Top-level files only:
• dag.py – defines the Airflow DAG and tasks
• ETL.py – contains data-extraction, transformation, and loading functions
• requirements.txt – declares Python dependencies
• README.md – project overview and instructions
• No nested directories; functionality is split between orchestration (dag.py) and processing modules (ETL.py).
README for additional insights
• Data sources: Yahoo Finance (via yfinance) and currency rates (via BeautifulSoup).
• Transformations: compute RSI, SMA, EMA, Bollinger Bands, clean & normalize.
• Load target: Google BigQuery.
• Orchestration: Airflow DAG, scheduled daily, manual trigger available.
Elements
• Clear labels on every component (e.g., “Airflow Scheduler,” “ETL.extract_stock_data()”).
• Rectangles for code modules/tasks
• Parallelograms for external systems/APIs