How salgadosalgadoje/tech-market-data-pipeline works
This repository implements a scheduled financial-market ETL pipeline: Airflow runs extraction, Bronze storage, Silver validation/enrichment, and Gold warehouse loading for technology-stock data. The README also describes downstream analytical consumption, but no consumer implementation is present in the tree. Component relationships below are supported by the DAG excerpt or README; the stage internals are summarized from the excerpts, with no claims about unshown implementation details.
Main entry points: transform.py Silver Transformer · load_postgres.py Postgres Loader · finance_etl_dag.py Airflow DAG · extract_s3.py Extraction Pipeline
Components 15
Orchestration
- Airflow DAG
dags/finance_etl_dag.py
Bronze Ingestion
- Extraction Pipeline
src/extract_s3.py - Market Extractor
src/extract_s3.py - S3 Bronze Lake
Silver Curation
- Silver Transformer
src/transform.py - Record Validation
src/transform.py - Cleaning and Indicators
src/transform.py - S3 Silver Lake
Gold Warehouse
- Postgres Loader
src/load_postgres.py - Silver Parquet Reader
src/load_postgres.py - Record Preparation
src/load_postgres.py - Metrics ORM Model
src/load_postgres.py - PostgreSQL Warehouse
Analytics Consumption
- Streamlit / FastAPI
Other components
- Yahoo Finance
Connections 18
- Airflow DAG → Extraction Pipeline: runs extraction
dags/finance_etl_dag.py - Airflow DAG → Silver Transformer: runs transformation
dags/finance_etl_dag.py - Airflow DAG → Postgres Loader: runs loading
dags/finance_etl_dag.py - Airflow DAG → Silver Transformer: orders after extraction
dags/finance_etl_dag.py - Airflow DAG → Postgres Loader: orders after transformation
dags/finance_etl_dag.py - Extraction Pipeline → Market Extractor: fetches ticker data
src/extract_s3.py - Market Extractor → Yahoo Finance: downloads prices
README.md - Extraction Pipeline → S3 Bronze Lake: uploads raw data
src/extract_s3.py - Silver Transformer → S3 Bronze Lake: reads raw Parquet
src/transform.py - Silver Transformer → Record Validation: validates records
src/transform.py - Silver Transformer → Cleaning and Indicators: cleans and enriches
README.md - Silver Transformer → S3 Silver Lake: writes curated Parquet
src/transform.py - Postgres Loader → Silver Parquet Reader: loads Silver files
src/load_postgres.py - Silver Parquet Reader → S3 Silver Lake: reads latest Parquet
src/load_postgres.py - Postgres Loader → Record Preparation: prepares records
README.md - Postgres Loader → Metrics ORM Model: upserts model rows
src/load_postgres.py - Postgres Loader → PostgreSQL Warehouse: writes idempotently
src/load_postgres.py - PostgreSQL Warehouse → Streamlit / FastAPI: serves analytics
README.md