# salgadosalgadoje/tech-market-data-pipeline architecture

> Architecture diagram of the GitHub repository salgadosalgadoje/tech-market-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salgadosalgadoje/tech-market-data-pipeline
- Repository: https://github.com/salgadosalgadoje/tech-market-data-pipeline

## Overview

This repository implements a scheduled financial-market ETL pipeline: Airflow runs extraction, Bronze storage, Silver validation/enrichment, and Gold warehouse loading for technology-stock data. The README also describes downstream analytical consumption, but no consumer implementation is present in the tree. Component relationships below are supported by the DAG excerpt or README; the stage internals are summarized from the excerpts, with no claims about unshown implementation details.

Main entry points: [`src/transform.py`](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/transform.py) (Silver Transformer), [`src/load_postgres.py`](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py) (Postgres Loader), [`dags/finance_etl_dag.py`](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/dags/finance_etl_dag.py) (Airflow DAG), [`src/extract_s3.py`](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/extract_s3.py) (Extraction Pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_airflow["Airflow DAG<br/>[finance_etl_dag.py]"]
end

subgraph group_bronze["Bronze Ingestion"]
  node_extract_pipeline["Extraction Pipeline<br/>[extract_s3.py]"]
  node_extractor["Market Extractor<br/>[extract_s3.py]"]
  node_s3_bronze[("S3 Bronze Lake")]
end

subgraph group_silver["Silver Curation"]
  node_transformer["Silver Transformer<br/>[transform.py]"]
  node_record_schema["Record Validation<br/>[transform.py]"]
  node_silver_clean["Cleaning and Indicators<br/>[transform.py]"]
  node_s3_silver[("S3 Silver Lake")]
end

subgraph group_gold["Gold Warehouse"]
  node_gold_loader["Postgres Loader<br/>[load_postgres.py]"]
  node_silver_reader["Silver Parquet Reader<br/>[load_postgres.py]"]
  node_record_preparer["Record Preparation<br/>[load_postgres.py]"]
  node_warehouse_model["Metrics ORM Model<br/>[load_postgres.py]"]
  node_postgres[("PostgreSQL Warehouse")]
end

subgraph group_consumption["Analytics Consumption"]
  node_analytical_clients["Streamlit / FastAPI"]
end

node_market_source["Yahoo Finance"]

node_airflow -->|"runs extraction"| node_extract_pipeline
node_airflow -->|"runs transformation"| node_transformer
node_airflow -->|"runs loading"| node_gold_loader
node_airflow -->|"orders after extraction"| node_transformer
node_airflow -->|"orders after transformation"| node_gold_loader
node_extract_pipeline -->|"fetches ticker data"| node_extractor
node_extractor -->|"downloads prices"| node_market_source
node_extract_pipeline -->|"uploads raw data"| node_s3_bronze
node_transformer -->|"reads raw Parquet"| node_s3_bronze
node_transformer -->|"validates records"| node_record_schema
node_transformer -->|"cleans and enriches"| node_silver_clean
node_transformer -->|"writes curated Parquet"| node_s3_silver
node_gold_loader -->|"loads Silver files"| node_silver_reader
node_silver_reader -->|"reads latest Parquet"| node_s3_silver
node_gold_loader -->|"prepares records"| node_record_preparer
node_gold_loader -->|"upserts model rows"| node_warehouse_model
node_gold_loader -->|"writes idempotently"| node_postgres
node_postgres -->|"serves analytics"| node_analytical_clients

click node_airflow "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/dags/finance_etl_dag.py"
click node_extract_pipeline "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/extract_s3.py"
click node_extractor "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/extract_s3.py"
click node_transformer "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/transform.py"
click node_record_schema "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/transform.py"
click node_silver_clean "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/transform.py"
click node_gold_loader "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py"
click node_silver_reader "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py"
click node_record_preparer "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py"
click node_warehouse_model "https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow toneBlue
class node_extract_pipeline,node_extractor,node_s3_bronze toneAmber
class node_transformer,node_record_schema,node_silver_clean,node_s3_silver toneMint
class node_gold_loader,node_silver_reader,node_record_preparer,node_warehouse_model,node_postgres toneRose
class node_analytical_clients,node_market_source toneIndigo
```

## Components

### Orchestration

- [Airflow DAG](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/dags/finance_etl_dag.py): `dags/finance_etl_dag.py`

### Bronze Ingestion

- [Extraction Pipeline](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/extract_s3.py): `src/extract_s3.py`
- [Market Extractor](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/extract_s3.py): `src/extract_s3.py`
- **S3 Bronze Lake**

### Silver Curation

- [Silver Transformer](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/transform.py): `src/transform.py`
- [Record Validation](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/transform.py): `src/transform.py`
- [Cleaning and Indicators](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/transform.py): `src/transform.py`
- **S3 Silver Lake**

### Gold Warehouse

- [Postgres Loader](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py): `src/load_postgres.py`
- [Silver Parquet Reader](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py): `src/load_postgres.py`
- [Record Preparation](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py): `src/load_postgres.py`
- [Metrics ORM Model](https://github.com/salgadosalgadoje/tech-market-data-pipeline/blob/main/src/load_postgres.py): `src/load_postgres.py`
- **PostgreSQL Warehouse**

### Analytics Consumption

- **Streamlit / FastAPI**

### Other components

- **Yahoo Finance**

## Connections

- Airflow DAG → Extraction Pipeline: runs extraction (evidence: `dags/finance_etl_dag.py`)
- Airflow DAG → Silver Transformer: runs transformation (evidence: `dags/finance_etl_dag.py`)
- Airflow DAG → Postgres Loader: runs loading (evidence: `dags/finance_etl_dag.py`)
- Airflow DAG → Silver Transformer: orders after extraction (evidence: `dags/finance_etl_dag.py`)
- Airflow DAG → Postgres Loader: orders after transformation (evidence: `dags/finance_etl_dag.py`)
- Extraction Pipeline → Market Extractor: fetches ticker data (evidence: `src/extract_s3.py`)
- Market Extractor → Yahoo Finance: downloads prices (evidence: `README.md`)
- Extraction Pipeline → S3 Bronze Lake: uploads raw data (evidence: `src/extract_s3.py`)
- Silver Transformer → S3 Bronze Lake: reads raw Parquet (evidence: `src/transform.py`)
- Silver Transformer → Record Validation: validates records (evidence: `src/transform.py`)
- Silver Transformer → Cleaning and Indicators: cleans and enriches (evidence: `README.md`)
- Silver Transformer → S3 Silver Lake: writes curated Parquet (evidence: `src/transform.py`)
- Postgres Loader → Silver Parquet Reader: loads Silver files (evidence: `src/load_postgres.py`)
- Silver Parquet Reader → S3 Silver Lake: reads latest Parquet (evidence: `src/load_postgres.py`)
- Postgres Loader → Record Preparation: prepares records (evidence: `README.md`)
- Postgres Loader → Metrics ORM Model: upserts model rows (evidence: `src/load_postgres.py`)
- Postgres Loader → PostgreSQL Warehouse: writes idempotently (evidence: `src/load_postgres.py`)
- PostgreSQL Warehouse → Streamlit / FastAPI: serves analytics (evidence: `README.md`)
