# sameernaeem777/crypto-data-engineering-lab architecture

> Architecture diagram of the GitHub repository sameernaeem777/crypto-data-engineering-lab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sameernaeem777/crypto-data-engineering-lab
- Repository: https://github.com/sameernaeem777/crypto-data-engineering-lab

## Overview

This repository is a cryptocurrency market-data ETL project. Its main workflow fetches market data, validates records, and upserts them into PostgreSQL; an Airflow DAG can schedule that pipeline hourly. A separate transformation module fetches the same source data and writes a CSV. SQL files also define database setup, views, and analytics, but the sampled runtime code does not show those SQL files being executed. The README is empty, so capabilities and connections beyond the sampled code are inferred only where filenames provide evidence.

Main entry points: [`src/load/load_market_data.py`](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/load/load_market_data.py) (Load market data), [`src/extract/fetch_market_data.py`](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/extract/fetch_market_data.py) (Fetch markets), [`src/load/db_connection.py`](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/load/db_connection.py) (Database connection), [`src/transform/transform_market_data.py`](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/transform/transform_market_data.py) (CSV transformation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Scheduling"]
  node_dag["Hourly DAG<br/>[crypto_etl_dag.py]"]
end

subgraph group_pipeline["ETL Pipeline"]
  node_entrypoint["Pipeline entrypoint<br/>[run_pipeline.py]"]
  node_loader["Load market data"]
  node_extractor["Fetch markets"]
  node_validator["Validate coin"]
  node_connection["Database connection<br/>[db_connection.py]"]
end

subgraph group_analytics["CSV and SQL Analytics"]
  node_transform["CSV transformation"]
  node_csv["Market data CSV"]
  node_ddl["Market table schema"]
  node_indexes["Database indexes<br/>[indexes.sql]"]
  node_views["Analytics views<br/>[create_views.sql]"]
  node_queries["Analytics queries"]
end

subgraph group_configuration["Configuration"]
  node_config["Runtime settings<br/>[config.py]"]
end

node_scheduler(("Airflow scheduler"))
node_marketdb[("PostgreSQL market data")]
node_marketapi["CoinGecko API"]

node_scheduler -->|"schedules"| node_dag
node_dag -->|"dispatches"| node_entrypoint
node_entrypoint -->|"calls"| node_loader
node_loader -->|"calls"| node_extractor
node_extractor -->|"requests data"| node_marketapi
node_loader -->|"validates records"| node_validator
node_loader -->|"opens connection"| node_connection
node_connection -->|"connects"| node_marketdb
node_loader -->|"upserts records"| node_marketdb
node_transform -->|"calls"| node_extractor
node_transform -->|"writes CSV"| node_csv
node_extractor -->|"reads API settings"| node_config
node_connection -->|"reads database settings"| node_config

click node_dag "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/dags/crypto_etl_dag.py"
click node_entrypoint "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/run_pipeline.py"
click node_loader "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/load/load_market_data.py"
click node_extractor "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/extract/fetch_market_data.py"
click node_validator "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/validate/validate_market_data.py"
click node_connection "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/load/db_connection.py"
click node_transform "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/transform/transform_market_data.py"
click node_ddl "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/sql/ddl/001_create_market_data.sql"
click node_indexes "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/sql/indexes.sql"
click node_views "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/sql/create_views.sql"
click node_queries "https://github.com/sameernaeem777/crypto-data-engineering-lab/tree/main/sql/analytics"
click node_config "https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/utils/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dag toneBlue
class node_entrypoint,node_loader,node_extractor,node_validator,node_connection,node_marketdb toneAmber
class node_transform,node_csv,node_ddl,node_indexes,node_views,node_queries,node_marketapi toneMint
class node_config,node_scheduler toneRose
```

## Components

### Scheduling

- [Hourly DAG](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/dags/crypto_etl_dag.py): `dags/crypto_etl_dag.py`

### ETL Pipeline

- [Pipeline entrypoint](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/run_pipeline.py): `run_pipeline.py`
- [Load market data](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/load/load_market_data.py): `src/load/load_market_data.py`
- [Fetch markets](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/extract/fetch_market_data.py): `src/extract/fetch_market_data.py`
- [Validate coin](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/validate/validate_market_data.py): `src/validate/validate_market_data.py`
- [Database connection](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/load/db_connection.py): `src/load/db_connection.py`

### CSV and SQL Analytics

- [CSV transformation](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/transform/transform_market_data.py): `src/transform/transform_market_data.py`
- **Market data CSV**
- [Market table schema](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/sql/ddl/001_create_market_data.sql): `sql/ddl/001_create_market_data.sql`
- [Database indexes](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/sql/indexes.sql): `sql/indexes.sql`
- [Analytics views](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/sql/create_views.sql): `sql/create_views.sql`
- [Analytics queries](https://github.com/sameernaeem777/crypto-data-engineering-lab/tree/main/sql/analytics): `sql/analytics`

### Configuration

- [Runtime settings](https://github.com/sameernaeem777/crypto-data-engineering-lab/blob/main/src/utils/config.py): `src/utils/config.py`

### Other components

- **Airflow scheduler**
- **PostgreSQL market data**
- **CoinGecko API**

## Connections

- Airflow scheduler → Hourly DAG: schedules
- Hourly DAG → Pipeline entrypoint: dispatches
- Pipeline entrypoint → Load market data: calls
- Load market data → Fetch markets: calls
- Fetch markets → CoinGecko API: requests data
- Load market data → Validate coin: validates records
- Load market data → Database connection: opens connection
- Database connection → PostgreSQL market data: connects
- Load market data → PostgreSQL market data: upserts records
- CSV transformation → Fetch markets: calls
- CSV transformation → Market data CSV: writes CSV
- Fetch markets → Runtime settings: reads API settings
- Database connection → Runtime settings: reads database settings