How sameernaeem777/crypto-data-engineering-lab works
This repository is a cryptocurrency market-data ETL project. Its main workflow fetches market data, validates records, and upserts them into PostgreSQL; an Airflow DAG can schedule that pipeline hourly. A separate transformation module fetches the same source data and writes a CSV. SQL files also define database setup, views, and analytics, but the sampled runtime code does not show those SQL files being executed. The README is empty, so capabilities and connections beyond the sampled code are inferred only where filenames provide evidence.
Main entry points: load_market_data.py Load market data · fetch_market_data.py Fetch markets · db_connection.py Database connection · transform_market_data.py CSV transformation
Components 16
Scheduling
- Hourly DAG
dags/crypto_etl_dag.py
ETL Pipeline
- Pipeline entrypoint
run_pipeline.py - Load market data
src/load/load_market_data.py - Fetch markets
src/extract/fetch_market_data.py - Validate coin
src/validate/validate_market_data.py - Database connection
src/load/db_connection.py
CSV and SQL Analytics
- CSV transformation
src/transform/transform_market_data.py - Market data CSV
- Market table schema
sql/ddl/001_create_market_data.sql - Database indexes
sql/indexes.sql - Analytics views
sql/create_views.sql - Analytics queries
sql/analytics
Configuration
- Runtime settings
src/utils/config.py
Other components
- Airflow scheduler
- PostgreSQL market data
- CoinGecko API
Connections 13
- Airflow scheduler → Hourly DAG: schedules
- Hourly DAG → Pipeline entrypoint: dispatches
- Pipeline entrypoint → Load market data: calls
- Load market data → Fetch markets: calls
- Fetch markets → CoinGecko API: requests data
- Load market data → Validate coin: validates records
- Load market data → Database connection: opens connection
- Database connection → PostgreSQL market data: connects
- Load market data → PostgreSQL market data: upserts records
- CSV transformation → Fetch markets: calls
- CSV transformation → Market data CSV: writes CSV
- Fetch markets → Runtime settings: reads API settings
- Database connection → Runtime settings: reads database settings