How hassan-mehsen/cryptolakehouse works
Project Type & Purpose
• CryptoLakehouse is a data platform (Lakehouse + DWH) built as an automated ELT pipeline for crypto market data.
• It ingests data from public APIs (e.g., CoinMarketCap), historizes raw snapshots in a multi-layer Data Lake (bronze/silver/gold), transforms/enriches with Spark, and loads analytics-ready tables into a PostgreSQL Data Warehouse.
• It is fully orchestrated by Apache Airflow and containerized with Docker.
Main Architectural Components
A. External Systems
• Public Crypto APIs (multiple endpoints for listings, global metrics, sentiment, etc.)
B. Extraction Layer
• Python “extractors” (OOP classes inheriting from BaseExtractor) invoked by Airflow PythonOperators
• Writes timestamped Parquet snapshots into the Bronze zone of the Data Lake
• Logs & metadata stored under logs/ and metadata/extract/
C. Data Lake
• Bronze (raw, historized .parquet)
• Silver (cleaned, enriched Delta Lake tables)
• Gold (backups of DWH or ML output)
D. Transformation Layer
• PySpark jobs (OOP BaseTransformer + domain transformers)
• Reads bronze → writes silver (Delta)
• Orchestrated by Airflow via BashOperator spark-submit
• Metadata & logs under metadata/transform/ and logs/transform.log
E. Load Layer
• PySpark loaders (OOP BaseLoader + domain loaders)
• Reads silver Delta → writes to PostgreSQL DWH (fact & dimension tables)
• PK/FK checks, idempotent append loads
• Metadata & logs under metadata/load/ and logs/load.log
F. Data Warehouse
• PostgreSQL container with star/snowflake schema (dims: dim_crypto_id, dim_exchange_id, link tables, facts: fact_crypto_market, fact_global_market, etc.)
• SQL scripts, views, tests, stored procedures under warehouse/
• Schema managed via SQLAlchemy & Alembic (src/db/)
G. Orchestration & Infrastructure
• Apache Airflow DAGs in dags/ organize frequency-based Extract / Transform / Load / backup pipelines
• Docker Compose spins up containers: Airflow, Postgres, Spark, etc.
• Configuration files under docker/
Relationships & Data Flows
• Extraction Layer → writes to Bronze storage
• Metadata & Logs components are side-cars on each layer for traceability
Architectural Patterns & Design Principles
• Lakehouse Pattern: unified Data Lake (bronze/silver/gold) + DWH for BI
• ELT: extract raw → load to lake → transform in lake → final load to DWH
• OOP Frameworks: BaseExtractor/BaseTransformer/BaseLoader enable consistent logging, metadata management, and easy extensibility
• Frequency-Based Orchestration: 10×/5×/daily/weekly runs grouped by volatility
• Incremental & Append-Only: snapshot timestamping, idempotence checks, historical versioning (Delta history, DWH time keys)
• Hybrid Star/Snowflake Schema in DWH for performance + modularity
Technologies & Frameworks to Annotate
• Python (requests, pandas, PySpark)
• Apache Spark (Delta Lake)
• Apache Airflow
• Docker & Docker Compose
• PostgreSQL
• Parquet & Delta formats
• SQLAlchemy & Alembic for migrations
• Metadata & logging in JSONL / flat files
Essential Elements
• Clear component labels (e.g., “CryptoListingsExtractor”).
• Annotations for technologies on each component (e.g., “PySpark + Delta”).
• Frequency notes on the orchestration edges.
• Metadata/Log tracks alongside each layer as separate pipelines.