# hassan-mehsen/cryptolakehouse architecture

> Architecture diagram of the GitHub repository hassan-mehsen/cryptolakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hassan-mehsen/cryptolakehouse
- Repository: https://github.com/hassan-mehsen/cryptolakehouse

## Overview

##### Project Type & Purpose
- CryptoLakehouse is a **data platform (Lakehouse + DWH)** built as an automated ELT pipeline for **crypto market data**.
- It ingests data from public APIs (e.g., CoinMarketCap), historizes raw snapshots in a **multi-layer Data Lake** (bronze/silver/gold), transforms/enriches with Spark, and loads analytics-ready tables into a **PostgreSQL Data Warehouse**.
- It is fully orchestrated by **Apache Airflow** and containerized with **Docker**.

##### Main Architectural Components
A. External Systems
- Public Crypto APIs (multiple endpoints for listings, global metrics, sentiment, etc.)
B. Extraction Layer
- Python “extractors” (OOP classes inheriting from BaseExtractor) invoked by Airflow PythonOperators
- Writes timestamped Parquet snapshots into the **Bronze** zone of the Data Lake
- Logs & metadata stored under `logs/` and `metadata/extract/`
C. Data Lake
- Bronze (raw, historized .parquet)
- Silver (cleaned, enriched Delta Lake tables)
- Gold (backups of DWH or ML output)
D. Transformation Layer
- PySpark jobs (OOP BaseTransformer + domain transformers)
- Reads bronze → writes silver (Delta)
- Orchestrated by Airflow via BashOperator `spark-submit`
- Metadata & logs under `metadata/transform/` and `logs/transform.log`
E. Load Layer
- PySpark loaders (OOP BaseLoader + domain loaders)
- Reads silver Delta → writes to PostgreSQL DWH (fact & dimension tables)
- PK/FK checks, idempotent append loads
- Metadata & logs under `metadata/load/` and `logs/load.log`
F. Data Warehouse
- PostgreSQL container with star/snowflake schema (dims: dim_crypto_id, dim_exchange_id, link tables, facts: fact_crypto_market, fact_global_market, etc.)
- SQL scripts, views, tests, stored procedures under `warehouse/`
- Schema managed via SQLAlchemy & Alembic (`src/db/`)
G. Orchestration & Infrastructure
- **Apache Airflow** DAGs in `dags/` organize frequency-based Extract / Transform / Load / backup pipelines
- **Docker Compose** spins up containers: Airflow, Postgres, Spark, etc.
- Configuration files under `docker/`

##### Relationships & Data Flows
- Extraction Layer → writes to **Bronze** storage
- **Metadata & Logs** components are side-cars on each layer for traceability

##### Architectural Patterns & Design Principles
- **Lakehouse Pattern**: unified Data Lake (bronze/silver/gold) + DWH for BI
- **ELT**: extract raw → load to lake → transform in lake → final load to DWH
- **OOP Frameworks**: BaseExtractor/BaseTransformer/BaseLoader enable consistent logging, metadata management, and easy extensibility
- **Frequency-Based Orchestration**: 10×/5×/daily/weekly runs grouped by volatility
- **Incremental & Append-Only**: snapshot timestamping, idempotence checks, historical versioning (Delta history, DWH time keys)
- **Hybrid Star/Snowflake Schema** in DWH for performance + modularity

##### Technologies & Frameworks to Annotate
- Python (requests, pandas, PySpark)
- Apache Spark (Delta Lake)
- Apache Airflow
- Docker & Docker Compose
- PostgreSQL
- Parquet & Delta formats
- SQLAlchemy & Alembic for migrations
- Metadata & logging in JSONL / flat files

##### Essential Elements
- Clear **component labels** (e.g., “CryptoListingsExtractor”).
- **Annotations** for technologies on each component (e.g., “PySpark + Delta”).
- **Frequency notes** on the orchestration edges.
- **Metadata/Log tracks** alongside each layer as separate pipelines.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration
    Airflow["Apache Airflow"]:::orchestration
    DAGs["DAGs"]:::orchestration
    click DAGs "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/dags/"

    %% External Systems
    API["Public Crypto APIs"]:::external

    %% Extraction Layer
    subgraph "Extraction Layer"
        direction TB
        BaseExtractor["BaseExtractor"]:::extraction
        Extractors["Crypto Extractors"]:::extraction
        LogsExtract["logs/extract.log"]:::metadata
        MetaExtract["metadata/extract"]:::metadata
    end

    %% Bronze Data Lake
    Bronze["Bronze Data Lake<br/>(Raw Parquet)"]:::bronze

    %% Transformation Layer
    subgraph "Transformation Layer"
        direction TB
        BaseTransformer["BaseTransformer"]:::transform
        Transformers["Domain Transformers"]:::transform
        OrchestratorsTransform["Transform Orchestrator"]:::transform
        LogsTransform["logs/transform.log"]:::metadata
        MetaTransform["metadata/transform"]:::metadata
    end

    %% Silver Data Lake
    Silver["Silver Data Lake<br/>(Delta Tables)"]:::silver

    %% Load Layer
    subgraph "Load Layer"
        direction TB
        BaseLoader["BaseLoader"]:::load
        Loaders["Domain Loaders"]:::load
        OrchestratorsLoad["Load Orchestrator"]:::load
        LogsLoad["logs/load.log"]:::metadata
        MetaLoad["metadata/load"]:::metadata
    end

    %% Data Warehouse
    DWH["PostgreSQL DWH<br/>(Star/Snowflake Schema)"]:::warehouse
    Models["src/db/models.py"]:::warehouse
    BackupPostgres["backup_postgres.py"]:::warehouse
    Alembic["alembic/"]:::warehouse
    WarehouseScripts["warehouse/"]:::warehouse

    %% Gold Data Lake
    Gold["Gold Data Lake<br/>(DWH Backups)"]:::gold

    %% Infrastructure
    subgraph "Infrastructure & Containers"
        direction TB
        Compose["Docker Compose"]:::infra
        AirflowSvc["Airflow Container"]:::infra
        PostgresSvc["Postgres Container"]:::infra
        SparkSvc["Spark Container"]:::infra
    end

    %% Data Flows
    Airflow -->|schedules| DAGs
    DAGs -->|10×/5×/daily/weekly| BaseExtractor
    API -->|REST API| BaseExtractor
    BaseExtractor -->|Parquet write| Bronze
    Extractors --> Bronze
    Bronze -->|Delta write| BaseTransformer
    BaseTransformer --> Silver
    Transformers --> Silver
    Silver -->|JDBC load| BaseLoader
    BaseLoader --> DWH
    Loaders --> DWH
    DWH -->|backup snapshots| Gold

    %% Metadata & Logs Flows
    BaseExtractor --> LogsExtract
    BaseExtractor --> MetaExtract
    BaseTransformer --> LogsTransform
    BaseTransformer --> MetaTransform
    BaseLoader --> LogsLoad
    BaseLoader --> MetaLoad

    %% Infrastructure Links
    Compose --> AirflowSvc
    Compose --> PostgresSvc
    Compose --> SparkSvc
    AirflowSvc --> Airflow
    PostgresSvc --> DWH
    SparkSvc --> BaseTransformer

    %% Click Events - Extraction
    click BaseExtractor "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/src/extract/base_extractor.py"
    click Extractors "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/src/extract/extractors/"
    click LogsExtract "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/logs/extract.log"
    click MetaExtract "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/metadata/extract/"

    %% Click Events - Bronze
    click Bronze "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/data/bronze/"

    %% Click Events - Transform
    click BaseTransformer "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/src/transform/base/base_transformer.py"
    click Transformers "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/src/transform/transformers/"
    click OrchestratorsTransform "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/src/transform/orchestrators/"
    click LogsTransform "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/logs/transform.log"
    click MetaTransform "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/metadata/transform/"
    click Silver "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/data/silver/"

    %% Click Events - Load
    click BaseLoader "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/src/load/base/base_loader.py"
    click Loaders "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/src/load/loaders/"
    click OrchestratorsLoad "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/src/load/orchestrators/"
    click LogsLoad "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/logs/load.log"
    click MetaLoad "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/metadata/load/"

    %% Click Events - DWH
    click Models "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/src/db/models.py"
    click BackupPostgres "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/src/db/backup_postgres.py"
    click Alembic "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/alembic/"
    click WarehouseScripts "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/warehouse/"

    %% Click Events - Infra
    click Compose "https://github.com/hassan-mehsen/cryptolakehouse/blob/master/docker/docker-compose.yml"
    click AirflowSvc "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/docker/airflow/"
    click PostgresSvc "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/docker/postgres/"
    click SparkSvc "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/docker/spark/"

    %% Click Events - Gold
    click Gold "https://github.com/hassan-mehsen/cryptolakehouse/tree/master/data/gold/"

    %% Styles
    classDef extraction fill:#D0E8FF,stroke:#0366d6,color:#0366d6
    classDef bronze fill:#B3DFFF,stroke:#0366d6,color:#0366d6
    classDef transform fill:#DFFFE0,stroke:#28a745,color:#28a745
    classDef silver fill:#E6FFE6,stroke:#28a745,color:#28a745
    classDef load fill:#FFE8CC,stroke:#d2691e,color:#d2691e
    classDef warehouse fill:#FFEFD5,stroke:#d2691e,color:#d2691e
    classDef metadata fill:#F0F0F0,stroke:#666666,color:#666666
    classDef orchestration fill:#F5E6FF,stroke:#6f42c1,color:#6f42c1
    classDef infra fill:#F0F5FF,stroke:#005bcc,color:#005bcc
    classDef external fill:#FFF5F5,stroke:#d73a49,color:#d73a49
    classDef gold fill:#FFF1F2,stroke:#d73a49,color:#d73a49
```
