# rmendes1/de_music_data_lake architecture

> Architecture diagram of the GitHub repository rmendes1/de_music_data_lake, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rmendes1/de_music_data_lake
- Repository: https://github.com/rmendes1/de_music_data_lake

## Overview

##### Project Type and Purpose
- This is a data‐engineering project implementing a Data Lake for music metadata.
- It ingests data from external Music APIs, performs both full loads and CDC ingestion via Debezium/Kafka into cloud storage (GCS), and organizes data into Databricks catalogs (`raw`, `bronze`, then further transformations into `silver`).
- Core technologies: Apache Spark, Docker (Compose), Debezium, Kafka, PostgreSQL, GCS, Databricks.

##### File Structure
Top-level components:
- api_consumer/ – downloads initial music data from public APIs into JSON or directly into Postgres.
- queries/ – SQL scripts to bootstrap or reset the Postgres source.
- docker-compose.yml, Dockerfile, entrypoint.sh – define a multi-service environment: Postgres, Kafka (with Zookeeper), Debezium connector, Spark, etc.
- spark_files/ – Spark applications for:
– `full_load.py`: one-time bulk ingestion into GCS
– `cdc_ingest.py`: continuous ingestion of CDC events from Kafka
– `manage_offset.py`: offset management for reliable consumption
– `writer.py`, `utils.py`, plus `config.py`
Also includes spark-defaults.conf, Postgres JDBC driver, Python requirements.
- src/bronze & src/silver – notebooks and SQL transform layers for Databricks:
– bronze: notebooks or scripts ingesting from GCS to Databricks bronze tables
– silver: `.sql` scripts for curated tables
- src/lib – shared Python utilities for ingestion, preprocessing.
- src/workflows – Python workflows (create/reset) for orchestrating tasks in Databricks workflows (JSON definitions for jobs).

##### Key Insights from the README
- CDC ingest uses Debezium + Kafka.
- Full load: uses Spark to extract entire Postgres data and write to GCS.
- Bronze layer ingestion and silver transformations run in Databricks.
- Docker Compose ties together Postgres, Kafka, Debezium, and Spark driver containers.

##### Main Components
A. External Data Sources
- Music APIs → `api_consumer` container
B. Source Database
- PostgreSQL container (initialized via `queries/insert_data_pg.sql`)
C. Change Data Capture Pipeline
- Debezium connector (configured via `debezium.json`)
- Apache Kafka cluster & CDC topics
D. Batch Ingestion (Full Load)
- Spark Driver (full_load.py) reads Postgres → writes Parquet/Delta to GCS (`raw` area)
E. Streaming Ingestion (CDC)
- Spark Driver (cdc_ingest.py) reads from Kafka topics → writes incremental Parquet/Delta to GCS (`raw` area)
- Offset management by `manage_offset.py`
F. Data Lake Storage
- Google Cloud Storage buckets:
– `/raw/music_data/...`
– `/bronze/music_data/...`
G. Databricks Workspace
- Raw → ingested as unmanaged tables
- Bronze → refined via notebooks in `src/bronze`
- Silver → curated tables via SQL scripts in `src/silver`
- Workflows orchestrated by `src/workflows` definitions
H. Consumers / Analytics
- BI tools or end-user queries against Silver tables

##### Relationships & Data Flows
- Postgres → Debezium → Kafka
- Postgres → Spark Full Load → GCS Raw
- Kafka → Spark CDC Ingest → GCS Raw
- GCS Raw → Databricks Bronze layer → GCS Bronze
- GCS Bronze → Databricks Silver layer → final tables
- Downstream analytics consume Silver

##### Architectural Patterns & Design Principles
- Lambda architecture: batch (full load) + stream (CDC) layered into a unified data lake
- Separation of concerns:
– Ingestion (api_consumer, Spark apps)
– Storage (GCS buckets)
– Serving/Transformation (Databricks notebooks & SQL)
- Modular, containerized services for portability and reproducibility

##### Checklist for the Principal Software Engineer
- Components listed in step 4.
- Related containers (Postgres + Debezium + Kafka) in a “Docker Compose” boundary.
- External dependencies (Music APIs, GCS, Databricks).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External
    API["External Music APIs"]:::external
    API --> APIConsumer

    %% Ingestion Services
    APIConsumer["API Consumer\n(api_consumer)"]:::ingestion
    click APIConsumer "https://github.com/rmendes1/de_music_data_lake/tree/main/api_consumer/"
    click APIConsumer "https://github.com/rmendes1/de_music_data_lake/blob/main/api_consumer/add_music_data.py"

    %% Docker Compose Group
    subgraph "Docker Compose"
        Postgres[(PostgreSQL)]:::storage
        Debez["Debezium\nConnector"]:::processing
        Kafka["Kafka & Zookeeper"]:::processing
    end
    APIConsumer --> Postgres
    click Postgres "https://github.com/rmendes1/de_music_data_lake/blob/main/docker-compose.yml"
    click Postgres "https://github.com/rmendes1/de_music_data_lake/blob/main/queries/insert_data_pg.sql"
    click Postgres "https://github.com/rmendes1/de_music_data_lake/blob/main/queries/offset_table_kafka.sql"
    click Postgres "https://github.com/rmendes1/de_music_data_lake/blob/main/entrypoint.sh"
    Postgres --> Debez
    click Debez "https://github.com/rmendes1/de_music_data_lake/blob/main/debezium.json"
    Debez --> Kafka
    click Kafka "https://github.com/rmendes1/de_music_data_lake/blob/main/docker-compose.yml"

    %% Shared Utilities
    SharedUtils["Shared Ingestion Utilities\n(src/lib)"]:::ingestion
    click SharedUtils "https://github.com/rmendes1/de_music_data_lake/blob/main/src/lib/ingestors.py"
    click SharedUtils "https://github.com/rmendes1/de_music_data_lake/blob/main/src/lib/preprocessing.py"
    click SharedUtils "https://github.com/rmendes1/de_music_data_lake/blob/main/src/lib/preprocessing_silver.py"
    click SharedUtils "https://github.com/rmendes1/de_music_data_lake/blob/main/src/lib/utils.py"

    %% Batch Ingestion
    FullLoad["Spark Full Load App\n(full_load.py)"]:::processing
    click FullLoad "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/spark_apps/full_load.py"
    click FullLoad "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/conf/spark-defaults.conf"
    click FullLoad "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/jars/postgresql-42.6.2.jar"
    click FullLoad "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/requirements.txt"
    Postgres -->|"batch (Parquet/Delta)"| FullLoad
    SharedUtils --> FullLoad

    %% Streaming Ingestion
    CDC["Spark CDC Ingest App\n(cdc_ingest.py)"]:::processing
    click CDC "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/spark_apps/cdc_ingest.py"
    click CDC "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/conf/spark-defaults.conf"
    click CDC "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/requirements.txt"
    ManageOffset["Offset Manager\n(manage_offset.py)"]:::processing
    click ManageOffset "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/spark_apps/manage_offset.py"
    Writer["Writer Module\n(writer.py)"]:::processing
    click Writer "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/spark_apps/writer.py"
    UtilsApp["Utils Module\n(utils.py)"]:::processing
    click UtilsApp "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/spark_apps/utils.py"
    ConfigApp["Config Module\n(config.py)"]:::processing
    click ConfigApp "https://github.com/rmendes1/de_music_data_lake/blob/main/spark_files/spark_apps/config.py"
    Kafka -->|"stream (CDC)"| CDC
    SharedUtils --> CDC
    ManageOffset -.-> CDC
    Writer -.-> CDC
    UtilsApp -.-> CDC
    ConfigApp -.-> CDC

    %% Data Lake Storage
    GCSRaw[(GCS Raw Bucket)]:::storage
    GCSBronze[(GCS Bronze Bucket)]:::storage
    FullLoad -->|"write raw"| GCSRaw
    CDC -->|"write raw"| GCSRaw

    %% Databricks Workspace
    subgraph "Databricks Workspace"
        DatabrRaw[(Raw Tables)]:::storage
        DatabrBronze[(Bronze Layer)]:::storage
        DatabrSilver[(Silver Layer)]:::storage
    end
    GCSRaw -->|"ingest bronze"| DatabrRaw
    DatabrRaw -->|"transform"| DatabrBronze
    DatabrBronze -->|"write bronze"| GCSBronze
    GCSBronze -->|"ingest silver"| DatabrSilver

    %% Bronze Transformations
    BronzeNB["Bronze Notebooks\n(src/bronze)"]:::processing
    click BronzeNB "https://github.com/rmendes1/de_music_data_lake/blob/main/src/bronze/ingestion.ipynb"
    click BronzeNB "https://github.com/rmendes1/de_music_data_lake/blob/main/src/bronze/ingestion_refactored.ipynb"
    DatabrRaw --> BronzeNB --> DatabrBronze

    %% Silver Transformations
    SilverNS["Silver Scripts\n(src/silver)"]:::processing
    click SilverNS "https://github.com/rmendes1/de_music_data_lake/blob/main/src/silver/albums.sql"
    click SilverNS "https://github.com/rmendes1/de_music_data_lake/blob/main/src/silver/albums_genres.sql"
    click SilverNS "https://github.com/rmendes1/de_music_data_lake/blob/main/src/silver/artists.sql"
    click SilverNS "https://github.com/rmendes1/de_music_data_lake/blob/main/src/silver/genres.sql"
    click SilverNS "https://github.com/rmendes1/de_music_data_lake/blob/main/src/silver/tracks.sql"
    click SilverNS "https://github.com/rmendes1/de_music_data_lake/blob/main/src/silver/ingestion.ipynb"
    DatabrBronze --> SilverNS --> DatabrSilver

    %% Workflows
    WF["Workflows Definitions\n(src/workflows)"]:::processing
    click WF "https://github.com/rmendes1/de_music_data_lake/blob/main/src/workflows/main.py"
    click WF "https://github.com/rmendes1/de_music_data_lake/blob/main/src/workflows/requirements.txt"
    click WF "https://github.com/rmendes1/de_music_data_lake/blob/main/src/workflows/create/music_data.json"
    click WF "https://github.com/rmendes1/de_music_data_lake/blob/main/src/workflows/reset/music_data.json"
    WF --> DatabrBronze
    WF --> DatabrSilver

    %% Analytics Consumers
    BI["BI Tools / Analytics"]:::external
    DatabrSilver --> BI

    %% Styles
    classDef external fill:#d3d3d3,stroke:#333;
    classDef ingestion fill:#add8e6,stroke:#333;
    classDef processing fill:#ffa500,stroke:#333;
    classDef storage fill:#98fb98,stroke:#333;
```
