How rmendes1/de_music_data_lake works
Project Type and Purpose
• This is a data‐engineering project implementing a Data Lake for music metadata.
• It ingests data from external Music APIs, performs both full loads and CDC ingestion via Debezium/Kafka into cloud storage (GCS), and organizes data into Databricks catalogs (raw, bronze, then further transformations into silver).
• Core technologies: Apache Spark, Docker (Compose), Debezium, Kafka, PostgreSQL, GCS, Databricks.
File Structure
Top-level components:
• api_consumer/ – downloads initial music data from public APIs into JSON or directly into Postgres.
• queries/ – SQL scripts to bootstrap or reset the Postgres source.
• docker-compose.yml, Dockerfile, entrypoint.sh – define a multi-service environment: Postgres, Kafka (with Zookeeper), Debezium connector, Spark, etc.
• spark_files/ – Spark applications for:
– full_load.py: one-time bulk ingestion into GCS
– cdc_ingest.py: continuous ingestion of CDC events from Kafka
– manage_offset.py: offset management for reliable consumption
– writer.py, utils.py, plus config.py
Also includes spark-defaults.conf, Postgres JDBC driver, Python requirements.
• src/bronze & src/silver – notebooks and SQL transform layers for Databricks:
– bronze: notebooks or scripts ingesting from GCS to Databricks bronze tables
– silver: .sql scripts for curated tables
• src/lib – shared Python utilities for ingestion, preprocessing.
• src/workflows – Python workflows (create/reset) for orchestrating tasks in Databricks workflows (JSON definitions for jobs).
Key Insights from the README
• CDC ingest uses Debezium + Kafka.
• Full load: uses Spark to extract entire Postgres data and write to GCS.
• Bronze layer ingestion and silver transformations run in Databricks.
• Docker Compose ties together Postgres, Kafka, Debezium, and Spark driver containers.
Main Components
A. External Data Sources
• Music APIs → api_consumer container
B. Source Database
• PostgreSQL container (initialized via queries/insert_data_pg.sql)
C. Change Data Capture Pipeline
• Debezium connector (configured via debezium.json)
• Apache Kafka cluster & CDC topics
D. Batch Ingestion (Full Load)
• Spark Driver (full_load.py) reads Postgres → writes Parquet/Delta to GCS (raw area)
E. Streaming Ingestion (CDC)
• Spark Driver (cdc_ingest.py) reads from Kafka topics → writes incremental Parquet/Delta to GCS (raw area)
• Offset management by manage_offset.py
F. Data Lake Storage
• Google Cloud Storage buckets:
– /raw/music_data/...
– /bronze/music_data/...
G. Databricks Workspace
• Raw → ingested as unmanaged tables
• Bronze → refined via notebooks in src/bronze
• Silver → curated tables via SQL scripts in src/silver
• Workflows orchestrated by src/workflows definitions
H. Consumers / Analytics
• BI tools or end-user queries against Silver tables
Relationships & Data Flows
• Postgres → Debezium → Kafka
• Postgres → Spark Full Load → GCS Raw
• Kafka → Spark CDC Ingest → GCS Raw
• GCS Raw → Databricks Bronze layer → GCS Bronze
• GCS Bronze → Databricks Silver layer → final tables
• Downstream analytics consume Silver
Architectural Patterns & Design Principles
• Lambda architecture: batch (full load) + stream (CDC) layered into a unified data lake
• Separation of concerns:
– Ingestion (api_consumer, Spark apps)
– Storage (GCS buckets)
– Serving/Transformation (Databricks notebooks & SQL)
• Modular, containerized services for portability and reproducibility
Checklist for the Principal Software Engineer
• Components listed in step 4.
• Related containers (Postgres + Debezium + Kafka) in a “Docker Compose” boundary.
• External dependencies (Music APIs, GCS, Databricks).