How rangareddy/datalake-playground works
This is a run-only, single-machine lakehouse playground: users start a Docker Compose profile, then submit Spark work against the stack. The core profile provides MinIO-backed storage, Hive Metastore, Spark with Iceberg, Hudi and Delta, and Postgres; the all profile adds Kafka-related services, MySQL, Trino and Jupyter. The graph preserves these documented capabilities and the sample lakehouse check. Compose internals and the start script were not excerpted, so their links reflect README statements rather than inspected implementation details; database SQL files are shown without inferred runtime connections.
Main entry points: docker-compose_all.yml All profile · docker-compose.yml Core profile · run_datalake.sh Stack start · lakehouse_check.py Format check
Components 19
Stack orchestration
- Stack start
run_datalake.sh - Core profile
docker-compose.yml - All profile
docker-compose_all.yml
Core lakehouse
- MinIO S3
docker-compose.yml - Hive Metastore
docker-compose.yml - Spark and formats
docker-compose.yml - Postgres
docker-compose.yml - Format check
lakehouse_check.py
All-profile services
- Kafka stack
docker-compose_all.yml - Kafka Connect
docker-compose_all.yml - MySQL
docker-compose_all.yml - Trino
docker-compose_all.yml - Jupyter Lab
docker-compose_all.yml
Database bootstrap
- MySQL SQL setup
db_scripts/mysql/db_init.sql - MySQL employees
db_scripts/mysql/employees.sql - Postgres SQL setup
db_scripts/postgres/db_init.sql - Postgres employees
db_scripts/postgres/employees.sql - Postgres multi-table data
db_scripts/postgres/multi_table_data.sql
Other components
- Engineer
Connections 15
- Engineer → Stack start: starts
README.md - Stack start → Core profile: starts core
README.md - Stack start → All profile: starts all
README.md - Core profile → MinIO S3: runs
README.md - Core profile → Hive Metastore: runs
README.md - Core profile → Spark and formats: runs
README.md - Core profile → Postgres: runs
README.md - All profile → Kafka stack: adds broker
README.md - All profile → Kafka Connect: adds connector
README.md - All profile → MySQL: adds database
README.md - All profile → Trino: adds query engine
README.md - All profile → Jupyter Lab: adds notebook
README.md - Spark and formats → Format check: runs checks
README.md - Format check → Hive Metastore: uses catalog
lakehouse_check.py - Format check → MinIO S3: writes tables
lakehouse_check.py