# rangareddy/datalake-playground architecture

> Architecture diagram of the GitHub repository rangareddy/datalake-playground, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rangareddy/datalake-playground
- Repository: https://github.com/rangareddy/datalake-playground

## Overview

This is a run-only, single-machine lakehouse playground: users start a Docker Compose profile, then submit Spark work against the stack. The core profile provides MinIO-backed storage, Hive Metastore, Spark with Iceberg, Hudi and Delta, and Postgres; the all profile adds Kafka-related services, MySQL, Trino and Jupyter. The graph preserves these documented capabilities and the sample lakehouse check. Compose internals and the start script were not excerpted, so their links reflect README statements rather than inspected implementation details; database SQL files are shown without inferred runtime connections.

Main entry points: [`docker-compose_all.yml`](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml) (All profile), [`docker-compose.yml`](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml) (Core profile), [`run_datalake.sh`](https://github.com/rangareddy/datalake-playground/blob/master/run_datalake.sh) (Stack start), [`lakehouse_check.py`](https://github.com/rangareddy/datalake-playground/blob/master/lakehouse_check.py) (Format check)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Stack orchestration"]
  node_start["Stack start<br/>[run_datalake.sh]"]
  node_core["Core profile<br/>[docker-compose.yml]"]
  node_all["All profile"]
end

subgraph group_lakehouse["Core lakehouse"]
  node_minio[("MinIO S3<br/>[docker-compose.yml]")]
  node_hive[("Hive Metastore<br/>[docker-compose.yml]")]
  node_spark["Spark and formats<br/>[docker-compose.yml]"]
  node_postgres[("Postgres<br/>[docker-compose.yml]")]
  node_check["Format check<br/>[lakehouse_check.py]"]
end

subgraph group_integrations["All-profile services"]
  node_kafka["Kafka stack"]
  node_connect["Kafka Connect"]
  node_mysql[("MySQL")]
  node_trino["Trino"]
  node_jupyter["Jupyter Lab"]
end

subgraph group_database["Database bootstrap"]
  node_mysqlsql["MySQL SQL setup<br/>[db_init.sql]"]
  node_mysqlrows["MySQL employees<br/>[employees.sql]"]
  node_pgsql["Postgres SQL setup<br/>[db_init.sql]"]
  node_pgemployees["Postgres employees<br/>[employees.sql]"]
  node_pgmulti["Postgres multi-table data"]
end

node_user(("Engineer"))

node_user -->|"starts"| node_start
node_start -->|"starts core"| node_core
node_start -->|"starts all"| node_all
node_core -->|"runs"| node_minio
node_core -->|"runs"| node_hive
node_core -->|"runs"| node_spark
node_core -->|"runs"| node_postgres
node_all -->|"adds broker"| node_kafka
node_all -->|"adds connector"| node_connect
node_all -->|"adds database"| node_mysql
node_all -->|"adds query engine"| node_trino
node_all -->|"adds notebook"| node_jupyter
node_spark -->|"runs checks"| node_check
node_check -->|"uses catalog"| node_hive
node_check -->|"writes tables"| node_minio

click node_start "https://github.com/rangareddy/datalake-playground/blob/master/run_datalake.sh"
click node_core "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml"
click node_all "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml"
click node_minio "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml"
click node_hive "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml"
click node_spark "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml"
click node_postgres "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml"
click node_check "https://github.com/rangareddy/datalake-playground/blob/master/lakehouse_check.py"
click node_kafka "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml"
click node_connect "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml"
click node_mysql "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml"
click node_trino "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml"
click node_jupyter "https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml"
click node_mysqlsql "https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/mysql/db_init.sql"
click node_mysqlrows "https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/mysql/employees.sql"
click node_pgsql "https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/postgres/db_init.sql"
click node_pgemployees "https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/postgres/employees.sql"
click node_pgmulti "https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/postgres/multi_table_data.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_start,node_core,node_all toneBlue
class node_minio,node_hive,node_spark,node_postgres,node_check toneAmber
class node_kafka,node_connect,node_mysql,node_trino,node_jupyter toneMint
class node_mysqlsql,node_mysqlrows,node_pgsql,node_pgemployees,node_pgmulti toneRose
class node_user toneIndigo
```

## Components

### Stack orchestration

- [Stack start](https://github.com/rangareddy/datalake-playground/blob/master/run_datalake.sh): `run_datalake.sh`
- [Core profile](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml): `docker-compose.yml`
- [All profile](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml): `docker-compose_all.yml`

### Core lakehouse

- [MinIO S3](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml): `docker-compose.yml`
- [Hive Metastore](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml): `docker-compose.yml`
- [Spark and formats](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml): `docker-compose.yml`
- [Postgres](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose.yml): `docker-compose.yml`
- [Format check](https://github.com/rangareddy/datalake-playground/blob/master/lakehouse_check.py): `lakehouse_check.py`

### All-profile services

- [Kafka stack](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml): `docker-compose_all.yml`
- [Kafka Connect](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml): `docker-compose_all.yml`
- [MySQL](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml): `docker-compose_all.yml`
- [Trino](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml): `docker-compose_all.yml`
- [Jupyter Lab](https://github.com/rangareddy/datalake-playground/blob/master/docker-compose_all.yml): `docker-compose_all.yml`

### Database bootstrap

- [MySQL SQL setup](https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/mysql/db_init.sql): `db_scripts/mysql/db_init.sql`
- [MySQL employees](https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/mysql/employees.sql): `db_scripts/mysql/employees.sql`
- [Postgres SQL setup](https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/postgres/db_init.sql): `db_scripts/postgres/db_init.sql`
- [Postgres employees](https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/postgres/employees.sql): `db_scripts/postgres/employees.sql`
- [Postgres multi-table data](https://github.com/rangareddy/datalake-playground/blob/master/db_scripts/postgres/multi_table_data.sql): `db_scripts/postgres/multi_table_data.sql`

### Other components

- **Engineer**

## Connections

- Engineer → Stack start: starts (evidence: `README.md`)
- Stack start → Core profile: starts core (evidence: `README.md`)
- Stack start → All profile: starts all (evidence: `README.md`)
- Core profile → MinIO S3: runs (evidence: `README.md`)
- Core profile → Hive Metastore: runs (evidence: `README.md`)
- Core profile → Spark and formats: runs (evidence: `README.md`)
- Core profile → Postgres: runs (evidence: `README.md`)
- All profile → Kafka stack: adds broker (evidence: `README.md`)
- All profile → Kafka Connect: adds connector (evidence: `README.md`)
- All profile → MySQL: adds database (evidence: `README.md`)
- All profile → Trino: adds query engine (evidence: `README.md`)
- All profile → Jupyter Lab: adds notebook (evidence: `README.md`)
- Spark and formats → Format check: runs checks (evidence: `README.md`)
- Format check → Hive Metastore: uses catalog (evidence: `lakehouse_check.py`)
- Format check → MinIO S3: writes tables (evidence: `lakehouse_check.py`)