# thomazrossito/sparkstream-delta-minio architecture

> Architecture diagram of the GitHub repository thomazrossito/sparkstream-delta-minio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thomazrossito/sparkstream-delta-minio
- Repository: https://github.com/thomazrossito/sparkstream-delta-minio

## Overview

This repository is a small Spark Structured Streaming ingestion application: it discovers a JSON schema from MinIO’s raw bucket, reads the stream, adds Bronze metadata columns, writes Delta files with a checkpoint, then optimizes and counts the Bronze output. The graph follows the calls in the two excerpted files; it does not infer unshown Spark or MinIO internals.

Main entry points: [`code/raw2Bronze.py`](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/raw2Bronze.py) (Raw to Bronze job), [`code/functions.py`](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py) (Schema evolution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Pipeline runtime"]
  node_ingest["Raw to Bronze job<br/>[raw2Bronze.py]"]
end

subgraph group_helpers["Spark helpers"]
  node_session["Spark session<br/>[functions.py]"]
  node_schema["Schema evolution<br/>[functions.py]"]
  node_reader["JSON stream reader<br/>[functions.py]"]
  node_transform["Bronze transformation<br/>[functions.py]"]
  node_writer["Delta stream writer<br/>[functions.py]"]
  node_optimizer["Delta optimizer<br/>[functions.py]"]
end

subgraph group_storage["Data lake"]
  node_minio[("MinIO buckets")]
  node_raw["Raw JSON data"]
  node_bronze[("Bronze Delta data")]
end

node_operator(("Pipeline operator"))

node_operator -->|"starts"| node_ingest
node_ingest -->|"creates"| node_session
node_session -->|"configures access"| node_minio
node_ingest -->|"gets schema"| node_schema
node_schema -->|"creates"| node_session
node_schema -->|"lists objects"| node_minio
node_ingest -->|"reads stream"| node_reader
node_reader -->|"loads JSON"| node_raw
node_ingest -->|"transforms"| node_transform
node_ingest -->|"configures writer"| node_writer
node_writer -->|"writes Delta"| node_bronze
node_writer -->|"stores checkpoint"| node_minio
node_ingest -->|"optimizes path"| node_optimizer
node_optimizer -->|"optimizes Delta"| node_bronze
node_ingest -->|"counts rows"| node_bronze

click node_ingest "https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/raw2Bronze.py"
click node_session "https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py"
click node_schema "https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py"
click node_reader "https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py"
click node_transform "https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py"
click node_writer "https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py"
click node_optimizer "https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest toneBlue
class node_session,node_schema,node_reader,node_transform,node_writer,node_optimizer toneAmber
class node_minio,node_raw,node_bronze toneMint
class node_operator toneIndigo
```

## Components

### Pipeline runtime

- [Raw to Bronze job](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/raw2Bronze.py): `code/raw2Bronze.py`

### Spark helpers

- [Spark session](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py): `code/functions.py`
- [Schema evolution](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py): `code/functions.py`
- [JSON stream reader](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py): `code/functions.py`
- [Bronze transformation](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py): `code/functions.py`
- [Delta stream writer](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py): `code/functions.py`
- [Delta optimizer](https://github.com/thomazrossito/sparkstream-delta-minio/blob/main/code/functions.py): `code/functions.py`

### Data lake

- **MinIO buckets**
- **Raw JSON data**
- **Bronze Delta data**

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Raw to Bronze job: starts (evidence: `README.md`)
- Raw to Bronze job → Spark session: creates (evidence: `code/raw2Bronze.py`)
- Spark session → MinIO buckets: configures access (evidence: `code/functions.py`)
- Raw to Bronze job → Schema evolution: gets schema (evidence: `code/raw2Bronze.py`)
- Schema evolution → Spark session: creates (evidence: `code/functions.py`)
- Schema evolution → MinIO buckets: lists objects (evidence: `code/functions.py`)
- Raw to Bronze job → JSON stream reader: reads stream (evidence: `code/raw2Bronze.py`)
- JSON stream reader → Raw JSON data: loads JSON
- Raw to Bronze job → Bronze transformation: transforms (evidence: `code/raw2Bronze.py`)
- Raw to Bronze job → Delta stream writer: configures writer (evidence: `code/raw2Bronze.py`)
- Delta stream writer → Bronze Delta data: writes Delta
- Delta stream writer → MinIO buckets: stores checkpoint
- Raw to Bronze job → Delta optimizer: optimizes path (evidence: `code/raw2Bronze.py`)
- Delta optimizer → Bronze Delta data: optimizes Delta (evidence: `code/functions.py`)
- Raw to Bronze job → Bronze Delta data: counts rows (evidence: `code/raw2Bronze.py`)
