# abeltavares/real-time-data-pipeline architecture

> Architecture diagram of the GitHub repository abeltavares/real-time-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abeltavares/real-time-data-pipeline
- Repository: https://github.com/abeltavares/real-time-data-pipeline

## Overview

This repository demonstrates a real-time clickstream pipeline: generated events are streamed through Kafka, filtered by Flink, stored as Iceberg tables on MinIO, queried with Trino, and visualized in Superset. The README documents Kafka/ZooKeeper coordination and the end-to-end integrations; the sampled source confirms the producer’s event generation and Kafka publishing. Flink SQL, service wiring, and most integrations were not sampled, so those relationships follow the README rather than inspected implementation.

Main entry points: [`docker-compose.yaml`](https://github.com/abeltavares/real-time-data-pipeline/blob/main/docker-compose.yaml) (Kafka topic), [`flink/sql-jobs/clickstream-filtering.sql`](https://github.com/abeltavares/real-time-data-pipeline/blob/main/flink/sql-jobs/clickstream-filtering.sql) (Iceberg tables), [`trino/iceberg.properties`](https://github.com/abeltavares/real-time-data-pipeline/blob/main/trino/iceberg.properties) (Trino query engine), [`superset/superset_config.py`](https://github.com/abeltavares/real-time-data-pipeline/blob/main/superset/superset_config.py) (Superset dashboards)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Event ingestion"]
  node_producer["Clickstream producer<br/>[producer.py]"]
  node_kafka["Kafka topic"]
  node_zookeeper["ZooKeeper"]
end

subgraph group_process["Stream processing"]
  node_flink["Flink SQL job"]
end

subgraph group_lake["Lakehouse storage"]
  node_iceberg[("Iceberg tables")]
  node_minio[("MinIO storage")]
end

subgraph group_analytics["Query and visualization"]
  node_trino["Trino query engine<br/>[iceberg.properties]"]
  node_superset["Superset dashboards<br/>[superset_config.py]"]
end

node_engineer(("Data engineer"))

node_producer -->|"publishes events"| node_kafka
node_zookeeper -->|"coordinates broker"| node_kafka
node_kafka -->|"streams events"| node_flink
node_flink -->|"filters and writes"| node_iceberg
node_iceberg -->|"stores table data"| node_minio
node_trino -->|"queries tables"| node_iceberg
node_superset -->|"queries data"| node_trino
node_engineer -->|"runs SQL"| node_trino
node_engineer -->|"views dashboards"| node_superset

click node_producer "https://github.com/abeltavares/real-time-data-pipeline/blob/main/producer/producer.py"
click node_kafka "https://github.com/abeltavares/real-time-data-pipeline/blob/main/docker-compose.yaml"
click node_zookeeper "https://github.com/abeltavares/real-time-data-pipeline/blob/main/docker-compose.yaml"
click node_flink "https://github.com/abeltavares/real-time-data-pipeline/blob/main/flink/sql-jobs/clickstream-filtering.sql"
click node_iceberg "https://github.com/abeltavares/real-time-data-pipeline/blob/main/flink/sql-jobs/clickstream-filtering.sql"
click node_minio "https://github.com/abeltavares/real-time-data-pipeline/blob/main/docker-compose.yaml"
click node_trino "https://github.com/abeltavares/real-time-data-pipeline/blob/main/trino/iceberg.properties"
click node_superset "https://github.com/abeltavares/real-time-data-pipeline/blob/main/superset/superset_config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_kafka,node_zookeeper toneBlue
class node_flink toneAmber
class node_iceberg,node_minio toneMint
class node_trino,node_superset toneRose
class node_engineer toneIndigo
```

## Components

### Event ingestion

- [Clickstream producer](https://github.com/abeltavares/real-time-data-pipeline/blob/main/producer/producer.py): `producer/producer.py`
- [Kafka topic](https://github.com/abeltavares/real-time-data-pipeline/blob/main/docker-compose.yaml): `docker-compose.yaml`
- [ZooKeeper](https://github.com/abeltavares/real-time-data-pipeline/blob/main/docker-compose.yaml): `docker-compose.yaml`

### Stream processing

- [Flink SQL job](https://github.com/abeltavares/real-time-data-pipeline/blob/main/flink/sql-jobs/clickstream-filtering.sql): `flink/sql-jobs/clickstream-filtering.sql`

### Lakehouse storage

- [Iceberg tables](https://github.com/abeltavares/real-time-data-pipeline/blob/main/flink/sql-jobs/clickstream-filtering.sql): `flink/sql-jobs/clickstream-filtering.sql`
- [MinIO storage](https://github.com/abeltavares/real-time-data-pipeline/blob/main/docker-compose.yaml): `docker-compose.yaml`

### Query and visualization

- [Trino query engine](https://github.com/abeltavares/real-time-data-pipeline/blob/main/trino/iceberg.properties): `trino/iceberg.properties`
- [Superset dashboards](https://github.com/abeltavares/real-time-data-pipeline/blob/main/superset/superset_config.py): `superset/superset_config.py`

### Other components

- **Data engineer**

## Connections

- Clickstream producer → Kafka topic: publishes events
- ZooKeeper → Kafka topic: coordinates broker
- Kafka topic → Flink SQL job: streams events
- Flink SQL job → Iceberg tables: filters and writes
- Iceberg tables → MinIO storage: stores table data
- Trino query engine → Iceberg tables: queries tables
- Superset dashboards → Trino query engine: queries data
- Data engineer → Trino query engine: runs SQL
- Data engineer → Superset dashboards: views dashboards