How abeltavares/real-time-data-pipeline works
This repository demonstrates a real-time clickstream pipeline: generated events are streamed through Kafka, filtered by Flink, stored as Iceberg tables on MinIO, queried with Trino, and visualized in Superset. The README documents Kafka/ZooKeeper coordination and the end-to-end integrations; the sampled source confirms the producer’s event generation and Kafka publishing. Flink SQL, service wiring, and most integrations were not sampled, so those relationships follow the README rather than inspected implementation.
Main entry points: docker-compose.yaml Kafka topic · clickstream-filtering.sql Iceberg tables · iceberg.properties Trino query engine · superset_config.py Superset dashboards
Components 9
Event ingestion
- Clickstream producer
producer/producer.py - Kafka topic
docker-compose.yaml - ZooKeeper
docker-compose.yaml
Stream processing
- Flink SQL job
flink/sql-jobs/clickstream-filtering.sql
Lakehouse storage
- Iceberg tables
flink/sql-jobs/clickstream-filtering.sql - MinIO storage
docker-compose.yaml
Query and visualization
- Trino query engine
trino/iceberg.properties - Superset dashboards
superset/superset_config.py
Other components
- Data engineer
Connections 9
- Clickstream producer → Kafka topic: publishes events
- ZooKeeper → Kafka topic: coordinates broker
- Kafka topic → Flink SQL job: streams events
- Flink SQL job → Iceberg tables: filters and writes
- Iceberg tables → MinIO storage: stores table data
- Trino query engine → Iceberg tables: queries tables
- Superset dashboards → Trino query engine: queries data
- Data engineer → Trino query engine: runs SQL
- Data engineer → Superset dashboards: views dashboards