How dhia4884/tp06-real-time-processing works
This repository demonstrates two independent real-time processing workflows: campus events are produced to Kafka and aggregated by Spark, while department check-ins are produced to Kafka and aggregated with Flink SQL. The graph preserves the documented parsing, event-time windowing, watermark, and aggregation stages. Flink SQL files are represented from the README and tree; their contents were not sampled, so those relationships rely on README evidence. Kafka is an external service, and the graph does not infer an unsampled Flink job launcher.
Main entry points: create_table.sql Check-ins Table · spark_streaming.py Kafka Ingestion · department_checkins.sql Department Windows · producer.py Campus Producer
Components 11
Spark Streaming
- Campus Producer
Lab-A-Spark-Streaming/producer.py - Kafka Ingestion
Lab-A-Spark-Streaming/spark_streaming.py - Parse Events
Lab-A-Spark-Streaming/spark_streaming.py - Window Counts
Lab-A-Spark-Streaming/spark_streaming.py - Console Results
Lab-A-Spark-Streaming/spark_streaming.py
Flink SQL
- Check-in Producer
Lab-B-Flink-SQL/producer_flink.py - Check-ins Table
Lab-B-Flink-SQL/sql/create_table.sql - Department Windows
Lab-B-Flink-SQL/sql/department_checkins.sql - Kafka Connector
Lab-B-Flink-SQL/lib/flink-sql-connector-kafka-3.3.0-1.20.jar - Check-in Results
Kafka Messaging
- Kafka Broker
Connections 10
- Campus Producer → Kafka Broker: sends events
Lab-A-Spark-Streaming/producer.py - Kafka Broker → Kafka Ingestion: streams topic
Lab-A-Spark-Streaming/spark_streaming.py - Kafka Ingestion → Parse Events: parses JSON
Lab-A-Spark-Streaming/spark_streaming.py - Parse Events → Window Counts: groups events
Lab-A-Spark-Streaming/spark_streaming.py - Window Counts → Console Results: writes updates
Lab-A-Spark-Streaming/spark_streaming.py - Check-in Producer → Kafka Broker: sends check-ins
Lab-B-Flink-SQL/producer_flink.py - Kafka Broker → Check-ins Table: reads topic
README.md - Check-ins Table → Department Windows: queries table
README.md - Department Windows → Check-in Results: produces counts
README.md - Kafka Connector → Check-ins Table: enables Kafka