How a-vijai/spark_structured_streaming works
Project type and purpose
• This is a simple event‐driven, real‐time data‐processing pipeline built on Apache Spark Structured Streaming.
• The two “producer” scripts generate event streams (orders and payments), and the “join_stream.py” script reads those two streams, performs a time‐aware join, and writes the enriched output to a sink.
File structure
• Top‐level files:
• orders_producer.py – publishes order events to a Kafka topic (e.g. “orders”)
• payments_producer.py – publishes payment events to a Kafka topic (e.g. “payments”)
• join_stream.py – a Spark Structured Streaming job that reads both topics, joins them, and writes results out
• README.md – project name (“spark_structured_streaming”)
• No subdirectories indicates a single‐module/lightweight demo or POC.
• Absence of build files suggests it’s run directly with spark-submit / python.
README
• Confirms this is purely about Spark Structured Streaming (no other frameworks or services documented).
a. Main components:
1. Two Event Producers
– orders_producer.py
– payments_producer.py
2. Kafka Cluster
– Topic: “orders”
– Topic: “payments”
3. Spark Structured Streaming Application (join_stream.py)
– Reads the two Kafka topics as streaming DataFrames
– Applies watermarks and a streaming join on order_id
4. Sink for Joined Stream
– Could be a Kafka topic (e.g. “joined”), console output, HDFS, or a database
5. (Optional) Monitoring / Metrics System
– Spark UI / Kafka monitoring / logging
b. Relationships & data flows:
– orders_producer.py → Kafka “orders” topic
– payments_producer.py → Kafka “payments” topic
– Spark job ← reads both topics ← Kafka cluster
– Spark job → transformation (join stream)
– Spark job → output sink
c. Architectural patterns & design principles:
– Stream‐processing (Lambda‐style micro‐batch under the hood of Structured Streaming)
– Fault tolerance via Kafka’s offset tracking and Spark’s checkpointing
– Event‐time handling with watermarks to bound state
d. Key technologies/frameworks:
– Python scripts using kafka-python or confluent-kafka for producers
– Apache Kafka for durable event transport
– Apache Spark Structured Streaming for real‐time joins
– (Optional) Sink connector (Kafka, file, JDBC)