How devlucho/spark-streaming-y-kafka works
Identifying the Main Components
• Kafka Producer: Represented by the file “kafka_producer.py” which simulates sensor data (temperature, humidity, and timestamp) and publishes it to a Kafka topic named "sensor_data".
• Kafka Infrastructure: Although not part of the source code, the README details the setup of Apache Kafka and ZooKeeper. This should be illustrated as crucial external components managing message brokering and coordination.
• Spark Streaming Consumer: Represented by “spark_streaming_consumer.py”, this component consumes data from Kafka, performs window-based aggregations (calculating average temperature and humidity per sensor over one-minute windows), and outputs processed data to the Spark console.
• External Dependencies: Include infrastructure elements such as ZooKeeper, which manages Kafka, and Spark’s integration with Kafka (via spark-submit with relevant packages).
Architectural Patterns and Design Principles
• Real-time streaming architecture where the data production and consumption happens concurrently.
• Decoupling between the producer and consumer, enabled by Kafka acting as a message broker.
• Use of microservices-like separation: one script dedicated to data generation and another for data processing.
Including Relevant Technologies
• Apache Kafka: Indicate its use for message brokering; underline the topic ("sensor_data") creation step.
• Apache ZooKeeper: Mention its role in managing Kafka services.
• Apache Spark with Spark Streaming: Identify Spark’s role in consuming data from Kafka and processing it in defined time windows.
Additional Notes
• That the producer must be running before the consumer to guarantee data availability.