# devlucho/spark-streaming-y-kafka architecture

> Architecture diagram of the GitHub repository devlucho/spark-streaming-y-kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devlucho/spark-streaming-y-kafka
- Repository: https://github.com/devlucho/spark-streaming-y-kafka

## Overview

##### Identifying the Main Components
- Kafka Producer: Represented by the file “kafka_producer.py” which simulates sensor data (temperature, humidity, and timestamp) and publishes it to a Kafka topic named "sensor_data".
- Kafka Infrastructure: Although not part of the source code, the README details the setup of Apache Kafka and ZooKeeper. This should be illustrated as crucial external components managing message brokering and coordination.
- Spark Streaming Consumer: Represented by “spark_streaming_consumer.py”, this component consumes data from Kafka, performs window-based aggregations (calculating average temperature and humidity per sensor over one-minute windows), and outputs processed data to the Spark console.
- External Dependencies: Include infrastructure elements such as ZooKeeper, which manages Kafka, and Spark’s integration with Kafka (via spark-submit with relevant packages).

##### Architectural Patterns and Design Principles
- Real-time streaming architecture where the data production and consumption happens concurrently.
- Decoupling between the producer and consumer, enabled by Kafka acting as a message broker.
- Use of microservices-like separation: one script dedicated to data generation and another for data processing.

##### Including Relevant Technologies
- Apache Kafka: Indicate its use for message brokering; underline the topic ("sensor_data") creation step.
- Apache ZooKeeper: Mention its role in managing Kafka services.
- Apache Spark with Spark Streaming: Identify Spark’s role in consuming data from Kafka and processing it in defined time windows.

##### Additional Notes
- That the producer must be running before the consumer to guarantee data availability.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Generation"
        KP["Kafka Producer (kafka_producer.py)"]:::producer
    end

    subgraph "Messaging Infrastructure"
        K["Kafka Broker (Apache Kafka)<br>Topic: sensor_data"]:::broker
        Z["ZooKeeper"]:::external
    end

    subgraph "Data Processing"
        SC["Spark Streaming Consumer (spark_streaming_consumer.py)"]:::consumer
        SP["Spark Console Output"]:::spark
    end

    KP -->|"publishes"| K
    Z -->|"manages"| K
    K -->|"streamsData"| SC
    SC -->|"displays"| SP

    click KP "https://github.com/devlucho/spark-streaming-y-kafka/blob/main/kafka_producer.py"
    click SC "https://github.com/devlucho/spark-streaming-y-kafka/blob/main/spark_streaming_consumer.py"

    classDef producer fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef broker fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef external fill:#FFCC00,stroke:#000,stroke-width:2px;
    classDef consumer fill:#FFDAB9,stroke:#000,stroke-width:2px;
    classDef spark fill:#FFB6C1,stroke:#000,stroke-width:2px;
```
