# a-vijai/spark_structured_streaming architecture

> Architecture diagram of the GitHub repository a-vijai/spark_structured_streaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/a-vijai/spark_structured_streaming
- Repository: https://github.com/a-vijai/spark_structured_streaming

## Overview

##### Project type and purpose
- This is a simple event‐driven, real‐time data‐processing pipeline built on Apache Spark Structured Streaming.
- The two “producer” scripts generate event streams (orders and payments), and the “join_stream.py” script reads those two streams, performs a time‐aware join, and writes the enriched output to a sink.

##### File structure
- Top‐level files:
- orders_producer.py – publishes order events to a Kafka topic (e.g. “orders”)
- payments_producer.py – publishes payment events to a Kafka topic (e.g. “payments”)
- join_stream.py – a Spark Structured Streaming job that reads both topics, joins them, and writes results out
- README.md – project name (“spark_structured_streaming”)
- No subdirectories indicates a single‐module/lightweight demo or POC.
- Absence of build files suggests it’s run directly with spark-submit / python.

##### README
- Confirms this is purely about Spark Structured Streaming (no other frameworks or services documented).

a. Main components:
1. Two Event Producers
– orders_producer.py
– payments_producer.py
2. Kafka Cluster
– Topic: “orders”
– Topic: “payments”
3. Spark Structured Streaming Application (join_stream.py)
– Reads the two Kafka topics as streaming DataFrames
– Applies watermarks and a streaming join on order_id
4. Sink for Joined Stream
– Could be a Kafka topic (e.g. “joined”), console output, HDFS, or a database
5. (Optional) Monitoring / Metrics System
– Spark UI / Kafka monitoring / logging
b. Relationships & data flows:
– orders_producer.py → Kafka “orders” topic
– payments_producer.py → Kafka “payments” topic
– Spark job ← reads both topics ← Kafka cluster
– Spark job → transformation (join stream)
– Spark job → output sink
c. Architectural patterns & design principles:
– Stream‐processing (Lambda‐style micro‐batch under the hood of Structured Streaming)
– Fault tolerance via Kafka’s offset tracking and Spark’s checkpointing
– Event‐time handling with watermarks to bound state
d. Key technologies/frameworks:
– Python scripts using kafka-python or confluent-kafka for producers
– Apache Kafka for durable event transport
– Apache Spark Structured Streaming for real‐time joins
– (Optional) Sink connector (Kafka, file, JDBC)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Ingest Layer"
        OPO["Orders Producer"]:::producer
        PPO["Payments Producer"]:::producer
    end

    subgraph "Messaging Layer"
        subgraph "Kafka Cluster"
            KT1["orders topic"]:::messaging
            KT2["payments topic"]:::messaging
        end
    end

    subgraph "Processing Layer"
        SJ["Spark Structured Streaming Job\n(inner join on order_id, 10min watermark)"]:::processing
    end

    subgraph "Sink Layer"
        SNK["Joined Output Sink"]:::storage
    end

    OPO -->|publishes events| KT1
    PPO -->|publishes events| KT2
    KT1 -->|stream read| SJ
    KT2 -->|stream read| SJ
    SJ -->|writes enriched stream| SNK

    click OPO "https://github.com/a-vijai/spark_structured_streaming/blob/main/orders_producer.py"
    click PPO "https://github.com/a-vijai/spark_structured_streaming/blob/main/payments_producer.py"
    click SJ "https://github.com/a-vijai/spark_structured_streaming/blob/main/join_stream.py"

    classDef producer fill:#9f6,stroke:#333,stroke-width:1px
    classDef messaging fill:#fc6,stroke:#333,stroke-width:1px
    classDef processing fill:#6cf,stroke:#333,stroke-width:1px
    classDef storage fill:#ccc,stroke:#333,stroke-width:1px
```
