# dogukannulu/kafka_spark_structured_streaming architecture

> Architecture diagram of the GitHub repository dogukannulu/kafka_spark_structured_streaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dogukannulu/kafka_spark_structured_streaming
- Repository: https://github.com/dogukannulu/kafka_spark_structured_streaming

## Overview

##### Project Type and Purpose
- This project demonstrates a real-time streaming pipeline using Apache Spark Structured Streaming, Kafka, and Apache Airflow.
- The original design fetches random names from an API and sends them to Kafka (using an Airflow DAG), then consumes these messages using a PySpark job which writes them into a database (originally Cassandra).

##### Analysis of the File Structure
- Top-level files include the README, multiple Docker Compose files, and key Python scripts:
– stream_to_kafka.py: Contains the logic to fetch data from the API and send it to a Kafka topic.
– stream_to_kafka_dag.py: The Airflow DAG script that schedules the data ingestion every 10 seconds.
– spark_streaming.py: Implements the Spark Structured Streaming consumer to read messages from Kafka.
– Files such as docker-compose.yml and docker-compose-LocalExecutor.yml help deploy a multi-node Kafka cluster and an Airflow container, respectively.

##### Key Components and Their Interactions
- Data Producer / Ingestion:
- API: Provides random name data (represented by the sample response.json).
- Airflow DAG (stream_to_kafka_dag.py): Orchestrates the data ingestion by periodically triggering stream_to_kafka.py to pull API data and send it to Kafka.
- This part of the pipeline is clearly managed as a DAG. (So answer the question in the custom instruction: yes, the data ingestion is DAG-driven.)
- Messaging Middleware:
- Kafka Cluster: Represented by three nodes as configured in docker-compose.yml (e.g., kafka1, kafka2, kafka3) and includes a Kafka UI for monitoring.
- The data flows from the Airflow DAG (producer) into the Kafka topic (e.g., "random_names").
- Data Consumer / Processing:
- Spark Process: The spark_streaming.py script uses Spark Structured Streaming to consume data from the Kafka topic.
- Custom instruction question “spark process도 dag인가?” – In the current design, the Spark streaming process is not managed by Airflow as a DAG; it’s executed manually (via spark-submit) in the Spark container. However, you may integrate it as an additional DAG task if a fully orchestrated workflow is desired.
- Database Storage:
- MySQL: Replace the original Cassandra storage. The Spark consumer will write the processed data (random names) into a MySQL table. Ensure that the connection parameters and table definition are updated accordingly in the Spark script.

##### Architectural Patterns and Design Principles
- Containerization: Use Docker Compose for orchestrating the Kafka cluster, the Airflow environment, (and now MySQL instead of Cassandra), and Spark containers.
- Decoupled Components: Data ingestion is separated from the data processing (stream consumer), and communication occurs via Kafka topics.
- Orchestration with DAGs: The ingestion process is scheduled with an Airflow DAG, which allows for monitoring and scheduling. (At present, only the ingestion process is orchestrated; the Spark job is a manual or external submission.)
- Extensibility: If desired, both the data ingestion and Spark processing could be unified into a single orchestration framework by wrapping the Spark submit command within an Airflow DAG.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% API Source
    API["API Source (response.json)"]:::external

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        DAG["Airflow DAG (stream_to_kafka_dag.py)"]:::orchestration
        DP["Data Producer (stream_to_kafka.py)"]:::orchestration
    end

    %% Messaging Middleware
    Kafka["Kafka Cluster"]:::messaging

    %% Data Processing
    Spark["Spark Streaming Process (spark_streaming.py)\n(Not DAG-driven: triggered manually)"]:::processing

    %% Storage Component
    MySQL["MySQL Database"]:::storage

    %% Data Flow
    API -->|"fetches"| DAG
    DAG -->|"triggers"| DP
    DP -->|"publishes"| Kafka
    Kafka -->|"consumes"| Spark
    Spark -->|"writes"| MySQL

    %% Click Events
    click API "https://github.com/dogukannulu/kafka_spark_structured_streaming/blob/main/response.json"
    click DAG "https://github.com/dogukannulu/kafka_spark_structured_streaming/blob/main/stream_to_kafka_dag.py"
    click DP "https://github.com/dogukannulu/kafka_spark_structured_streaming/blob/main/stream_to_kafka.py"
    click Kafka "https://github.com/dogukannulu/kafka_spark_structured_streaming/blob/main/docker-compose.yml"
    click Spark "https://github.com/dogukannulu/kafka_spark_structured_streaming/blob/main/spark_streaming.py"
    click MySQL "https://github.com/dogukannulu/kafka_spark_structured_streaming/blob/main/docker-compose.yml"

    %% Styles
    classDef external fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef orchestration fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef messaging fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef processing fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef storage fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
```
