# mekwiset/realtime_data_streaming architecture

> Architecture diagram of the GitHub repository mekwiset/realtime_data_streaming, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mekwiset/realtime_data_streaming
- Repository: https://github.com/mekwiset/realtime_data_streaming

## Overview

##### Main Components
- External Data Source: Mark the randomuser API as the data origin.
- Orchestration and Scheduling: Highlight the Airflow DAG (located in the dags folder, specifically stream_to_kafka.py) which triggers the data ingestion.
- Data Ingestion (Producer): Identify the Kafka producer logic in jobs/kafka_stream.py and plugins/kafka_streamer.py.
- Data Processing (Consumer): Represent the Spark streaming job that consumes data from Kafka (jobs/spark_stream.py) and the corresponding plugins (spark_streamer.py).
- Data Storage: Show Cassandra as the final destination for processed data (with PostgreSQL mentioned as an intermediate store per the README, though the table creation script is in helpers).
- Container Environment: Include Docker and Docker Compose (noting docker-compose.yml and script/entrypoint.sh) to emphasize how each component runs in its own container.

##### Relationships and Interactions
- Airflow (dags) to the Kafka producer, which streams the data into Apache Kafka.
- Flow from Apache Kafka (which is further monitored via Confluent’s Control Center and Schema Registry) to Apache Spark for real-time processing.

##### Important Architectural Patterns and Design Principles
- Containerization: Clearly show that each major service (Airflow, Kafka, Spark, Cassandra, and Confluent) is isolated within Docker containers as orchestrated by docker-compose.
- Real-time Data Flow: Emphasize the continuous streaming pipeline – from data fetch, production to Kafka, real-time processing with Spark, and finally storage.
- Modular Design: Recognize that the project separates concerns neatly by isolating DAGs (Airflow related code), job scripts (stream processing), and plugins (custom integrations and utilities). This points to maintainability and extensibility in the design.
- Configuration Driven: Note the presence of config.yaml and a custom configparser under plugins/utils to show that the setup is environment-specific and easily adjustable.

##### Technologies, Frameworks, and Libraries
- Apache Airflow, Apache Kafka, and Apache Spark as the core processing engines.
- Cassandra (and optionally PostgreSQL) for data storage.
- Docker and Docker Compose for container orchestration.
- Confluent tools for Kafka monitoring.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: External API
    API["RandomUser API"]:::external

    %% Ingestion & Orchestration Layer
    subgraph "Ingestion & Orchestration"
        AIRFLOW["Airflow (DAG: stream_to_kafka.py)"]:::orchestration
    end

    %% Data Ingestion (Kafka Producer) Layer
    subgraph "Kafka Producer"
        KP_JOBS["Kafka Producer (jobs/kafka_stream.py)"]:::processing
        KP_PLUGINS["Kafka Streamer (plugins/kafka_streamer.py)"]:::processing
    end

    %% Messaging Layer: Apache Kafka and Monitoring
    subgraph "Messaging & Monitoring"
        KAFKA["Apache Kafka"]:::messaging
        CONFLUENT["Confluent Monitoring"]:::monitoring
    end

    %% Data Processing (Spark Consumer) Layer
    subgraph "Spark Consumer"
        SPARK_JOBS["Spark Consumer (jobs/spark_stream.py)"]:::processing
        SPARK_PLUGINS["Spark Streamer (plugins/spark_streamer.py)"]:::processing
    end

    %% Data Storage Layer
    subgraph "Data Storage"
        CASSANDRA["Cassandra Database"]:::storage
        POSTGRES["PostgreSQL Intermediate"]:::storage
    end

    %% Configuration Management Subgraph
    subgraph "Configuration Management"
        CONFIG_FILE["Config File"]:::config
        CONFIG_PARSER["Config Parser"]:::config
    end

    %% Container Environment Subgraph
    subgraph "Container Environment"
        DOCKER_COMPOSE["Docker Compose"]:::container
        ENTRYPOINT["Entrypoint Script"]:::container
    end

    %% Data Flow Connections
    API -->|"triggers"| AIRFLOW
    AIRFLOW -->|"initiates"| KP_JOBS
    AIRFLOW -->|"initiates"| KP_PLUGINS
    KP_JOBS -->|"sends data"| KAFKA
    KP_PLUGINS -->|"sends data"| KAFKA
    KAFKA -->|"delivers data"| SPARK_JOBS
    KAFKA -->|"delivers data"| SPARK_PLUGINS
    KAFKA -->|"monitored-by"| CONFLUENT
    SPARK_JOBS -->|"stores processed data"| CASSANDRA
    SPARK_PLUGINS -->|"stores processed data"| POSTGRES

    %% Configuration dashed connections to core processing components
    CONFIG_FILE --- AIRFLOW
    CONFIG_FILE --- KP_JOBS
    CONFIG_FILE --- SPARK_JOBS
    CONFIG_PARSER --- AIRFLOW
    CONFIG_PARSER --- KP_JOBS
    CONFIG_PARSER --- SPARK_JOBS

    %% Container Environment dashed connections (indicating containerization)
    DOCKER_COMPOSE --- AIRFLOW
    DOCKER_COMPOSE --- KP_JOBS
    DOCKER_COMPOSE --- KP_PLUGINS
    DOCKER_COMPOSE --- KAFKA
    DOCKER_COMPOSE --- SPARK_JOBS
    DOCKER_COMPOSE --- SPARK_PLUGINS
    DOCKER_COMPOSE --- CASSANDRA
    DOCKER_COMPOSE --- POSTGRES
    ENTRYPOINT --- AIRFLOW
    ENTRYPOINT --- KAFKA
    ENTRYPOINT --- SPARK_JOBS
    ENTRYPOINT --- CASSANDRA

    %% Click Events for Component Mappings
    click AIRFLOW "https://github.com/mekwiset/realtime_data_streaming/blob/main/dags/stream_to_kafka.py"
    click KP_JOBS "https://github.com/mekwiset/realtime_data_streaming/blob/main/jobs/kafka_stream.py"
    click KP_PLUGINS "https://github.com/mekwiset/realtime_data_streaming/blob/main/plugins/kafka_streamer.py"
    click SPARK_JOBS "https://github.com/mekwiset/realtime_data_streaming/blob/main/jobs/spark_stream.py"
    click SPARK_PLUGINS "https://github.com/mekwiset/realtime_data_streaming/blob/main/plugins/spark_streamer.py"
    click CASSANDRA "https://github.com/mekwiset/realtime_data_streaming/blob/main/helpers/create_table_created_users.sql"
    click POSTGRES "https://github.com/mekwiset/realtime_data_streaming/blob/main/helpers/create_table_created_users.sql"
    click DOCKER_COMPOSE "https://github.com/mekwiset/realtime_data_streaming/blob/main/docker-compose.yml"
    click ENTRYPOINT "https://github.com/mekwiset/realtime_data_streaming/blob/main/script/entrypoint.sh"
    click CONFIG_FILE "https://github.com/mekwiset/realtime_data_streaming/blob/main/config.yaml"
    click CONFIG_PARSER "https://github.com/mekwiset/realtime_data_streaming/blob/main/plugins/utils/configparser.py"

    %% Styles
    classDef external fill:#f4d03f,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#d4e6f1,stroke:#333,stroke-width:2px;
    classDef processing fill:#a9dfbf,stroke:#333,stroke-width:2px;
    classDef messaging fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef monitoring fill:#fadbd8,stroke:#333,stroke-width:2px;
    classDef storage fill:#f5cba7,stroke:#333,stroke-width:2px;
    classDef config fill:#d7bde2,stroke:#333,stroke-width:2px;
    classDef container fill:#d5dbdb,stroke:#333,stroke-width:2px;
```
