# pinildissanayaka/realtime-data-streaming-pipeline architecture

> Architecture diagram of the GitHub repository pinildissanayaka/realtime-data-streaming-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pinildissanayaka/realtime-data-streaming-pipeline
- Repository: https://github.com/pinildissanayaka/realtime-data-streaming-pipeline

## Overview

##### Project type and purpose
- This is an end-to-end real-time data engineering pipeline (full-stack ETL/streaming solution).
- Purpose: ingest user data from an external API, orchestrate via Airflow, stream with Kafka, process with Spark, and store results in Cassandra (with Postgres as intermediate storage).

##### File structure
- Top level:
- README.md – project overview
- requirements.txt – Python dependencies
- apache_stream.py – Kafka producer/consumer orchestration script
- fetch_db.py – script to fetch from Postgres and publish to Kafka
- log.log – pipeline log output
- checkpoint/ – Kafka stream offset/state files
- dags/Streaming_ETL.py – Airflow DAG definition for ingestion
- test/producer.py – Kafka producer for testing
- Patterns & indicators:
- “dags/” → Airflow orchestration (Directed Acyclic Graph)
- “checkpoint/” → Kafka Streams state management
- Separate scripts for ingestion (fetch_db.py), streaming (apache_stream.py) and tests

##### README for additional insights
- Confirms tech stack and high-level architecture:
- Data Source: Randomuser.me API
- Orchestration: Apache Airflow → fetch+persist to PostgreSQL
- Streaming: Apache Kafka + Zookeeper
- Processing: Apache Spark
- Storage: Cassandra (final), PostgreSQL (intermediate)

##### Key reminders
- Critical components and where they live in the file tree
- Don’t overthink—break into as many logical parts as makes sense (scripts, clusters, storage)
- Be detailed: include DAG definitions, checkpoint directories, test scripts
- The goal is a one-glance view of how data flows from source to final storage

## Diagram (Mermaid)

```mermaid
graph TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        API["Randomuser.me API"]:::external
        Airflow["Airflow Orchestrator\n(dags/Streaming_ETL.py)"]:::ingestion
        Fetch["Data Fetcher\n(fetch_db.py)"]:::ingestion
        Postgres["PostgreSQL"]:::storage
    end

    %% Streaming Layer
    subgraph "Streaming Layer"
        Producer["Kafka Producer\n(apache_stream.py)"]:::streaming
        TestProducer["Test Data Producer\n(test/producer.py)"]:::streaming
        Kafka["Kafka Cluster\n+ Zookeeper Ensemble"]:::streaming
        Checkpoint["Checkpoint Storage\n(checkpoint/)"]:::streaming
    end

    %% Processing Layer
    subgraph "Processing Layer"
        Spark["Spark Streaming Job"]:::processing
    end

    %% Storage Layer
    subgraph "Storage Layer"
        Cassandra["Cassandra Cluster"]:::storage
    end

    %% Monitoring/Logging
    subgraph "Monitoring"
        Logging["Pipeline Logs\n(log.log)"]:::monitor
    end

    %% Documentation
    subgraph "Documentation"
        README["Project Overview\n(README.md)"]:::doc
        Req["Python Dependencies\n(requirements.txt)"]:::doc
    end

    %% Data Flow
    API -->|"fetch data"| Airflow
    Airflow -->|"runs DAG"| Fetch
    Fetch -->|"writes to"| Postgres
    Postgres -->|"read from"| Producer
    Producer -->|"publishes to"| Kafka
    TestProducer -->|"publishes to"| Kafka
    Kafka -->|"stores offsets in"| Checkpoint
    Kafka -->|"streams to"| Spark
    Spark -->|"writes to"| Cassandra
    Airflow -->|"logs status to"| Logging
    Spark -->|"logs output to"| Logging

    %% Click Events
    click Airflow "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/blob/main/dags/Streaming_ETL.py"
    click Fetch "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/blob/main/fetch_db.py"
    click Producer "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/blob/main/apache_stream.py"
    click Checkpoint "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/tree/main/checkpoint/"
    click TestProducer "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/blob/main/test/producer.py"
    click Logging "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/blob/main/log.log"
    click Req "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/blob/main/requirements.txt"
    click README "https://github.com/pinildissanayaka/realtime-data-streaming-pipeline/blob/main/README.md"

    %% Styles
    classDef ingestion fill:#AED6F1,stroke:#1F618D,color:#1B4F72
    classDef streaming fill:#F5B041,stroke:#B9770E,color:#7D6608
    classDef processing fill:#82E0AA,stroke:#28B463,color:#1D8348
    classDef storage fill:#D7BDE2,stroke:#8E44AD,color:#4A235A
    classDef external fill:#D6DBDF,stroke:#566573,color:#1C2833
    classDef monitor fill:#F9E79F,stroke:#B7950B,color:#7D6608
    classDef doc fill:#E5E7E9,stroke:#ABB2B9,color:#566573
```
