# chinmay13579/real_time_data_ingestion architecture

> Architecture diagram of the GitHub repository chinmay13579/real_time_data_ingestion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chinmay13579/real_time_data_ingestion
- Repository: https://github.com/chinmay13579/real_time_data_ingestion

## Overview

##### Project Type and Purpose
- This is a real-time streaming data ingestion pipeline.
- Purpose: simulate order and payment events, join them in flight, and persist the merged records into a Cassandra “fact” table.

##### File Structure
- Top level scripts:
- order_data_producer.py
- payments_data_producer.py
- order_data_consumer.py
- ingest_in_fact_table.py
- Infrastructure definition:
- docker-compose-cassandra.yml (brings up Cassandra, maybe Zookeeper/Kafka in other compose file or separate)
- Supporting docs:
- README.md
- setup_and_requirement_readme.txt
- No explicit “src”/“lib” subdirectories: each script is a standalone component.

##### README Insights
- Two Kafka producers to simulate two event streams (orders, payments).
- One Kafka consumer that joins streams and passes merged payloads to Cassandra.
- Uses Apache Kafka + Zookeeper for messaging, Cassandra for storage.
- Python 3.8+, kafka-python, cassandra-driver.

a. Main Components
- Order Producer service
- Payment Producer service
- Kafka Cluster (Brokers + Zookeeper)
- Order+Payment Consumer (stream join)
- Cassandra cluster (fact table storage)
b. Relationships and Interactions
1. Order Producer → publishes to “orders” Kafka topic
2. Payment Producer → publishes to “payments” Kafka topic
3. Consumer subscribes to both topics, joins events by key (e.g., order_id)
4. Consumer writes merged events into Cassandra fact table
c. Architectural Patterns & Principles
- Streaming ETL pattern
- Loose coupling via message bus (Kafka topics)
- Idempotent writes into Cassandra (fact table)
- Docker-based deployment for infra components
d. Relevant Technologies
- kafka-python library for producers/consumer
- cassandra-driver for DB IO
- Docker Compose for Cassandra (and possibly Kafka)
- Python 3.8 runtime

##### Elements
- Clear labels for each component (service or infra)
- Producers → Kafka topics → Consumer → Cassandra
- Rectangles for application services (blue)
- Annotations for message topics (orders, payments) and fact table name.

##### Detailed Guidance
- Optionally, show Docker Compose orchestration around Kafka & Cassandra to indicate containerized deployment.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Producers Layer
    subgraph "Producers"
        direction TB
        OP["Order Producer\n(order_data_producer.py)"]:::service
        PP["Payment Producer\n(payments_data_producer.py)"]:::service
    end

    %% Messaging Layer
    subgraph "Docker Compose"
        direction TB
        subgraph "Kafka Cluster" 
            direction TB
            KC["Kafka Cluster\n(topic: orders, payments)"]:::broker
            ZK["Zookeeper"]:::broker
        end
        subgraph "Cassandra Cluster"
            direction TB
            CS["Cassandra Cluster\n(fact table)"]:::database
        end
    end

    %% Consumer & Ingestion Layer
    subgraph "Stream Processing"
        direction TB
        OC["Order+Payment Consumer\n(order_data_consumer.py)"]:::service
        FI["Fact Table Ingestion\n(ingest_in_fact_table.py)"]:::service
    end

    %% Documentation
    subgraph "Project Documentation"
        direction TB
        README["README.md"]:::docs
        SETUP["setup_and_requirement_readme.txt"]:::docs
    end

    %% Data Flow
    OP -->|"orders topic"| KC
    PP -->|"payments topic"| KC
    KC --> OC
    OC --> FI
    FI --> CS

    %% Click Events
    click OP "https://github.com/chinmay13579/real_time_data_ingestion/blob/main/order_data_producer.py"
    click PP "https://github.com/chinmay13579/real_time_data_ingestion/blob/main/payments_data_producer.py"
    click OC "https://github.com/chinmay13579/real_time_data_ingestion/blob/main/order_data_consumer.py"
    click FI "https://github.com/chinmay13579/real_time_data_ingestion/blob/main/ingest_in_fact_table.py"
    click CS "https://github.com/chinmay13579/real_time_data_ingestion/blob/main/docker-compose-cassandra.yml"
    click README "https://github.com/chinmay13579/real_time_data_ingestion/blob/main/README.md"
    click SETUP "https://github.com/chinmay13579/real_time_data_ingestion/blob/main/setup_and_requirement_readme.txt"

    %% Styles
    classDef service fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef broker fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef database fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef docs fill:#D3D3D3,stroke:#000,stroke-width:1px;
```
