# pranav-8660/realtimedatastreamer-apachekafka architecture

> Architecture diagram of the GitHub repository pranav-8660/realtimedatastreamer-apachekafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pranav-8660/realtimedatastreamer-apachekafka
- Repository: https://github.com/pranav-8660/realtimedatastreamer-apachekafka

## Overview

##### Project Type and Purpose
- Type: Real-time data streaming pipeline.
- Purpose: Generate synthetic sensor data, stream it through Kafka, and persist it into PostgreSQL for downstream analysis.

##### File Structure
- Top-level:
– kafka-producer/
- data_producer.py (publishes JSON messages to Kafka topic)
- venv/ (Python virtual environment for isolation)
– kafka-consumer/
- kafka_consumer.py (subscribes to Kafka topic, writes to Postgres)
– venv/ (root-level environment, alternative setup)
– commands.txt (CLI instructions, optional)
- No web UI—pure data pipeline.

##### Extract Key Components from README
- Producer side:
– Python script using kafka-python, uuid, python-dotenv
– Kafka topic “sensor-data” on a Kafka broker (localhost:9092)
- Messaging layer:
– Zookeeper (coordination), Kafka broker, topic partitions
- Consumer side:
– Python script using kafka-python, psycopg2-binary
– PostgreSQL 14+ database “sensor_data” with table sensor_readings
- Configuration via .env or environment variables

a. Data Producer
- Tech: Python + kafka-python
b. Kafka Cluster
- Annotation: Partitions = 1, Replication factor = 1
c. Data Consumer
- Tech: Python + kafka-python + psycopg2
d. PostgreSQL Database
e. (Optional) CLI/Monitoring

##### Architectural Patterns & Principles
- Asynchronous, decoupled pipeline (pub/sub)
- Single responsibility: each script handles one step
- Config via environment variables (12-factor)
- Fault-tolerant messaging (Kafka) and durable storage (Postgres)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Producer Service"
        Prod["Producer Service\ndata_producer.py"]:::compute
        ProdEnv["Virtual Env"]:::compute_sub
    end

    subgraph "Kafka Cluster"
        Zoo["Zookeeper"]:::messaging
        Broker["Kafka Broker\n(9092)"]:::messaging
        Topic["Topic: sensor-data\nPartitions=1, RF=1"]:::messaging
    end

    subgraph "Consumer Service"
        Cons["Consumer Service\nkafka_consumer.py"]:::compute
        ConsEnv["Virtual Env"]:::compute_sub
    end

    DB["PostgreSQL\nsensor_data\nsensor_readings"]:::storage

    CLI["CLI / Monitoring\n(commands.txt)"]:::monitoring

    Prod -->|"Produce JSON →"| Zoo
    Zoo --> Broker
    Broker --> Topic
    Topic -->|"Subscribe & Deserialize →"| Cons
    Cons -->|"INSERT →"| DB

    CLI -.-> Zoo
    CLI -.-> DB

    RootEnv["Root Virtual Env"]:::compute_sub
    Doc["Project Overview\nREADME.md"]:::documentation

    click Prod "https://github.com/pranav-8660/realtimedatastreamer-apachekafka/blob/main/kafka-producer/data_producer.py"
    click ProdEnv "https://github.com/pranav-8660/realtimedatastreamer-apachekafka/tree/main/kafka-producer/venv"
    click Cons "https://github.com/pranav-8660/realtimedatastreamer-apachekafka/blob/main/kafka-consumer/kafka_consumer.py"
    click CLI "https://github.com/pranav-8660/realtimedatastreamer-apachekafka/blob/main/commands.txt"
    click RootEnv "https://github.com/pranav-8660/realtimedatastreamer-apachekafka/tree/main/venv"
    click Doc "https://github.com/pranav-8660/realtimedatastreamer-apachekafka/blob/main/README.md"

    classDef compute fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute_sub fill:#E0FFFF,stroke:#333,stroke-width:1px
    classDef messaging fill:#FFA500,stroke:#333,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#333,stroke-width:1px
    classDef monitoring fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef documentation fill:#FFD700,stroke:#333,stroke-width:1px
```
