# bhanuteja2001/kafka-based-real-time-data-processing-pipeline- architecture

> Architecture diagram of the GitHub repository bhanuteja2001/kafka-based-real-time-data-processing-pipeline-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-
- Repository: https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-

## Overview

This repository implements a Kafka-based real-time login-event pipeline. The main loop consumes events, parses and validates them, transforms privacy-sensitive fields, filters by app version, publishes successful or rejected records, records per-message metrics, periodically emits summaries, and commits offsets after successful publication. Kafka topics are the pipeline’s observable outputs; summary and metrics state are maintained in memory. The flow and topic names are documented or visible in the sampled sources; the age-filter behavior described in the README is not represented in the sampled main-loop excerpt.

Main entry points: [`kafka_processor/main.py`](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/main.py) (Pipeline Main Loop), [`kafka_processor/producer.py`](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/producer.py) (Kafka Producer), [`kafka_processor/consumer.py`](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/consumer.py) (Kafka Consumer), [`kafka_processor/metrics.py`](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/metrics.py) (Message Metrics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Kafka Ingestion"]
  node_input_topic["user-login"]
  node_consumer["Kafka Consumer<br/>[consumer.py]"]
end

subgraph group_pipeline["Event Processing"]
  node_runtime["Pipeline Main Loop<br/>[main.py]"]
  node_transformer["Privacy Transformer<br/>[transformer.py]"]
end

subgraph group_telemetry["Metrics And Summaries"]
  node_summary["Summary Counters<br/>[summary_printer.py]"]
  node_metrics["Message Metrics<br/>[metrics.py]"]
end

subgraph group_outputs["Kafka Outputs"]
  node_producer["Kafka Producer<br/>[producer.py]"]
  node_processed_topic["processed-output"]
  node_errors_topic["processed-errors"]
  node_cleaned_topic["cleaned-data"]
  node_summary_topic["summary-output"]
  node_metrics_topic["metrics-output"]
end

node_event_source(("Login Event Source"))
node_broker[("Kafka Broker")]

node_event_source -->|"sends events"| node_input_topic
node_runtime -->|"creates and polls"| node_consumer
node_input_topic -->|"supplies events"| node_consumer
node_consumer -->|"returns messages"| node_runtime
node_runtime -->|"transforms messages"| node_transformer
node_runtime -->|"updates counts"| node_summary
node_runtime -->|"records metrics"| node_metrics
node_runtime -->|"publishes via"| node_producer
node_producer -->|"publishes"| node_processed_topic
node_producer -->|"logs errors"| node_errors_topic
node_producer -->|"logs filtered data"| node_cleaned_topic
node_runtime -->|"publishes summaries"| node_summary_topic
node_metrics -->|"publishes through"| node_producer
node_producer -->|"publishes metrics"| node_metrics_topic
node_broker -->|"hosts topic"| node_input_topic
node_broker -->|"hosts topic"| node_processed_topic
node_broker -->|"hosts topic"| node_errors_topic
node_broker -->|"hosts topic"| node_cleaned_topic
node_broker -->|"hosts topic"| node_summary_topic
node_broker -->|"hosts topic"| node_metrics_topic

click node_consumer "https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/consumer.py"
click node_runtime "https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/main.py"
click node_transformer "https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/transformer.py"
click node_producer "https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/producer.py"
click node_summary "https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/summary_printer.py"
click node_metrics "https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_input_topic,node_consumer toneBlue
class node_runtime,node_transformer,node_broker toneAmber
class node_summary,node_metrics toneMint
class node_producer,node_processed_topic,node_errors_topic,node_cleaned_topic,node_summary_topic,node_metrics_topic toneRose
class node_event_source toneIndigo
```

## Components

### Kafka Ingestion

- **user-login**
- [Kafka Consumer](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/consumer.py): `kafka_processor/consumer.py`

### Event Processing

- [Pipeline Main Loop](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/main.py): `kafka_processor/main.py`
- [Privacy Transformer](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/transformer.py): `kafka_processor/transformer.py`

### Metrics And Summaries

- [Summary Counters](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/summary_printer.py): `kafka_processor/summary_printer.py`
- [Message Metrics](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/metrics.py): `kafka_processor/metrics.py`

### Kafka Outputs

- [Kafka Producer](https://github.com/bhanuteja2001/kafka-based-real-time-data-processing-pipeline-/blob/main/kafka_processor/producer.py): `kafka_processor/producer.py`
- **processed-output**
- **processed-errors**
- **cleaned-data**
- **summary-output**
- **metrics-output**

### Other components

- **Login Event Source**
- **Kafka Broker**

## Connections

- Login Event Source → user-login: sends events (evidence: `README.md`)
- Pipeline Main Loop → Kafka Consumer: creates and polls (evidence: `kafka_processor/main.py`)
- user-login → Kafka Consumer: supplies events
- Kafka Consumer → Pipeline Main Loop: returns messages (evidence: `kafka_processor/main.py`)
- Pipeline Main Loop → Privacy Transformer: transforms messages (evidence: `kafka_processor/main.py`)
- Pipeline Main Loop → Summary Counters: updates counts (evidence: `kafka_processor/main.py`)
- Pipeline Main Loop → Message Metrics: records metrics (evidence: `kafka_processor/main.py`)
- Pipeline Main Loop → Kafka Producer: publishes via (evidence: `kafka_processor/main.py`)
- Kafka Producer → processed-output: publishes
- Kafka Producer → processed-errors: logs errors
- Kafka Producer → cleaned-data: logs filtered data
- Pipeline Main Loop → summary-output: publishes summaries (evidence: `kafka_processor/main.py`)
- Message Metrics → Kafka Producer: publishes through (evidence: `kafka_processor/metrics.py`)
- Kafka Producer → metrics-output: publishes metrics
- Kafka Broker → user-login: hosts topic (evidence: `README.md`)
- Kafka Broker → processed-output: hosts topic (evidence: `README.md`)
- Kafka Broker → processed-errors: hosts topic (evidence: `README.md`)
- Kafka Broker → cleaned-data: hosts topic (evidence: `README.md`)
- Kafka Broker → summary-output: hosts topic (evidence: `README.md`)
- Kafka Broker → metrics-output: hosts topic (evidence: `README.md`)
