How bhanuteja2001/kafka-based-real-time-data-processing-pipeline- works
This repository implements a Kafka-based real-time login-event pipeline. The main loop consumes events, parses and validates them, transforms privacy-sensitive fields, filters by app version, publishes successful or rejected records, records per-message metrics, periodically emits summaries, and commits offsets after successful publication. Kafka topics are the pipeline’s observable outputs; summary and metrics state are maintained in memory. The flow and topic names are documented or visible in the sampled sources; the age-filter behavior described in the README is not represented in the sampled main-loop excerpt.
Main entry points: main.py Pipeline Main Loop · producer.py Kafka Producer · consumer.py Kafka Consumer · metrics.py Message Metrics
Components 14
Kafka Ingestion
- user-login
- Kafka Consumer
kafka_processor/consumer.py
Event Processing
- Pipeline Main Loop
kafka_processor/main.py - Privacy Transformer
kafka_processor/transformer.py
Metrics And Summaries
- Summary Counters
kafka_processor/summary_printer.py - Message Metrics
kafka_processor/metrics.py
Kafka Outputs
- Kafka Producer
kafka_processor/producer.py - processed-output
- processed-errors
- cleaned-data
- summary-output
- metrics-output
Other components
- Login Event Source
- Kafka Broker
Connections 20
- Login Event Source → user-login: sends events
README.md - Pipeline Main Loop → Kafka Consumer: creates and polls
kafka_processor/main.py - user-login → Kafka Consumer: supplies events
- Kafka Consumer → Pipeline Main Loop: returns messages
kafka_processor/main.py - Pipeline Main Loop → Privacy Transformer: transforms messages
kafka_processor/main.py - Pipeline Main Loop → Summary Counters: updates counts
kafka_processor/main.py - Pipeline Main Loop → Message Metrics: records metrics
kafka_processor/main.py - Pipeline Main Loop → Kafka Producer: publishes via
kafka_processor/main.py - Kafka Producer → processed-output: publishes
- Kafka Producer → processed-errors: logs errors
- Kafka Producer → cleaned-data: logs filtered data
- Pipeline Main Loop → summary-output: publishes summaries
kafka_processor/main.py - Message Metrics → Kafka Producer: publishes through
kafka_processor/metrics.py - Kafka Producer → metrics-output: publishes metrics
- Kafka Broker → user-login: hosts topic
README.md - Kafka Broker → processed-output: hosts topic
README.md - Kafka Broker → processed-errors: hosts topic
README.md - Kafka Broker → cleaned-data: hosts topic
README.md - Kafka Broker → summary-output: hosts topic
README.md - Kafka Broker → metrics-output: hosts topic
README.md