How shahaba/event-driven-kafka-pipeline works
This repository is a Python/Faust stock-price event pipeline: it generates simulated prices, publishes them to Kafka, stores raw events in Postgres, and produces aggregate CSV reports. The README documents these stages; sampled code shows the producer, consumer, and aggregator behavior. Note that the consumer and aggregator configure different database names, so the intended shared-database wiring is uncertain.
Main entry points: aggregator.py Aggregate and export · consumer.py Raw event consumer · producer.py Price producer
Components 6
Event generation
- Price producer
stock-prices/producer.py
Kafka streaming
- Kafka topic
- Raw event consumer
stock-prices/consumer.py
Storage and reporting
- Aggregate and export
stock-prices/aggregator.py - Postgres
- CSV reports
Connections 8
- Price producer → Kafka topic: publishes prices
- Kafka topic → Raw event consumer: delivers events
- Kafka topic → Aggregate and export: delivers events
- Raw event consumer → Postgres: writes raw prices
- Aggregate and export → Postgres: writes aggregates
- Aggregate and export → Postgres: queries prices
- Postgres → Aggregate and export: returns rows
- Aggregate and export → CSV reports: writes reports