# shahaba/event-driven-kafka-pipeline architecture

> Architecture diagram of the GitHub repository shahaba/event-driven-kafka-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shahaba/event-driven-kafka-pipeline
- Repository: https://github.com/shahaba/event-driven-kafka-pipeline

## Overview

This repository is a Python/Faust stock-price event pipeline: it generates simulated prices, publishes them to Kafka, stores raw events in Postgres, and produces aggregate CSV reports. The README documents these stages; sampled code shows the producer, consumer, and aggregator behavior. Note that the consumer and aggregator configure different database names, so the intended shared-database wiring is uncertain.

Main entry points: [`stock-prices/aggregator.py`](https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/aggregator.py) (Aggregate and export), [`stock-prices/consumer.py`](https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/consumer.py) (Raw event consumer), [`stock-prices/producer.py`](https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/producer.py) (Price producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Event generation"]
  node_producer["Price producer<br/>[producer.py]"]
end

subgraph group_streaming["Kafka streaming"]
  node_kafka["Kafka topic"]
  node_consumer["Raw event consumer<br/>[consumer.py]"]
end

subgraph group_persistence["Storage and reporting"]
  node_aggregator["Aggregate and export<br/>[aggregator.py]"]
  node_postgres[("Postgres")]
  node_csv["CSV reports"]
end

node_producer -->|"publishes prices"| node_kafka
node_kafka -->|"delivers events"| node_consumer
node_kafka -->|"delivers events"| node_aggregator
node_consumer -->|"writes raw prices"| node_postgres
node_aggregator -->|"writes aggregates"| node_postgres
node_aggregator -->|"queries prices"| node_postgres
node_postgres -->|"returns rows"| node_aggregator
node_aggregator -->|"writes reports"| node_csv

click node_producer "https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/producer.py"
click node_consumer "https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/consumer.py"
click node_aggregator "https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/aggregator.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer toneBlue
class node_kafka,node_consumer toneAmber
class node_aggregator,node_postgres,node_csv toneMint
```

## Components

### Event generation

- [Price producer](https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/producer.py): `stock-prices/producer.py`

### Kafka streaming

- **Kafka topic**
- [Raw event consumer](https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/consumer.py): `stock-prices/consumer.py`

### Storage and reporting

- [Aggregate and export](https://github.com/shahaba/event-driven-kafka-pipeline/blob/main/stock-prices/aggregator.py): `stock-prices/aggregator.py`
- **Postgres**
- **CSV reports**

## Connections

- Price producer → Kafka topic: publishes prices
- Kafka topic → Raw event consumer: delivers events
- Kafka topic → Aggregate and export: delivers events
- Raw event consumer → Postgres: writes raw prices
- Aggregate and export → Postgres: writes aggregates
- Aggregate and export → Postgres: queries prices
- Postgres → Aggregate and export: returns rows
- Aggregate and export → CSV reports: writes reports