# whereevenami/smart-city-traffic-pipeline architecture

> Architecture diagram of the GitHub repository whereevenami/smart-city-traffic-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/whereevenami/smart-city-traffic-pipeline
- Repository: https://github.com/whereevenami/smart-city-traffic-pipeline

## Overview

The product simulates traffic readings, streams them through Kafka, and consumes each event into both a date-partitioned CSV lake and Cassandra. Hive computes historical aggregates from the lake; an export job materializes them as JSON. FastAPI serves live and historical Cassandra reads alongside the precomputed analytics, and the Streamlit dashboard presents both. The README documents the Hive table and query stages, but their SQL files were not sampled; those relationships are shown as documented. The graph focuses on runtime product components, not deployment or tests.

Main entry points: [`consumer/main.py`](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/main.py) (Event consumer), [`api/main.py`](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/api/main.py) (FastAPI service), [`ingestion/producer.py`](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/ingestion/producer.py) (Kafka producer), [`consumer/file_writer.py`](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/file_writer.py) (CSV lake writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Event generation"]
  node_simulator["Traffic simulator<br/>[simulator.py]"]
  node_producer["Kafka producer<br/>[producer.py]"]
end

subgraph group_streaming["Stream and storage"]
  node_kafka["Kafka topic"]
  node_consumer["Event consumer<br/>[main.py]"]
  node_file_writer["CSV lake writer<br/>[file_writer.py]"]
  node_data_lake[("Partitioned CSV lake")]
  node_cassandra_writer["Cassandra writer<br/>[cassandra_store.py]"]
end

subgraph group_analytics["Batch analytics"]
  node_hive["Hive analytics"]
  node_analytics_export["Analytics exporter"]
  node_analytics_json["Analytics JSON"]
end

subgraph group_serving["Serving and UI"]
  node_cassandra[("Cassandra store")]
  node_api["FastAPI service<br/>[main.py]"]
  node_dashboard["Traffic dashboard<br/>[app.py]"]
end

node_producer -->|"generates readings"| node_simulator
node_producer -->|"publishes events"| node_kafka
node_kafka -->|"delivers events"| node_consumer
node_consumer -->|"writes events"| node_file_writer
node_consumer -->|"saves events"| node_cassandra_writer
node_file_writer -->|"writes CSV"| node_data_lake
node_data_lake -->|"queries partitions"| node_hive
node_cassandra_writer -->|"writes readings"| node_cassandra
node_hive -->|"runs aggregates"| node_analytics_export
node_analytics_export -->|"writes results"| node_analytics_json
node_api -->|"queries live data"| node_cassandra
node_api -->|"reads results"| node_analytics_json
node_dashboard -->|"requests data"| node_api

click node_simulator "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/ingestion/simulator.py"
click node_producer "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/ingestion/producer.py"
click node_consumer "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/main.py"
click node_file_writer "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/file_writer.py"
click node_data_lake "https://github.com/whereevenami/smart-city-traffic-pipeline/tree/main/data"
click node_cassandra_writer "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/cassandra_store.py"
click node_hive "https://github.com/whereevenami/smart-city-traffic-pipeline/tree/main/hive"
click node_analytics_export "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/hive/export_analytics.py"
click node_analytics_json "https://github.com/whereevenami/smart-city-traffic-pipeline/tree/main/data/analytics"
click node_api "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/api/main.py"
click node_dashboard "https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/dashboard/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_simulator,node_producer toneBlue
class node_kafka,node_consumer,node_file_writer,node_data_lake,node_cassandra_writer toneAmber
class node_hive,node_analytics_export,node_analytics_json toneMint
class node_cassandra,node_api,node_dashboard toneRose
```

## Components

### Event generation

- [Traffic simulator](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/ingestion/simulator.py): `ingestion/simulator.py`
- [Kafka producer](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/ingestion/producer.py): `ingestion/producer.py`

### Stream and storage

- **Kafka topic**
- [Event consumer](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/main.py): `consumer/main.py`
- [CSV lake writer](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/file_writer.py): `consumer/file_writer.py`
- [Partitioned CSV lake](https://github.com/whereevenami/smart-city-traffic-pipeline/tree/main/data): `data`
- [Cassandra writer](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/consumer/cassandra_store.py): `consumer/cassandra_store.py`

### Batch analytics

- [Hive analytics](https://github.com/whereevenami/smart-city-traffic-pipeline/tree/main/hive): `hive`
- [Analytics exporter](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/hive/export_analytics.py): `hive/export_analytics.py`
- [Analytics JSON](https://github.com/whereevenami/smart-city-traffic-pipeline/tree/main/data/analytics): `data/analytics`

### Serving and UI

- **Cassandra store**
- [FastAPI service](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/api/main.py): `api/main.py`
- [Traffic dashboard](https://github.com/whereevenami/smart-city-traffic-pipeline/blob/main/dashboard/app.py): `dashboard/app.py`

## Connections

- Kafka producer → Traffic simulator: generates readings (evidence: `ingestion/producer.py`)
- Kafka producer → Kafka topic: publishes events (evidence: `README.md`)
- Kafka topic → Event consumer: delivers events (evidence: `README.md`)
- Event consumer → CSV lake writer: writes events (evidence: `consumer/main.py`)
- Event consumer → Cassandra writer: saves events (evidence: `consumer/main.py`)
- CSV lake writer → Partitioned CSV lake: writes CSV (evidence: `README.md`)
- Partitioned CSV lake → Hive analytics: queries partitions (evidence: `README.md`)
- Cassandra writer → Cassandra store: writes readings (evidence: `consumer/cassandra_store.py`)
- Hive analytics → Analytics exporter: runs aggregates (evidence: `hive/export_analytics.py`)
- Analytics exporter → Analytics JSON: writes results (evidence: `hive/export_analytics.py`)
- FastAPI service → Cassandra store: queries live data (evidence: `api/main.py`)
- FastAPI service → Analytics JSON: reads results (evidence: `api/main.py`)
- Traffic dashboard → FastAPI service: requests data (evidence: `dashboard/app.py`)