How whereevenami/smart-city-traffic-pipeline works
The product simulates traffic readings, streams them through Kafka, and consumes each event into both a date-partitioned CSV lake and Cassandra. Hive computes historical aggregates from the lake; an export job materializes them as JSON. FastAPI serves live and historical Cassandra reads alongside the precomputed analytics, and the Streamlit dashboard presents both. The README documents the Hive table and query stages, but their SQL files were not sampled; those relationships are shown as documented. The graph focuses on runtime product components, not deployment or tests.
Main entry points: consumer/main.py Event consumer · api/main.py FastAPI service · producer.py Kafka producer · file_writer.py CSV lake writer
Components 13
Event generation
- Traffic simulator
ingestion/simulator.py - Kafka producer
ingestion/producer.py
Stream and storage
- Kafka topic
- Event consumer
consumer/main.py - CSV lake writer
consumer/file_writer.py - Partitioned CSV lake
data - Cassandra writer
consumer/cassandra_store.py
Batch analytics
- Hive analytics
hive - Analytics exporter
hive/export_analytics.py - Analytics JSON
data/analytics
Serving and UI
- Cassandra store
- FastAPI service
api/main.py - Traffic dashboard
dashboard/app.py
Connections 13
- Kafka producer → Traffic simulator: generates readings
ingestion/producer.py - Kafka producer → Kafka topic: publishes events
README.md - Kafka topic → Event consumer: delivers events
README.md - Event consumer → CSV lake writer: writes events
consumer/main.py - Event consumer → Cassandra writer: saves events
consumer/main.py - CSV lake writer → Partitioned CSV lake: writes CSV
README.md - Partitioned CSV lake → Hive analytics: queries partitions
README.md - Cassandra writer → Cassandra store: writes readings
consumer/cassandra_store.py - Hive analytics → Analytics exporter: runs aggregates
hive/export_analytics.py - Analytics exporter → Analytics JSON: writes results
hive/export_analytics.py - FastAPI service → Cassandra store: queries live data
api/main.py - FastAPI service → Analytics JSON: reads results
api/main.py - Traffic dashboard → FastAPI service: requests data
dashboard/app.py