How mjngxwnj/traffic-and-weather-data-streaming-pipeline works
Project Type and Purpose
• This is a full‐stack streaming data pipeline for near‐real‐time processing of traffic camera images and weather data.
• It ingests images from traffic cameras (YOLOv8 via images_processing/), fetches weather data via OpenWeather API (weather_fetching/), streams data through Kafka (kafka_stream.py), processes it in Spark Streaming and Spark Batch (spark_script/), stores results in Cassandra and HDFS, and serves metrics via a Streamlit dashboard (application/).
File Structure Analysis
• Top-Level Components:
• application/ – Streamlit front-end and back-end, Dockerfile, requirements.
• images_processing/ – YOLOv8 image ingestion and vehicle count logic.
• weather_fetching/ – OpenWeather API client.
• spark_script/ – batch_job.py (daily batch logic), stream_job.py (real-time Spark Streaming), utils_spark.py.
• kafka_stream.py – Kafka producer for traffic and weather events.
• cql/ – Cassandra keyspace and table definitions.
• docker-compose.yaml & hadoop.env – container orchestration and Hadoop config.
• run_stream_job.sh, run_batch_job.sh – orchestration scripts.
Key Architectural Patterns
• Lambda (Hybrid) Architecture:
• Stream Layer – Spark Streaming consumes Kafka topics, writes to Cassandra (low latency).
• Batch Layer – Spark batch jobs consume historical Kafka data, write partitions to HDFS & aggregated results to Cassandra.
• Serving Layer – Cassandra serves combined real-time and batch data for low-latency queries; Streamlit provides interactive UI.
• Micro-services & Containerization – each major component runs in its own Docker container via docker-compose.
• Message Queue – Kafka for decoupling ingestion from processing, enabling fault tolerance and multiple consumers.
Main Components & Interactions
A. Data Ingestion
• Traffic Camera Module (images_processing/images_processor.py)
• Weather Fetching Module (weather_fetching/weather_utils.py)
→ produce events to Kafka topics (traffic_topic, weather_topic).
B. Kafka Cluster
• Brokers and Zookeeper (defined in docker-compose.yaml)
→ buffer and replicate incoming events.
C. Stream Processing
• Spark Streaming Job (spark_script/stream_job.py, run_stream_job.sh)
→ reads from Kafka, transforms data (density, humidex, etc.), writes to Cassandra stream_ tables.
D. Batch Processing
• Spark Batch Job (spark_script/batch_job.py, run_batch_job.sh)
→ scheduled daily via cron, reads historical events from Kafka or HDFS, writes raw to HDFS (partitioned by year/month/day), aggregates, writes to Cassandra batch_ tables.
E. Storage
• HDFS – raw and partitioned historical data (configured by hadoop.env).
• Cassandra – real-time and aggregated tables (cql/keyspace_table.cql).
F. Serving Layer
• Streamlit Application (application/main.py, streamlit_backend.py, streamlit_frontend.py)
→ queries Cassandra, renders dashboards.
• Dockerfile in application/ for containerizing the UI.
• Component names matching folder/script names (e.g., kafka_stream.py, stream_job.py).
• Environment configuration (docker-compose, hadoop.env).
• Data Ingestion (blue),
• Messaging (orange),
• Processing (green),
• Storage (purple),
• Serving (gray).
• Rectangles for services/apps,