How debanjansaha-git/kafka-spark-cassandra works
Project Type and Purpose
• This is a real-time data streaming pipeline (data engineering project), not a typical web or mobile app.
• Purpose: simulate weather data → stream it via Kafka → transform with Spark Streaming → store in Cassandra → orchestrate with Airflow.
File Structure
• Top-level:
• docker-compose.yml – defines Kafka, Zookeeper, Spark, Cassandra, Airflow services
• requirements*.txt – Python dependencies
• spark_stream.py – Spark Streaming job
• kafka_stream.py – Kafka producer (weather data generator)
• dags/ – Airflow DAG definitions (weather_data_streaming.py)
• script/entrypoint.sh – container entrypoint for startup
• Patterns:
• “dags” indicates orchestration layer (Airflow)
• Separate scripts for producer (kafka_stream.py) and processor (spark_stream.py)
README Insights
• Technology stack: Kafka, Spark, Cassandra, Airflow, Docker, Faker
• Architecture section enumerates components and includes a reference image
• Flow: Data generation → Kafka topic → Spark Streaming → Cassandra
Main Components & Their Interactions
a. Weather Data Generator (Kafka Producer)
– Executes kafka_stream.py (orchestrated by Airflow PythonOperator)
– Uses Faker to emit JSON messages to Kafka topic “weather_data”
b. Apache Kafka Cluster
– Kafka Broker + Zookeeper (defined in Docker Compose)
– Receives and buffers incoming weather events
c. Apache Spark Streaming
– spark_stream.py reads from Kafka, flattens nested JSON
– Writes transformed records into Cassandra
d. Cassandra Database
– Keyspace: spark_streams, Table: weather_data
– Stores final records for analytics
e. Apache Airflow
– dag/weather_data_streaming.py orchestrates end-to-end pipeline
– Tasks: trigger producer, monitor stream, possibly kick off Spark job
f. Docker Compose
– Groups all services in isolated Docker network
Architectural Patterns and Principles
• Modular separation: producer, broker, consumer, database
• Event-driven streaming architecture
• Orchestration vs. processing vs. storage layers
• Resilience via Kafka buffering
• Scalability: Spark cluster + Cassandra distributed writes
Detail Level
• PythonOperator in the DAG triggering kafka_stream.py
• Both broker and Zookeeper if desired
• Topic name (weather_data)
• Keyspace/table in Cassandra
• Entrypoint.sh as container startup