How dogukannulu/kafka_spark_structured_streaming works
Project Type and Purpose
• This project demonstrates a real-time streaming pipeline using Apache Spark Structured Streaming, Kafka, and Apache Airflow.
• The original design fetches random names from an API and sends them to Kafka (using an Airflow DAG), then consumes these messages using a PySpark job which writes them into a database (originally Cassandra).
Analysis of the File Structure
• Top-level files include the README, multiple Docker Compose files, and key Python scripts:
– stream_to_kafka.py: Contains the logic to fetch data from the API and send it to a Kafka topic.
– stream_to_kafka_dag.py: The Airflow DAG script that schedules the data ingestion every 10 seconds.
– spark_streaming.py: Implements the Spark Structured Streaming consumer to read messages from Kafka.
– Files such as docker-compose.yml and docker-compose-LocalExecutor.yml help deploy a multi-node Kafka cluster and an Airflow container, respectively.
Key Components and Their Interactions
• Data Producer / Ingestion:
• API: Provides random name data (represented by the sample response.json).
• Airflow DAG (stream_to_kafka_dag.py): Orchestrates the data ingestion by periodically triggering stream_to_kafka.py to pull API data and send it to Kafka.
• This part of the pipeline is clearly managed as a DAG. (So answer the question in the custom instruction: yes, the data ingestion is DAG-driven.)
• Messaging Middleware:
• Kafka Cluster: Represented by three nodes as configured in docker-compose.yml (e.g., kafka1, kafka2, kafka3) and includes a Kafka UI for monitoring.
• The data flows from the Airflow DAG (producer) into the Kafka topic (e.g., "random_names").
• Data Consumer / Processing:
• Spark Process: The spark_streaming.py script uses Spark Structured Streaming to consume data from the Kafka topic.
• Custom instruction question “spark process도 dag인가?” – In the current design, the Spark streaming process is not managed by Airflow as a DAG; it’s executed manually (via spark-submit) in the Spark container. However, you may integrate it as an additional DAG task if a fully orchestrated workflow is desired.
• Database Storage:
• MySQL: Replace the original Cassandra storage. The Spark consumer will write the processed data (random names) into a MySQL table. Ensure that the connection parameters and table definition are updated accordingly in the Spark script.
Architectural Patterns and Design Principles
• Containerization: Use Docker Compose for orchestrating the Kafka cluster, the Airflow environment, (and now MySQL instead of Cassandra), and Spark containers.
• Decoupled Components: Data ingestion is separated from the data processing (stream consumer), and communication occurs via Kafka topics.
• Orchestration with DAGs: The ingestion process is scheduled with an Airflow DAG, which allows for monitoring and scheduling. (At present, only the ingestion process is orchestrated; the Spark job is a manual or external submission.)
• Extensibility: If desired, both the data ingestion and Spark processing could be unified into a single orchestration framework by wrapping the Spark submit command within an Airflow DAG.