How mekwiset/realtime_data_streaming works
Main Components
• External Data Source: Mark the randomuser API as the data origin.
• Orchestration and Scheduling: Highlight the Airflow DAG (located in the dags folder, specifically stream_to_kafka.py) which triggers the data ingestion.
• Data Ingestion (Producer): Identify the Kafka producer logic in jobs/kafka_stream.py and plugins/kafka_streamer.py.
• Data Processing (Consumer): Represent the Spark streaming job that consumes data from Kafka (jobs/spark_stream.py) and the corresponding plugins (spark_streamer.py).
• Data Storage: Show Cassandra as the final destination for processed data (with PostgreSQL mentioned as an intermediate store per the README, though the table creation script is in helpers).
• Container Environment: Include Docker and Docker Compose (noting docker-compose.yml and script/entrypoint.sh) to emphasize how each component runs in its own container.
Relationships and Interactions
• Airflow (dags) to the Kafka producer, which streams the data into Apache Kafka.
• Flow from Apache Kafka (which is further monitored via Confluent’s Control Center and Schema Registry) to Apache Spark for real-time processing.
Important Architectural Patterns and Design Principles
• Containerization: Clearly show that each major service (Airflow, Kafka, Spark, Cassandra, and Confluent) is isolated within Docker containers as orchestrated by docker-compose.
• Real-time Data Flow: Emphasize the continuous streaming pipeline – from data fetch, production to Kafka, real-time processing with Spark, and finally storage.
• Modular Design: Recognize that the project separates concerns neatly by isolating DAGs (Airflow related code), job scripts (stream processing), and plugins (custom integrations and utilities). This points to maintainability and extensibility in the design.
• Configuration Driven: Note the presence of config.yaml and a custom configparser under plugins/utils to show that the setup is environment-specific and easily adjustable.
Technologies, Frameworks, and Libraries
• Apache Airflow, Apache Kafka, and Apache Spark as the core processing engines.
• Cassandra (and optionally PostgreSQL) for data storage.
• Docker and Docker Compose for container orchestration.
• Confluent tools for Kafka monitoring.