How ayoubnajjout/etl-data-pipeline works
Project Type and Purpose
• This is an ELT (Extract-Load-Transform) data pipeline orchestration project.
• Purpose: ingest real-time data via Kafka, orchestrate workflows with Airflow, transform with Spark, and persist results in Cassandra.
File Structure
• Top-level artifacts:
• docker-compose.yaml – defines all service containers (Airflow, Kafka, Spark, Cassandra).
• airflow/
• dags/kafka_dag.py – Airflow DAG integrating Kafka ingestion, Spark jobs, and Cassandra writes.
• logs/ – runtime logs for DAG executions.
• cassandra/init.cql – CQL schema initialization for the Cassandra keyspace/table.
• spark/stream_job.py – Spark streaming job that processes messages from Kafka and writes to Cassandra.
README for Additional Insights
• Confirms the four core components (Airflow, Kafka, Spark, Cassandra).
• Usage notes: where to drop DAGs, modify schema, or add Spark jobs.