# ayoubnajjout/etl-data-pipeline architecture

> Architecture diagram of the GitHub repository ayoubnajjout/etl-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayoubnajjout/etl-data-pipeline
- Repository: https://github.com/ayoubnajjout/etl-data-pipeline

## Overview

##### Project Type and Purpose
- This is an ELT (Extract-Load-Transform) data pipeline orchestration project.
- Purpose: ingest real-time data via Kafka, orchestrate workflows with Airflow, transform with Spark, and persist results in Cassandra.

##### File Structure
- Top-level artifacts:
- docker-compose.yaml – defines all service containers (Airflow, Kafka, Spark, Cassandra).
- airflow/
- dags/kafka_dag.py – Airflow DAG integrating Kafka ingestion, Spark jobs, and Cassandra writes.
- logs/ – runtime logs for DAG executions.
- cassandra/init.cql – CQL schema initialization for the Cassandra keyspace/table.
- spark/stream_job.py – Spark streaming job that processes messages from Kafka and writes to Cassandra.

##### README for Additional Insights
- Confirms the four core components (Airflow, Kafka, Spark, Cassandra).
- Usage notes: where to drop DAGs, modify schema, or add Spark jobs.

## Diagram (Mermaid)

```mermaid
flowchart TB
    externalProducers["Data Producers"]:::external
    subgraph "Docker Network"
        direction TB
        AirflowScheduler["Airflow Scheduler"]:::orchestration
        AirflowWeb["Airflow Webserver"]:::orchestration
        KafkaDAG["Airflow DAG (kafka_dag.py)"]:::orchestration
        KafkaBroker["Kafka Broker:9092"]:::streaming
        SparkJob["Spark Streaming Job"]:::streaming
        CassandraCluster["Cassandra Cluster:9042"]:::storage
        AirflowLogs["Airflow Logs"]:::infra
        DiagramGen["Architecture Diagram Generator"]:::infra
        DockerCompose["docker-compose.yaml"]:::infra
        README["Project README"]:::infra
    end

    externalProducers --> KafkaBroker
    KafkaBroker -->|"sensor"| KafkaDAG
    AirflowScheduler --> KafkaDAG
    KafkaDAG -->|"submit Spark job"| SparkJob
    SparkJob -->|"read stream"| KafkaBroker
    SparkJob -->|"write results"| CassandraCluster
    KafkaDAG -->|"log status"| AirflowLogs

    click KafkaDAG "https://github.com/ayoubnajjout/etl-data-pipeline/blob/main/airflow/dags/kafka_dag.py"
    click SparkJob "https://github.com/ayoubnajjout/etl-data-pipeline/blob/main/spark/stream_job.py"
    click CassandraCluster "https://github.com/ayoubnajjout/etl-data-pipeline/blob/main/cassandra/init.cql"
    click DockerCompose "https://github.com/ayoubnajjout/etl-data-pipeline/blob/main/docker-compose.yaml"
    click DiagramGen "https://github.com/ayoubnajjout/etl-data-pipeline/blob/main/diagram.py"
    click README "https://github.com/ayoubnajjout/etl-data-pipeline/blob/main/README.md"

    classDef orchestration fill:#0074D9,stroke:#005fa3,color:#fff
    classDef streaming fill:#FF851B,stroke:#cc7000,color:#fff
    classDef storage fill:#2ECC40,stroke:#28a536,color:#fff
    classDef infra fill:#AAAAAA,stroke:#888888,color:#000
    classDef external fill:#DDDDDD,stroke:#BBBBBB,color:#000
```
