# noggrj/spark-elt architecture

> Architecture diagram of the GitHub repository noggrj/spark-elt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/noggrj/spark-elt
- Repository: https://github.com/noggrj/spark-elt

## Overview

##### Project type and purpose
- This is an ELT (Extract‐Load‐Transform) streaming data pipeline combining:
- Data generation of synthetic customer and transaction records
- Real‐time messaging with Apache Kafka
- Streaming transformations with Apache Spark (PySpark)
- Orchestration and scheduling with Apache Airflow
- Containerized deployment via Docker/Docker Compose
- Purpose: simulate, ingest, validate, transform and orchestrate financial transaction streams end‐to‐end.

##### File structure
- Top-level artifacts:
- Data/ - raw CSV inputs
- generator/ - scripts to produce synthetic data
- kafka_module/ - Kafka producers and an optional REST webservice
- extract/ - extraction logic reading CSVs and pushing to Kafka
- pipeline/ - wrapper or entrypoint orchestrating Kafka+Spark pipeline
- transform/ - PySpark code for streaming transformations
- dags/ - Airflow DAG definitions for scheduling tasks
- main.py - script to run the full pipeline outside Airflow
- Dockerfile(.airflow), docker-compose.yml, Makefile - container build and orchestration
- Patterns:
- Modular separation of concerns (generation, extraction, messaging, processing, orchestration)
- Use of directory names to reflect “layers” of the pipeline
- Presence of configuration and requirements files for reproducible environments

##### README for additional insights
- Emphasizes CSV validation at extraction, transformation via Spark, and DAG orchestration in Airflow
- Docker Compose starts Kafka brokers, Zookeeper, Spark, Airflow webserver/scheduler, and other services

##### Elements
- Rectangles for data services (Kafka, Spark)
- Clouds or parallelograms for processes (generator, extract modules)
- Diamonds or annotations for Airflow DAG control
- Related containers (e.g., within a “Docker Compose” boundary)
- Notes on ports or topic names (e.g., “topic: clientes”, “topic: transacoes”)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Components
    subgraph "Data Components"
        direction TB
        RAW["Raw CSV Data"]:::storage
        subgraph "Data Generator"
            direction TB
            DGC["create-data-clientes.py"]:::generator
            DGT["create-data-transacoes.py"]:::generator
            DGD["data_generator.py"]:::generator
        end
        subgraph "Extract Module"
            direction TB
            EC["clientes.py"]:::extract
            ET["transacoes.py"]:::extract
            EP["producers.py"]:::extract
        end
        SINK["Data Sink"]:::storage
    end

    %% Docker Compose Services
    subgraph "Docker Compose Network"
        direction TB
        KAFKA["Kafka Cluster\n(Zookeeper & Brokers)"]:::kafka
        SPARK["Spark Streaming App"]:::spark
        WEBSVC["Kafka REST Webservice"]:::service
        AIRFLOW["Airflow Orchestrator"]:::airflow
    end

    %% Data Flow
    DGC --> RAW
    DGT --> RAW
    DGD --> RAW
    RAW --> EC
    RAW --> ET
    EC --> KAFKA
    ET --> KAFKA
    EP --> KAFKA
    WEBSVC --> KAFKA
    KAFKA --> SPARK
    SPARK --> SINK

    %% Orchestration Controls
    AIRFLOW -.-> DGC
    AIRFLOW -.-> EC
    AIRFLOW -.-> ET
    AIRFLOW -.-> SPARK
    AIRFLOW -.-> WEBSVC

    %% Click Events
    click DGC "https://github.com/noggrj/spark-elt/blob/main/generator/create-data-clientes.py"
    click DGT "https://github.com/noggrj/spark-elt/blob/main/generator/create-data-transacoes.py"
    click DGD "https://github.com/noggrj/spark-elt/blob/main/generator/data_generator.py"
    click EC "https://github.com/noggrj/spark-elt/blob/main/extract/clientes.py"
    click ET "https://github.com/noggrj/spark-elt/blob/main/extract/transacoes.py"
    click EP "https://github.com/noggrj/spark-elt/blob/main/kafka_module/producers.py"
    click KAFKA "https://github.com/noggrj/spark-elt/blob/main/docker-compose.yml"
    click SPARK "https://github.com/noggrj/spark-elt/blob/main/transform/spark_processing.py"
    click SPARK "https://github.com/noggrj/spark-elt/blob/main/pipeline/kafka_pipeline.py"
    click AIRFLOW "https://github.com/noggrj/spark-elt/blob/main/dags/kafka_pipeline_dag.py"
    click AIRFLOW "https://github.com/noggrj/spark-elt/blob/main/dags/kafka_monitoring_dag.py"
    click AIRFLOW "https://github.com/noggrj/spark-elt/blob/main/Dockerfile.airflow"
    click AIRFLOW "https://github.com/noggrj/spark-elt/blob/main/requirements-airflow.txt"
    click WEBSVC "https://github.com/noggrj/spark-elt/blob/main/kafka_module/webservice.py"
    click RAW "https://github.com/noggrj/spark-elt/blob/main/Data/raw/clientes.csv"
    click RAW "https://github.com/noggrj/spark-elt/blob/main/Data/raw/transacoes.csv"
    click SINK "https://github.com/noggrj/spark-elt/tree/main/Makefile"
    click SINK "https://github.com/noggrj/spark-elt/tree/main/Dockerfile"

    %% Styles
    classDef generator fill:#fdebd0,stroke:#e67e22
    classDef extract fill:#d6eaf8,stroke:#5dade2
    classDef kafka fill:#f9e79f,stroke:#f1c40f
    classDef spark fill:#e8f8f5,stroke:#17a589
    classDef airflow fill:#fadbd8,stroke:#c0392b,stroke-dasharray: 5 5
    classDef service fill:#d5f5e3,stroke:#27ae60
    classDef storage fill:#fcf3cf,stroke:#f39c12
    class RAW,SINK storage
    class EP kafka
    class KAFKA kafka
    class WEBSVC service
    class SPARK spark
    class AIRFLOW airflow
```
