How noggrj/spark-elt works
Project type and purpose
• This is an ELT (Extract‐Load‐Transform) streaming data pipeline combining:
• Data generation of synthetic customer and transaction records
• Real‐time messaging with Apache Kafka
• Streaming transformations with Apache Spark (PySpark)
• Orchestration and scheduling with Apache Airflow
• Containerized deployment via Docker/Docker Compose
• Purpose: simulate, ingest, validate, transform and orchestrate financial transaction streams end‐to‐end.
File structure
• Top-level artifacts:
• Data/ - raw CSV inputs
• generator/ - scripts to produce synthetic data
• kafka_module/ - Kafka producers and an optional REST webservice
• extract/ - extraction logic reading CSVs and pushing to Kafka
• pipeline/ - wrapper or entrypoint orchestrating Kafka+Spark pipeline
• transform/ - PySpark code for streaming transformations
• dags/ - Airflow DAG definitions for scheduling tasks
• main.py - script to run the full pipeline outside Airflow
• Dockerfile(.airflow), docker-compose.yml, Makefile - container build and orchestration
• Patterns:
• Modular separation of concerns (generation, extraction, messaging, processing, orchestration)
• Use of directory names to reflect “layers” of the pipeline
• Presence of configuration and requirements files for reproducible environments
README for additional insights
• Emphasizes CSV validation at extraction, transformation via Spark, and DAG orchestration in Airflow
• Docker Compose starts Kafka brokers, Zookeeper, Spark, Airflow webserver/scheduler, and other services
Elements
• Rectangles for data services (Kafka, Spark)
• Clouds or parallelograms for processes (generator, extract modules)
• Diamonds or annotations for Airflow DAG control
• Related containers (e.g., within a “Docker Compose” boundary)
• Notes on ports or topic names (e.g., “topic: clientes”, “topic: transacoes”)