# lnynhi02/coffee-sales-data-pipeline architecture

> Architecture diagram of the GitHub repository lnynhi02/coffee-sales-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lnynhi02/coffee-sales-data-pipeline
- Repository: https://github.com/lnynhi02/coffee-sales-data-pipeline

## Overview

##### Project type and purpose
- This is a Coffee Sales Data Pipeline combining batch and real-time ingestion, transformation, storage, and monitoring.
- Batch ETL is orchestrated via Apache Airflow and Spark/Python scripts (scripts/batch), loading raw CSVs into Bronze/Silver/Gold tables.
- Real-time events are produced/consumed via Kafka and sink into MySQL (scripts/real-time + Kafka Connect).
- Observability is provided by Prometheus (with JMX exporters) and Grafana for dashboards.

##### File structure
- Top-level “airflow/” contains Dockerfile and DAG definitions, pointing to a containerized Airflow scheduler.
- “scripts/batch/” implements Bronze → Silver → Gold layering and data–quality validation.
- “scripts/real-time/” holds Kafka client code, handler, and a JSON config for Kafka-to-MySQL connector.
- “scripts/database/” automates schema creation, data generation, lookup caching, and status updates.
- “docker-compose.yaml” and “docker-compose-batch.yaml” define multi-container environments for real-time vs batch stacks.
- “volumes/” contains persistent storage for Grafana, Prometheus, JMX exporters, and Kafka Connect state.

##### README
- Minimal instructions, points to hosted deployment docs.
- Confirms the overall goal: stand up the full pipeline end-to-end.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        direction TB
        dcCSV[/diamond_customers.csv/] 
        pmCSV[/payment_method.csv/] 
        pcCSV[/product_category.csv/] 
        prCSV[/products.csv/] 
        stCSV[/stores.csv/] 
        gen["Synthetic Data Generator"] 
        schema["Schema & Static Load Scripts"] 
        createTbl["create_table.py"] 
        loadStatic["load_static_file.py"] 
        mysqlConf["mysql_config.py"] 
    end

    %% Batch Processing Lane
    subgraph "Batch Processing" 
        direction TB
        Airflow["Airflow Scheduler & DAGs"] 
        SparkJob["Spark Job (Bronze Load)"] 
        BronzeVal["Bronze Validation"] 
        SilverDim["Silver Dimensions"] 
        SilverFact["Silver Facts"] 
        SilverVal["Silver Validation"] 
        GoldDimPay["Gold Dim Payment"] 
        GoldDimProd["Gold Dim Products"] 
        GoldDimStore["Gold Dim Stores"] 
        GoldFact["Gold Fact Orders"] 
        Utils["Shared Utilities"] 
    end

    %% Real-Time Processing Lane
    subgraph "Real-Time Processing"
        direction TB
        KafkaBroker("Kafka Broker") 
        KafkaClient["Kafka Client"] 
        KafkaHandler["Kafka Handler"] 
        ConnectCfg["Kafka Connect Config"] 
        WinScript["Windows Helper Script"] 
    end

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        MySQL[(MySQL Database)] 
        ConnectorJar["mysql-connector-j-8.0.33.jar"] 
    end

    %% Monitoring
    subgraph "Monitoring"
        direction TB
        Prometheus["Prometheus"] 
        JMXBroker["JMX Exporter (Kafka Broker)"] 
        JMXConnect["JMX Exporter (Kafka Connect)"] 
        Grafana["Grafana Dashboards"] 
    end

    %% Infrastructure
    subgraph "Infrastructure"
        direction TB
        DockerCompose["Docker Compose Definitions"] 
    end

    %% Connections: Data Sources to Batch
    dcCSV -->|"/airflow/dags spark_job trigger"| Airflow
    pmCSV -->|"/airflow/dags spark_job trigger"| Airflow
    pcCSV -->|"/airflow/dags spark_job trigger"| Airflow
    prCSV -->|"/airflow/dags spark_job trigger"| Airflow
    stCSV -->|"/airflow/dags spark_job trigger"| Airflow
    gen -->|"/scripts/database generate data"| schema
    schema -->|"/scripts/database create schema"| Airflow

    %% Batch Flow
    Airflow -->|"trigger Spark"| SparkJob
    SparkJob -->|"/scripts/batch/bronze_dimension_fact_load.py"| BronzeVal
    BronzeVal -->|"if passes"| SilverDim
    BronzeVal -->|"if passes"| SilverFact
    SilverDim -->|"/scripts/batch/silver_validation.py"| SilverVal
    SilverFact -->|"/scripts/batch/silver_validation.py"| SilverVal
    SilverVal -->|"load to gold"| GoldDimPay
    SilverVal -->|"load to gold"| GoldDimProd
    SilverVal -->|"load to gold"| GoldDimStore
    GoldDimPay -->|"/scripts/batch/gold_fact_orders.py"| GoldFact
    GoldDimProd -->|"data ready"| MySQL
    GoldDimStore -->|"data ready"| MySQL
    GoldFact -->|"data ready"| MySQL
    Utils -->|"/scripts/batch/utils.py"| Airflow

    %% Real-Time Flow
    KafkaClient -->|"produce events"| KafkaBroker
    KafkaHandler -->|"consume events"| KafkaBroker
    KafkaBroker -->|"sink via Connect"| ConnectCfg
    ConnectCfg -->|"JDBC"| MySQL
    WinScript -->|"setup Connect"| ConnectCfg

    %% Monitoring Flow
    Prometheus -->|"scrape JMX"| JMXBroker
    Prometheus -->|"scrape JMX"| JMXConnect
    Prometheus -->|"scrape metrics"| MySQL
    Grafana -->|"query Prometheus"| Prometheus

    %% Infrastructure Links
    DockerCompose -->|deploy| Airflow
    DockerCompose -->|deploy| KafkaBroker
    DockerCompose -->|deploy| Prometheus
    DockerCompose -->|deploy| Grafana

    %% Click Events
    click dcCSV "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/data/diamond_customers.csv"
    click pmCSV "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/data/payment_method.csv"
    click pcCSV "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/data/product_category.csv"
    click prCSV "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/data/products.csv"
    click stCSV "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/data/stores.csv"
    click gen "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/database/generate_data.py"
    click createTbl "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/database/create_table.py"
    click loadStatic "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/database/load_static_file.py"
    click mysqlConf "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/database/mysql_config.py"
    click Airflow "https://github.com/lnynhi02/coffee-sales-data-pipeline/tree/main/airflow/Dockerfile"
    click Airflow "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/airflow/dags/spark_job_airflow.py"
    click SparkJob "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/bronze_dimension_fact_load.py"
    click BronzeVal "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/data_quality/bronze_validation.py"
    click SilverVal "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/data_quality/silver_validation.py"
    click SilverDim "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/silver_dimensions.py"
    click SilverFact "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/silver_facts.py"
    click GoldDimPay "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/gold_dim_payment.py"
    click GoldDimProd "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/gold_dim_products.py"
    click GoldDimStore "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/gold_dim_stores.py"
    click GoldFact "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/gold_fact_orders.py"
    click Utils "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/batch/utils.py"
    click KafkaClient "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/real-time/kafka_client.py"
    click KafkaHandler "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/real-time/kafka_handler.py"
    click ConnectCfg "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/scripts/real-time/mysql-src-connector.json"
    click WinScript "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/real-time.ps1"
    click ConnectorJar "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/mysql-connector-j-8.0.33.jar"
    click DockerCompose "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/docker-compose.yaml"
    click DockerCompose "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/docker-compose-batch.yaml"
    click Prometheus "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/volumes/monitoring/prometheus.yml"
    click JMXBroker "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/volumes/jmx-exporter/kafka-broker.yml"
    click JMXConnect "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/volumes/jmx-exporter/kafka-connect.yml"
    click Grafana "https://github.com/lnynhi02/coffee-sales-data-pipeline/blob/main/volumes/monitoring/grafana/provisioning/datasources/prometheus_ds.yml"

    %% Styles
    classDef batch fill:#a3d5ff,stroke:#036;
    classDef streaming fill:#ffcc99,stroke:#c90;
    classDef storage fill:#c3f9c3,stroke:#090;
    classDef monitoring fill:#e1c3f9,stroke:#70c;
    classDef infra fill:#e8e8e8,stroke:#888;
    class Airflow,SparkJob,BronzeVal,SilverDim,SilverFact,SilverVal,GoldDimPay,GoldDimProd,GoldDimStore,GoldFact,Utils batch
    class KafkaBroker,KafkaClient,KafkaHandler,ConnectCfg,WinScript streaming
    class MySQL,ConnectorJar storage
    class Prometheus,JMXBroker,JMXConnect,Grafana monitoring
    class DockerCompose infra
```
