How lnynhi02/coffee-sales-data-pipeline works
Project type and purpose
• This is a Coffee Sales Data Pipeline combining batch and real-time ingestion, transformation, storage, and monitoring.
• Batch ETL is orchestrated via Apache Airflow and Spark/Python scripts (scripts/batch), loading raw CSVs into Bronze/Silver/Gold tables.
• Real-time events are produced/consumed via Kafka and sink into MySQL (scripts/real-time + Kafka Connect).
• Observability is provided by Prometheus (with JMX exporters) and Grafana for dashboards.
File structure
• Top-level “airflow/” contains Dockerfile and DAG definitions, pointing to a containerized Airflow scheduler.
• “scripts/batch/” implements Bronze → Silver → Gold layering and data–quality validation.
• “scripts/real-time/” holds Kafka client code, handler, and a JSON config for Kafka-to-MySQL connector.
• “scripts/database/” automates schema creation, data generation, lookup caching, and status updates.
• “docker-compose.yaml” and “docker-compose-batch.yaml” define multi-container environments for real-time vs batch stacks.
• “volumes/” contains persistent storage for Grafana, Prometheus, JMX exporters, and Kafka Connect state.
README
• Minimal instructions, points to hosted deployment docs.
• Confirms the overall goal: stand up the full pipeline end-to-end.