How mac-199902/urbanbrew-pipeline works
Urban Brew simulates café orders, validates and publishes them through a Django API, then fans the Kafka stream out to operational storage, historical storage, and live alerts. Spark processes the historical data into analytics and order-risk recommendations; the authenticated dashboard combines those batch results with current orders and alerts. The README documents these flows, while sampled source confirms key API, consumer, and dashboard behavior. Kafka Connect is described as an optional alternative, but its implementation is absent from the tree, so it is not mapped.
Main entry points: mysql_writer.py MySQL writer · dashboard/views.py Dashboard data views · api/views.py Order API · hdfs_writer.py HDFS writer
Components 19
Order ingestion
- Order simulator
urbanbrew/autoscript/simulate.py - Order API
urbanbrew/api/views.py - Order validation
urbanbrew/api/serializers.py - Kafka publisher
urbanbrew/api/kafka_producer.py - Order records
urbanbrew/api/models.py - Shared configuration
urbanbrew/config.py
Kafka consumers
- MySQL writer
urbanbrew/consumer/mysql_writer.py - HDFS writer
urbanbrew/consumer/hdfs_writer.py - Alert notifier
urbanbrew/consumer/notifier.py - Consumer plumbing
urbanbrew/consumer/kafka_utils.py - Live alerts
urbanbrew/api/models.py
Analytics
- Spark analytics and risk
urbanbrew/spark_jobs/analytics_job.py - Analytics results
urbanbrew/dashboard/models.py
Dashboard
- Dashboard data views
urbanbrew/dashboard/views.py - Dashboard page
urbanbrew/dashboard/templates/dashboard/dashboard.html
Other components
- Kafka topic
- MySQL
- HDFS
- Manager
Connections 23
- Order simulator → Order API: POSTs orders
urbanbrew/autoscript/simulate.py - Order API → Order validation: validates with
urbanbrew/api/views.py - Order API → Kafka publisher: publishes via
urbanbrew/api/views.py - Kafka publisher → Kafka topic: publishes orders
urbanbrew/api/kafka_producer.py - MySQL writer → Kafka topic: consumes orders
README.md - HDFS writer → Kafka topic: consumes orders
README.md - Alert notifier → Kafka topic: consumes orders
urbanbrew/consumer/notifier.py - MySQL writer → Consumer plumbing: uses
urbanbrew/consumer/mysql_writer.py - HDFS writer → Consumer plumbing: uses
urbanbrew/consumer/hdfs_writer.py - Alert notifier → Consumer plumbing: uses
urbanbrew/consumer/notifier.py - MySQL writer → Order records: creates records
urbanbrew/consumer/mysql_writer.py - MySQL writer → MySQL: writes orders
README.md - HDFS writer → HDFS: writes order files
README.md - Alert notifier → Live alerts: creates alerts
urbanbrew/consumer/notifier.py - Spark analytics and risk → HDFS: reads order files
README.md - Spark analytics and risk → MySQL: writes analytics
README.md - Dashboard data views → Order records: queries orders
urbanbrew/dashboard/views.py - Dashboard data views → Live alerts: queries alerts
urbanbrew/dashboard/views.py - Dashboard data views → Analytics results: queries analytics
urbanbrew/dashboard/views.py - Dashboard data views → Dashboard page: serves data to
- Manager → Dashboard page: views
- Order API → Shared configuration: loads settings
- Spark analytics and risk → Shared configuration: loads settings
urbanbrew/spark_jobs/analytics_job.py