# mac-199902/urbanbrew-pipeline architecture

> Architecture diagram of the GitHub repository mac-199902/urbanbrew-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mac-199902/urbanbrew-pipeline
- Repository: https://github.com/mac-199902/urbanbrew-pipeline

## Overview

Urban Brew simulates café orders, validates and publishes them through a Django API, then fans the Kafka stream out to operational storage, historical storage, and live alerts. Spark processes the historical data into analytics and order-risk recommendations; the authenticated dashboard combines those batch results with current orders and alerts. The README documents these flows, while sampled source confirms key API, consumer, and dashboard behavior. Kafka Connect is described as an optional alternative, but its implementation is absent from the tree, so it is not mapped.

Main entry points: [`urbanbrew/consumer/mysql_writer.py`](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/mysql_writer.py) (MySQL writer), [`urbanbrew/dashboard/views.py`](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/dashboard/views.py) (Dashboard data views), [`urbanbrew/api/views.py`](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/views.py) (Order API), [`urbanbrew/consumer/hdfs_writer.py`](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/hdfs_writer.py) (HDFS writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Order ingestion"]
  node_autoscript["Order simulator<br/>[simulate.py]"]
  node_api["Order API<br/>[views.py]"]
  node_serializer["Order validation<br/>[serializers.py]"]
  node_producer["Kafka publisher<br/>[kafka_producer.py]"]
  node_order_model["Order records<br/>[models.py]"]
  node_config["Shared configuration<br/>[config.py]"]
end

subgraph group_stream["Kafka consumers"]
  node_mysql_writer["MySQL writer<br/>[mysql_writer.py]"]
  node_hdfs_writer["HDFS writer<br/>[hdfs_writer.py]"]
  node_notifier["Alert notifier<br/>[notifier.py]"]
  node_consumer_helper["Consumer plumbing<br/>[kafka_utils.py]"]
  node_live_alerts["Live alerts<br/>[models.py]"]
end

subgraph group_analytics["Analytics"]
  node_spark["Spark analytics and risk<br/>[analytics_job.py]"]
  node_analytics_tables["Analytics results<br/>[models.py]"]
end

subgraph group_presentation["Dashboard"]
  node_dashboard_views["Dashboard data views<br/>[views.py]"]
  node_dashboard_page["Dashboard page<br/>[dashboard.html]"]
end

node_kafka["Kafka topic"]
node_mysql[("MySQL")]
node_hdfs[("HDFS")]
node_manager(("Manager"))

node_autoscript -->|"POSTs orders"| node_api
node_api -->|"validates with"| node_serializer
node_api -->|"publishes via"| node_producer
node_producer -->|"publishes orders"| node_kafka
node_mysql_writer -->|"consumes orders"| node_kafka
node_hdfs_writer -->|"consumes orders"| node_kafka
node_notifier -->|"consumes orders"| node_kafka
node_mysql_writer -->|"uses"| node_consumer_helper
node_hdfs_writer -->|"uses"| node_consumer_helper
node_notifier -->|"uses"| node_consumer_helper
node_mysql_writer -->|"creates records"| node_order_model
node_mysql_writer -->|"writes orders"| node_mysql
node_hdfs_writer -->|"writes order files"| node_hdfs
node_notifier -->|"creates alerts"| node_live_alerts
node_spark -->|"reads order files"| node_hdfs
node_spark -->|"writes analytics"| node_mysql
node_dashboard_views -->|"queries orders"| node_order_model
node_dashboard_views -->|"queries alerts"| node_live_alerts
node_dashboard_views -->|"queries analytics"| node_analytics_tables
node_dashboard_views -.->|"serves data to"| node_dashboard_page
node_manager -.->|"views"| node_dashboard_page
node_api -.->|"loads settings"| node_config
node_spark -->|"loads settings"| node_config

click node_autoscript "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/autoscript/simulate.py"
click node_api "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/views.py"
click node_serializer "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/serializers.py"
click node_producer "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/kafka_producer.py"
click node_order_model "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/models.py"
click node_mysql_writer "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/mysql_writer.py"
click node_hdfs_writer "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/hdfs_writer.py"
click node_notifier "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/notifier.py"
click node_consumer_helper "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/kafka_utils.py"
click node_live_alerts "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/models.py"
click node_spark "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/spark_jobs/analytics_job.py"
click node_analytics_tables "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/dashboard/models.py"
click node_dashboard_views "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/dashboard/views.py"
click node_dashboard_page "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/dashboard/templates/dashboard/dashboard.html"
click node_config "https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_autoscript,node_api,node_serializer,node_producer,node_order_model,node_config toneBlue
class node_mysql_writer,node_hdfs_writer,node_notifier,node_consumer_helper,node_live_alerts,node_mysql,node_hdfs toneAmber
class node_spark,node_analytics_tables toneMint
class node_dashboard_views,node_dashboard_page toneRose
class node_kafka,node_manager toneIndigo
```

## Components

### Order ingestion

- [Order simulator](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/autoscript/simulate.py): `urbanbrew/autoscript/simulate.py`
- [Order API](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/views.py): `urbanbrew/api/views.py`
- [Order validation](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/serializers.py): `urbanbrew/api/serializers.py`
- [Kafka publisher](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/kafka_producer.py): `urbanbrew/api/kafka_producer.py`
- [Order records](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/models.py): `urbanbrew/api/models.py`
- [Shared configuration](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/config.py): `urbanbrew/config.py`

### Kafka consumers

- [MySQL writer](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/mysql_writer.py): `urbanbrew/consumer/mysql_writer.py`
- [HDFS writer](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/hdfs_writer.py): `urbanbrew/consumer/hdfs_writer.py`
- [Alert notifier](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/notifier.py): `urbanbrew/consumer/notifier.py`
- [Consumer plumbing](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/consumer/kafka_utils.py): `urbanbrew/consumer/kafka_utils.py`
- [Live alerts](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/api/models.py): `urbanbrew/api/models.py`

### Analytics

- [Spark analytics and risk](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/spark_jobs/analytics_job.py): `urbanbrew/spark_jobs/analytics_job.py`
- [Analytics results](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/dashboard/models.py): `urbanbrew/dashboard/models.py`

### Dashboard

- [Dashboard data views](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/dashboard/views.py): `urbanbrew/dashboard/views.py`
- [Dashboard page](https://github.com/mac-199902/urbanbrew-pipeline/blob/main/urbanbrew/dashboard/templates/dashboard/dashboard.html): `urbanbrew/dashboard/templates/dashboard/dashboard.html`

### Other components

- **Kafka topic**
- **MySQL**
- **HDFS**
- **Manager**

## Connections

- Order simulator → Order API: POSTs orders (evidence: `urbanbrew/autoscript/simulate.py`)
- Order API → Order validation: validates with (evidence: `urbanbrew/api/views.py`)
- Order API → Kafka publisher: publishes via (evidence: `urbanbrew/api/views.py`)
- Kafka publisher → Kafka topic: publishes orders (evidence: `urbanbrew/api/kafka_producer.py`)
- MySQL writer → Kafka topic: consumes orders (evidence: `README.md`)
- HDFS writer → Kafka topic: consumes orders (evidence: `README.md`)
- Alert notifier → Kafka topic: consumes orders (evidence: `urbanbrew/consumer/notifier.py`)
- MySQL writer → Consumer plumbing: uses (evidence: `urbanbrew/consumer/mysql_writer.py`)
- HDFS writer → Consumer plumbing: uses (evidence: `urbanbrew/consumer/hdfs_writer.py`)
- Alert notifier → Consumer plumbing: uses (evidence: `urbanbrew/consumer/notifier.py`)
- MySQL writer → Order records: creates records (evidence: `urbanbrew/consumer/mysql_writer.py`)
- MySQL writer → MySQL: writes orders (evidence: `README.md`)
- HDFS writer → HDFS: writes order files (evidence: `README.md`)
- Alert notifier → Live alerts: creates alerts (evidence: `urbanbrew/consumer/notifier.py`)
- Spark analytics and risk → HDFS: reads order files (evidence: `README.md`)
- Spark analytics and risk → MySQL: writes analytics (evidence: `README.md`)
- Dashboard data views → Order records: queries orders (evidence: `urbanbrew/dashboard/views.py`)
- Dashboard data views → Live alerts: queries alerts (evidence: `urbanbrew/dashboard/views.py`)
- Dashboard data views → Analytics results: queries analytics (evidence: `urbanbrew/dashboard/views.py`)
- Dashboard data views → Dashboard page: serves data to
- Manager → Dashboard page: views
- Order API → Shared configuration: loads settings
- Spark analytics and risk → Shared configuration: loads settings (evidence: `urbanbrew/spark_jobs/analytics_job.py`)
