# row-76/bigdata-pipelinerr architecture

> Architecture diagram of the GitHub repository row-76/bigdata-pipelinerr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/row-76/bigdata-pipelinerr
- Repository: https://github.com/row-76/bigdata-pipelinerr

## Overview

#### Purpose and Entry Points
This is a compact streaming-data repository centered on event production, Spark-based stream processing, and Prometheus monitoring. Primary executable entry points are `producer.py` for generating/publishing input events and `spark_stream.py` for consuming and processing the stream.

#### Event Production
`producer.py` is the ingestion boundary. It creates the source-side event flow and is the likely place to define message schema, production cadence, and connection details for the downstream broker or stream transport.

#### Stream Processing
`spark_stream.py` is the processing boundary and likely hosts the Spark Structured Streaming job. It consumes produced events, applies transformations or aggregations, and writes results to its configured sink. Spark is the material runtime for stateful or distributed stream computation.

#### Runtime Topology
`docker-compose.yml` defines the local multi-service deployment boundary. It is the central integration point for the producer, Spark workload, and any required supporting services such as a message broker, storage, or metrics endpoints. Service names, network configuration, and startup dependencies should be treated as the authoritative runtime contract.

#### Observability
`prometheus.yml` defines Prometheus scraping configuration. Prometheus is the monitoring runtime and should collect metrics from services exposed through the Compose topology, potentially including Spark or application-level metrics endpoints.

#### Primary Control and Data Flow
1. `producer.py` emits streaming events to the configured transport.
2. `spark_stream.py` reads that event stream and executes streaming transformations.
3. The Spark job writes processed output to its configured destination.
4. Prometheus scrapes metrics according to `prometheus.yml`.
5. `docker-compose.yml` coordinates local service startup and connectivity across these stages.

#### Important Boundaries
The key contract boundaries are producer-to-stream transport, transport-to-Spark consumption, and Spark-to-output sink. Changes to event shape or broker/topic configuration must remain compatible between `producer.py` and `spark_stream.py`; infrastructure endpoint and port changes must remain aligned with `docker-compose.yml` and `prometheus.yml`.

Main entry points: [`docker-compose.yml`](https://github.com/row-76/bigdata-pipelinerr/blob/main/docker-compose.yml) (Compose Topology), [`spark_stream.py`](https://github.com/row-76/bigdata-pipelinerr/blob/main/spark_stream.py) (Spark Streaming Job), [`prometheus.yml`](https://github.com/row-76/bigdata-pipelinerr/blob/main/prometheus.yml) (Scrape Configuration), [`producer.py`](https://github.com/row-76/bigdata-pipelinerr/blob/main/producer.py) (Event Producer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_apps["Streaming Applications"]
  node_producer["Event Producer<br/>Python entry point<br/>[producer.py]"]
  node_event_schema["Event Schema &amp; Cadence<br/>producer contract<br/>[producer.py]"]
  node_spark_job["Spark Streaming Job<br/>Spark entry point<br/>[spark_stream.py]"]
  node_transformations["Streaming Transformations<br/>Spark processing layer<br/>[spark_stream.py]"]
end

subgraph group_runtime["Local Runtime"]
  node_transport["Stream Transport<br/>broker or stream service<br/>[docker-compose.yml]"]
  node_output_sink[("Processed Output Sink<br/>configured sink<br/>[spark_stream.py]")]
  node_compose["Compose Topology<br/>deployment manifest<br/>[docker-compose.yml]"]
  node_metrics_endpoints["Service Metrics Endpoints<br/>metrics interface<br/>[docker-compose.yml]"]
end

subgraph group_observability["Observability"]
  node_prometheus["Prometheus<br/>monitoring runtime<br/>[prometheus.yml]"]
  node_scrape_config["Scrape Configuration<br/>Prometheus config<br/>[prometheus.yml]"]
end

node_producer -->|"creates"| node_event_schema
node_event_schema -->|"publishes events"| node_transport
node_transport -->|"stream input"| node_spark_job
node_spark_job -->|"executes"| node_transformations
node_transformations -->|"writes results"| node_output_sink
node_compose -.->|"runs"| node_producer
node_compose -.->|"provisions"| node_transport
node_compose -.->|"runs"| node_spark_job
node_compose -.->|"configures"| node_output_sink
node_compose -.->|"exposes"| node_metrics_endpoints
node_scrape_config -->|"configures"| node_prometheus
node_scrape_config -->|"targets"| node_metrics_endpoints
node_metrics_endpoints -->|"metrics"| node_prometheus

click node_producer "https://github.com/row-76/bigdata-pipelinerr/blob/main/producer.py"
click node_event_schema "https://github.com/row-76/bigdata-pipelinerr/blob/main/producer.py"
click node_transport "https://github.com/row-76/bigdata-pipelinerr/blob/main/docker-compose.yml"
click node_spark_job "https://github.com/row-76/bigdata-pipelinerr/blob/main/spark_stream.py"
click node_transformations "https://github.com/row-76/bigdata-pipelinerr/blob/main/spark_stream.py"
click node_output_sink "https://github.com/row-76/bigdata-pipelinerr/blob/main/spark_stream.py"
click node_compose "https://github.com/row-76/bigdata-pipelinerr/blob/main/docker-compose.yml"
click node_prometheus "https://github.com/row-76/bigdata-pipelinerr/blob/main/prometheus.yml"
click node_scrape_config "https://github.com/row-76/bigdata-pipelinerr/blob/main/prometheus.yml"
click node_metrics_endpoints "https://github.com/row-76/bigdata-pipelinerr/blob/main/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_producer,node_event_schema,node_spark_job,node_transformations toneBlue
class node_transport,node_output_sink,node_compose,node_metrics_endpoints toneAmber
class node_prometheus,node_scrape_config toneMint
```

## Components

### Streaming Applications

- [Event Producer](https://github.com/row-76/bigdata-pipelinerr/blob/main/producer.py): Python entry point · Generates and publishes input events. · `producer.py`
- [Event Schema & Cadence](https://github.com/row-76/bigdata-pipelinerr/blob/main/producer.py): producer contract · Defines emitted event shape and production timing. · `producer.py`
- [Spark Streaming Job](https://github.com/row-76/bigdata-pipelinerr/blob/main/spark_stream.py): Spark entry point · Consumes events and runs streaming transformations. · `spark_stream.py`
- [Streaming Transformations](https://github.com/row-76/bigdata-pipelinerr/blob/main/spark_stream.py): Spark processing layer · Applies transformations or aggregations to input events. · `spark_stream.py`

### Local Runtime

- [Stream Transport](https://github.com/row-76/bigdata-pipelinerr/blob/main/docker-compose.yml): broker or stream service · Configured event transport between producer and Spark. · `docker-compose.yml`
- [Processed Output Sink](https://github.com/row-76/bigdata-pipelinerr/blob/main/spark_stream.py): configured sink · Receives processed streaming results. · `spark_stream.py`
- [Compose Topology](https://github.com/row-76/bigdata-pipelinerr/blob/main/docker-compose.yml): deployment manifest · Coordinates local services, networking, and startup dependencies. · `docker-compose.yml`
- [Service Metrics Endpoints](https://github.com/row-76/bigdata-pipelinerr/blob/main/docker-compose.yml): metrics interface · Metrics exposed by application or Spark services. · `docker-compose.yml`

### Observability

- [Prometheus](https://github.com/row-76/bigdata-pipelinerr/blob/main/prometheus.yml): monitoring runtime · Scrapes service metrics. · `prometheus.yml`
- [Scrape Configuration](https://github.com/row-76/bigdata-pipelinerr/blob/main/prometheus.yml): Prometheus config · Defines metrics targets exposed by the local topology. · `prometheus.yml`

## Connections

- Event Producer → Event Schema & Cadence: creates
- Event Schema & Cadence → Stream Transport: publishes events
- Stream Transport → Spark Streaming Job: stream input
- Spark Streaming Job → Streaming Transformations: executes
- Streaming Transformations → Processed Output Sink: writes results
- Compose Topology → Event Producer: runs
- Compose Topology → Stream Transport: provisions
- Compose Topology → Spark Streaming Job: runs
- Compose Topology → Processed Output Sink: configures
- Compose Topology → Service Metrics Endpoints: exposes
- Scrape Configuration → Prometheus: configures
- Scrape Configuration → Service Metrics Endpoints: targets
- Service Metrics Endpoints → Prometheus: metrics