How row-76/bigdata-pipelinerr works
Purpose and Entry Points
This is a compact streaming-data repository centered on event production, Spark-based stream processing, and Prometheus monitoring. Primary executable entry points are producer.py for generating/publishing input events and spark_stream.py for consuming and processing the stream.
Event Production
producer.py is the ingestion boundary. It creates the source-side event flow and is the likely place to define message schema, production cadence, and connection details for the downstream broker or stream transport.
Stream Processing
spark_stream.py is the processing boundary and likely hosts the Spark Structured Streaming job. It consumes produced events, applies transformations or aggregations, and writes results to its configured sink. Spark is the material runtime for stateful or distributed stream computation.
Runtime Topology
docker-compose.yml defines the local multi-service deployment boundary. It is the central integration point for the producer, Spark workload, and any required supporting services such as a message broker, storage, or metrics endpoints. Service names, network configuration, and startup dependencies should be treated as the authoritative runtime contract.
Observability
prometheus.yml defines Prometheus scraping configuration. Prometheus is the monitoring runtime and should collect metrics from services exposed through the Compose topology, potentially including Spark or application-level metrics endpoints.
Primary Control and Data Flow
1. producer.py emits streaming events to the configured transport.
2. spark_stream.py reads that event stream and executes streaming transformations.
3. The Spark job writes processed output to its configured destination.
4. Prometheus scrapes metrics according to prometheus.yml.
5. docker-compose.yml coordinates local service startup and connectivity across these stages.
Important Boundaries
The key contract boundaries are producer-to-stream transport, transport-to-Spark consumption, and Spark-to-output sink. Changes to event shape or broker/topic configuration must remain compatible between producer.py and spark_stream.py; infrastructure endpoint and port changes must remain aligned with docker-compose.yml and prometheus.yml.
Main entry points: docker-compose.yml Compose Topology · spark_stream.py Spark Streaming Job · prometheus.yml Scrape Configuration · producer.py Event Producer
Components 10
Streaming Applications
- Event Producer · Generates and publishes input events.
producer.py - Event Schema & Cadence · Defines emitted event shape and production timing.
producer.py - Spark Streaming Job · Consumes events and runs streaming transformations.
spark_stream.py - Streaming Transformations · Applies transformations or aggregations to input events.
spark_stream.py
Local Runtime
- Stream Transport · Configured event transport between producer and Spark.
docker-compose.yml - Processed Output Sink · Receives processed streaming results.
spark_stream.py - Compose Topology · Coordinates local services, networking, and startup dependencies.
docker-compose.yml - Service Metrics Endpoints · Metrics exposed by application or Spark services.
docker-compose.yml
Observability
- Prometheus · Scrapes service metrics.
prometheus.yml - Scrape Configuration · Defines metrics targets exposed by the local topology.
prometheus.yml
Connections 13
- Event Producer → Event Schema & Cadence: creates
- Event Schema & Cadence → Stream Transport: publishes events
- Stream Transport → Spark Streaming Job: stream input
- Spark Streaming Job → Streaming Transformations: executes
- Streaming Transformations → Processed Output Sink: writes results
- Compose Topology → Event Producer: runs
- Compose Topology → Stream Transport: provisions
- Compose Topology → Spark Streaming Job: runs
- Compose Topology → Processed Output Sink: configures
- Compose Topology → Service Metrics Endpoints: exposes
- Scrape Configuration → Prometheus: configures
- Scrape Configuration → Service Metrics Endpoints: targets
- Service Metrics Endpoints → Prometheus: metrics