murari123-byte/dataflow-processing-engine

How murari123-byte/dataflow-processing-engine works

DataFlow Processing Engine is a PySpark data pipeline. Its main workflow reads customer and order CSVs, validates and separates accepted/rejected records, enriches orders and aggregates daily sales, builds fact/dimension datasets, reconciles counts, and publishes Parquet outputs while recording run status. Separate capabilities include reproducible data generation, analytics over published data, and aggregation/join benchmarks. Edges follow the sampled pipeline calls and explicit README descriptions; analytics-to-output access is documented, but its implementation details were not sampled.

Main entry points: run_pipeline.py Pipeline orchestrator · publisher.py Staged Parquet publisher · validation.py Validation and quarantine · run_benchmark.py Aggregation benchmark

Components 23

Data inputs

Customer CSV
Order CSV
Dataset generator src/generator/generate_data.py

Processing pipeline

Runtime configuration src/config.py
Pipeline orchestrator src/run_pipeline.py
Validation and quarantine src/processing/validation.py
Enrichment and aggregation src/processing/optimized.py
Fact and dimensions src/processing/dimensional_model.py
Reconciliation checks src/processing/reconciliation.py

Published data

Staged Parquet publisher src/data/publisher.py
Valid orders
Daily sales
Rejected orders
Fact and dimensions
Reconciliation report
Pipeline run log

Operations and analysis

Run monitoring src/monitoring.py
SQL analytics src/processing/analytics.py
Analytics entry point src/run_analytics.py

Performance benchmarks

Baseline transform src/processing/baseline.py
Aggregation benchmark src/run_benchmark.py
Join benchmark src/run_join_benchmark.py

Other components

Pipeline operator

Connections 29

Pipeline operator → Pipeline orchestrator: starts README.md
Dataset generator → Customer CSV: generates README.md
Dataset generator → Order CSV: generates README.md
Pipeline orchestrator → Runtime configuration: loads settings src/run_pipeline.py
Pipeline orchestrator → Order CSV: reads src/run_pipeline.py
Pipeline orchestrator → Customer CSV: reads src/run_pipeline.py
Pipeline orchestrator → Validation and quarantine: validates src/run_pipeline.py
Validation and quarantine → Valid orders: returns accepted src/processing/validation.py
Validation and quarantine → Rejected orders: returns rejected src/processing/validation.py
Pipeline orchestrator → Enrichment and aggregation: transforms valid orders src/run_pipeline.py
Enrichment and aggregation → Daily sales: aggregates src/processing/optimized.py
Pipeline orchestrator → Fact and dimensions: builds model src/run_pipeline.py
Fact and dimensions → Fact and dimensions: returns fact and dimensions src/processing/dimensional_model.py
Pipeline orchestrator → Reconciliation checks: reconciles src/run_pipeline.py
Reconciliation checks → Daily sales: checks totals src/processing/reconciliation.py
Pipeline orchestrator → Reconciliation report: writes report src/run_pipeline.py
Pipeline orchestrator → Staged Parquet publisher: publishes datasets src/run_pipeline.py
Staged Parquet publisher → Valid orders: writes Parquet src/data/publisher.py
Staged Parquet publisher → Daily sales: writes Parquet src/data/publisher.py
Staged Parquet publisher → Rejected orders: writes Parquet src/data/publisher.py
Staged Parquet publisher → Fact and dimensions: writes Parquet src/data/publisher.py
Pipeline orchestrator → Run monitoring: records run src/run_pipeline.py
Run monitoring → Pipeline run log: appends record src/monitoring.py
Analytics entry point → SQL analytics: runs queries src/run_analytics.py
SQL analytics → Valid orders: queries published data README.md
Aggregation benchmark → Validation and quarantine: validates inputs src/run_benchmark.py
Aggregation benchmark → Baseline transform: compares transforms src/run_benchmark.py
Aggregation benchmark → Enrichment and aggregation: compares transforms src/run_benchmark.py
Pipeline operator → Join benchmark: runs benchmark README.md

Drawn by GitDiagram on November 14, 2026 from the repository's file tree, README and source files. Read as Markdown ·