How mouad-prog/pipeline-e-commerce works
This Scala application generates mock e-commerce clients, products, and transactions, publishes them to Kafka, then reads and processes those records in Spark. The main workflow cleans and deduplicates the data, computes revenue and customer KPIs, and writes both cleaned datasets and KPIs to PostgreSQL and Parquet. The orchestrator runs ingestion and processing, then saves a run summary. The README also describes health checks, but the sampled orchestrator source does not show their implementation, so they are not mapped here.
Main entry points: SparkStreamingConsumer.scala Spark streaming consumer · PipelineOrchestrator.scala Pipeline orchestrator · KafkaProducer.scala Kafka producer · ParquetWriter.scala Parquet writer
Components 13
Run orchestration
- Pipeline orchestrator
src/main/scala/orchestration/PipelineOrchestrator.scala - Run summary report
src/main/scala/orchestration/PipelineOrchestrator.scala
Data ingestion
- Mock data generator
src/main/scala/ingestion/MockDataGenerator.scala - Kafka producer
src/main/scala/ingestion/KafkaProducer.scala - Kafka topics
Spark processing
- Spark streaming consumer
src/main/scala/processing/SparkStreamingConsumer.scala - Data cleaner
src/main/scala/processing/DataCleaner.scala - KPI transformer
src/main/scala/processing/DataTransformer.scala
Data storage
- Parquet writer
src/main/scala/storage/ParquetWriter.scala - Postgres writer
src/main/scala/storage/PostgresWriter.scala - Parquet data lake
- PostgreSQL database
Other components
- Pipeline operator
Connections 13
- Pipeline operator → Pipeline orchestrator: starts
- Pipeline orchestrator → Kafka producer: runs ingestion
- Pipeline orchestrator → Spark streaming consumer: runs processing
- Pipeline orchestrator → Run summary report: writes report
- Mock data generator → Kafka producer: generates records
- Kafka producer → Kafka topics: publishes records
- Spark streaming consumer → Kafka topics: reads topics
- Spark streaming consumer → Data cleaner: cleans datasets
- Spark streaming consumer → KPI transformer: computes KPIs
- Spark streaming consumer → Parquet writer: writes datasets and KPIs
- Spark streaming consumer → Postgres writer: writes datasets and KPIs
- Parquet writer → Parquet data lake: writes Parquet
- Postgres writer → PostgreSQL database: writes tables