# mouad-prog/pipeline-e-commerce architecture

> Architecture diagram of the GitHub repository mouad-prog/pipeline-e-commerce, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mouad-prog/pipeline-e-commerce
- Repository: https://github.com/mouad-prog/pipeline-e-commerce

## Overview

This Scala application generates mock e-commerce clients, products, and transactions, publishes them to Kafka, then reads and processes those records in Spark. The main workflow cleans and deduplicates the data, computes revenue and customer KPIs, and writes both cleaned datasets and KPIs to PostgreSQL and Parquet. The orchestrator runs ingestion and processing, then saves a run summary. The README also describes health checks, but the sampled orchestrator source does not show their implementation, so they are not mapped here.

Main entry points: [`src/main/scala/processing/SparkStreamingConsumer.scala`](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/processing/SparkStreamingConsumer.scala) (Spark streaming consumer), [`src/main/scala/orchestration/PipelineOrchestrator.scala`](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/orchestration/PipelineOrchestrator.scala) (Pipeline orchestrator), [`src/main/scala/ingestion/KafkaProducer.scala`](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/ingestion/KafkaProducer.scala) (Kafka producer), [`src/main/scala/storage/ParquetWriter.scala`](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/storage/ParquetWriter.scala) (Parquet writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Run orchestration"]
  node_orchestrator["Pipeline orchestrator"]
  node_summary["Run summary report"]
end

subgraph group_ingestion["Data ingestion"]
  node_generator["Mock data generator"]
  node_producer["Kafka producer"]
  node_kafka["Kafka topics"]
end

subgraph group_processing["Spark processing"]
  node_consumer["Spark streaming consumer"]
  node_cleaner["Data cleaner<br/>[DataCleaner.scala]"]
  node_transformer["KPI transformer"]
end

subgraph group_storage["Data storage"]
  node_parquetwriter["Parquet writer"]
  node_postgreswriter["Postgres writer"]
  node_parquet[("Parquet data lake")]
  node_postgres[("PostgreSQL database")]
end

node_operator(("Pipeline operator"))

node_operator -->|"starts"| node_orchestrator
node_orchestrator -->|"runs ingestion"| node_producer
node_orchestrator -->|"runs processing"| node_consumer
node_orchestrator -->|"writes report"| node_summary
node_generator -->|"generates records"| node_producer
node_producer -->|"publishes records"| node_kafka
node_consumer -->|"reads topics"| node_kafka
node_consumer -->|"cleans datasets"| node_cleaner
node_consumer -->|"computes KPIs"| node_transformer
node_consumer -->|"writes datasets and KPIs"| node_parquetwriter
node_consumer -->|"writes datasets and KPIs"| node_postgreswriter
node_parquetwriter -->|"writes Parquet"| node_parquet
node_postgreswriter -->|"writes tables"| node_postgres

click node_orchestrator "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/orchestration/PipelineOrchestrator.scala"
click node_summary "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/orchestration/PipelineOrchestrator.scala"
click node_generator "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/ingestion/MockDataGenerator.scala"
click node_producer "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/ingestion/KafkaProducer.scala"
click node_consumer "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/processing/SparkStreamingConsumer.scala"
click node_cleaner "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/processing/DataCleaner.scala"
click node_transformer "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/processing/DataTransformer.scala"
click node_parquetwriter "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/storage/ParquetWriter.scala"
click node_postgreswriter "https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/storage/PostgresWriter.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_summary toneBlue
class node_generator,node_producer,node_kafka toneAmber
class node_consumer,node_cleaner,node_transformer toneMint
class node_parquetwriter,node_postgreswriter,node_parquet,node_postgres toneRose
class node_operator toneIndigo
```

## Components

### Run orchestration

- [Pipeline orchestrator](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/orchestration/PipelineOrchestrator.scala): `src/main/scala/orchestration/PipelineOrchestrator.scala`
- [Run summary report](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/orchestration/PipelineOrchestrator.scala): `src/main/scala/orchestration/PipelineOrchestrator.scala`

### Data ingestion

- [Mock data generator](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/ingestion/MockDataGenerator.scala): `src/main/scala/ingestion/MockDataGenerator.scala`
- [Kafka producer](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/ingestion/KafkaProducer.scala): `src/main/scala/ingestion/KafkaProducer.scala`
- **Kafka topics**

### Spark processing

- [Spark streaming consumer](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/processing/SparkStreamingConsumer.scala): `src/main/scala/processing/SparkStreamingConsumer.scala`
- [Data cleaner](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/processing/DataCleaner.scala): `src/main/scala/processing/DataCleaner.scala`
- [KPI transformer](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/processing/DataTransformer.scala): `src/main/scala/processing/DataTransformer.scala`

### Data storage

- [Parquet writer](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/storage/ParquetWriter.scala): `src/main/scala/storage/ParquetWriter.scala`
- [Postgres writer](https://github.com/mouad-prog/pipeline-e-commerce/blob/main/src/main/scala/storage/PostgresWriter.scala): `src/main/scala/storage/PostgresWriter.scala`
- **Parquet data lake**
- **PostgreSQL database**

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Pipeline orchestrator: starts
- Pipeline orchestrator → Kafka producer: runs ingestion
- Pipeline orchestrator → Spark streaming consumer: runs processing
- Pipeline orchestrator → Run summary report: writes report
- Mock data generator → Kafka producer: generates records
- Kafka producer → Kafka topics: publishes records
- Spark streaming consumer → Kafka topics: reads topics
- Spark streaming consumer → Data cleaner: cleans datasets
- Spark streaming consumer → KPI transformer: computes KPIs
- Spark streaming consumer → Parquet writer: writes datasets and KPIs
- Spark streaming consumer → Postgres writer: writes datasets and KPIs
- Parquet writer → Parquet data lake: writes Parquet
- Postgres writer → PostgreSQL database: writes tables