# anaasih/moroccan-fmcg-data-pipeline architecture

> Architecture diagram of the GitHub repository anaasih/moroccan-fmcg-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anaasih/moroccan-fmcg-data-pipeline
- Repository: https://github.com/anaasih/moroccan-fmcg-data-pipeline

## Overview

This is a Moroccan FMCG retail ETL and data-warehouse pipeline. Its main workflow is scheduled by Airflow: ingest sales into raw storage with batch/file tracking, seed store and product dimensions, validate and transform sales into accepted facts or quarantined rejects, then expose warehouse data through seven BI marts. The graph preserves the documented warehouse and reporting capabilities. Source excerpts establish the Python calls and database operations; mart-to-dashboard relationships come from the README, since the SQL and reporting implementations were not sampled.

Main entry points: [`src/retail_etl/transform/transform_sales.py`](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/transform/transform_sales.py) (Sales Quality Transform), [`src/retail_etl/pipeline.py`](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/pipeline.py) (ETL Pipeline), [`src/retail_etl/extract/raw_ingestion.py`](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/extract/raw_ingestion.py) (Sales Ingestion), [`src/retail_etl/transform/seed_dimensions.py`](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/transform/seed_dimensions.py) (Dimension Seeding)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["Source Data"]
  node_raw_files["Raw Source Files"]
end

subgraph group_orchestration["Orchestration"]
  node_airflow_dag["Airflow DAG"]
end

subgraph group_etl["ETL Processing"]
  node_pipeline["ETL Pipeline<br/>[pipeline.py]"]
  node_ingestion["Sales Ingestion<br/>[raw_ingestion.py]"]
  node_dimension_seed["Dimension Seeding<br/>[seed_dimensions.py]"]
  node_sales_transform["Sales Quality Transform<br/>[transform_sales.py]"]
  node_profiling["Source Profiling<br/>[profiling.py]"]
end

subgraph group_warehouse["Data Warehouse"]
  node_db_connection["Database Connection<br/>[connection.py]"]
  node_raw_sales[("Raw Sales")]
  node_ops_tracking[("Batch and File Tracking")]
  node_store_dim[("Store Dimension")]
  node_product_dim[("Product Dimension")]
  node_fact_sales[("Sales Fact")]
  node_rejects[("Rejected Records")]
end

subgraph group_reporting["Analytics"]
  node_bi_marts[("Seven BI Marts")]
  node_power_bi["Power BI Dashboard"]
end

node_scheduler(("Daily Scheduler"))

node_scheduler -->|"schedules daily"| node_airflow_dag
node_airflow_dag -->|"calls pipeline"| node_pipeline
node_pipeline -->|"calls ingestion"| node_ingestion
node_pipeline -->|"seeds dimensions"| node_dimension_seed
node_pipeline -->|"runs transform"| node_sales_transform
node_pipeline -->|"queries summary"| node_db_connection
node_raw_files -->|"provides sales CSV"| node_ingestion
node_ingestion -->|"gets engine"| node_db_connection
node_ingestion -->|"loads raw records"| node_raw_sales
node_ingestion -->|"records batch manifest"| node_ops_tracking
node_raw_files -->|"provides stores products"| node_dimension_seed
node_dimension_seed -->|"gets engine"| node_db_connection
node_dimension_seed -->|"seeds stores"| node_store_dim
node_dimension_seed -->|"seeds products"| node_product_dim
node_sales_transform -->|"gets engine"| node_db_connection
node_sales_transform -->|"reads batch rows"| node_raw_sales
node_sales_transform -->|"checks store IDs"| node_store_dim
node_sales_transform -->|"checks product IDs"| node_product_dim
node_sales_transform -->|"writes accepted rows"| node_fact_sales
node_sales_transform -->|"quarantines invalid rows"| node_rejects
node_fact_sales -->|"feeds sales views"| node_bi_marts
node_rejects -->|"feeds quality views"| node_bi_marts
node_bi_marts -->|"powers reporting"| node_power_bi
node_raw_files -->|"profiles source files"| node_profiling

click node_raw_files "https://github.com/anaasih/moroccan-fmcg-data-pipeline/tree/main/data/raw"
click node_airflow_dag "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/dags/moroccan_retail_dag.py"
click node_pipeline "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/pipeline.py"
click node_ingestion "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/extract/raw_ingestion.py"
click node_dimension_seed "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/transform/seed_dimensions.py"
click node_sales_transform "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/transform/transform_sales.py"
click node_db_connection "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/database/connection.py"
click node_raw_sales "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql"
click node_ops_tracking "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql"
click node_store_dim "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql"
click node_product_dim "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql"
click node_fact_sales "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql"
click node_rejects "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql"
click node_bi_marts "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/marts/02_create_bi_views.sql"
click node_profiling "https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/profiling.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_files toneBlue
class node_airflow_dag toneAmber
class node_pipeline,node_ingestion,node_dimension_seed,node_sales_transform,node_profiling toneMint
class node_db_connection,node_raw_sales,node_ops_tracking,node_store_dim,node_product_dim,node_fact_sales,node_rejects,node_scheduler toneRose
class node_bi_marts,node_power_bi toneIndigo
```

## Components

### Source Data

- [Raw Source Files](https://github.com/anaasih/moroccan-fmcg-data-pipeline/tree/main/data/raw): `data/raw`

### Orchestration

- [Airflow DAG](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/dags/moroccan_retail_dag.py): `dags/moroccan_retail_dag.py`

### ETL Processing

- [ETL Pipeline](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/pipeline.py): `src/retail_etl/pipeline.py`
- [Sales Ingestion](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/extract/raw_ingestion.py): `src/retail_etl/extract/raw_ingestion.py`
- [Dimension Seeding](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/transform/seed_dimensions.py): `src/retail_etl/transform/seed_dimensions.py`
- [Sales Quality Transform](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/transform/transform_sales.py): `src/retail_etl/transform/transform_sales.py`
- [Source Profiling](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/profiling.py): `src/retail_etl/profiling.py`

### Data Warehouse

- [Database Connection](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/src/retail_etl/database/connection.py): `src/retail_etl/database/connection.py`
- [Raw Sales](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql): `sql/ddl/01_create_schemas_and_tables.sql`
- [Batch and File Tracking](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql): `sql/ddl/01_create_schemas_and_tables.sql`
- [Store Dimension](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql): `sql/ddl/01_create_schemas_and_tables.sql`
- [Product Dimension](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql): `sql/ddl/01_create_schemas_and_tables.sql`
- [Sales Fact](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql): `sql/ddl/01_create_schemas_and_tables.sql`
- [Rejected Records](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/ddl/01_create_schemas_and_tables.sql): `sql/ddl/01_create_schemas_and_tables.sql`

### Analytics

- [Seven BI Marts](https://github.com/anaasih/moroccan-fmcg-data-pipeline/blob/main/sql/marts/02_create_bi_views.sql): `sql/marts/02_create_bi_views.sql`
- **Power BI Dashboard**

### Other components

- **Daily Scheduler**

## Connections

- Daily Scheduler → Airflow DAG: schedules daily (evidence: `README.md`)
- Airflow DAG → ETL Pipeline: calls pipeline (evidence: `dags/moroccan_retail_dag.py`)
- ETL Pipeline → Sales Ingestion: calls ingestion (evidence: `src/retail_etl/pipeline.py`)
- ETL Pipeline → Dimension Seeding: seeds dimensions (evidence: `src/retail_etl/pipeline.py`)
- ETL Pipeline → Sales Quality Transform: runs transform (evidence: `src/retail_etl/pipeline.py`)
- ETL Pipeline → Database Connection: queries summary (evidence: `src/retail_etl/pipeline.py`)
- Raw Source Files → Sales Ingestion: provides sales CSV (evidence: `src/retail_etl/extract/raw_ingestion.py`)
- Sales Ingestion → Database Connection: gets engine (evidence: `src/retail_etl/extract/raw_ingestion.py`)
- Sales Ingestion → Raw Sales: loads raw records (evidence: `src/retail_etl/extract/raw_ingestion.py`)
- Sales Ingestion → Batch and File Tracking: records batch manifest (evidence: `src/retail_etl/extract/raw_ingestion.py`)
- Raw Source Files → Dimension Seeding: provides stores products (evidence: `src/retail_etl/transform/seed_dimensions.py`)
- Dimension Seeding → Database Connection: gets engine (evidence: `src/retail_etl/transform/seed_dimensions.py`)
- Dimension Seeding → Store Dimension: seeds stores (evidence: `src/retail_etl/transform/seed_dimensions.py`)
- Dimension Seeding → Product Dimension: seeds products (evidence: `src/retail_etl/transform/seed_dimensions.py`)
- Sales Quality Transform → Database Connection: gets engine (evidence: `src/retail_etl/transform/transform_sales.py`)
- Sales Quality Transform → Raw Sales: reads batch rows (evidence: `src/retail_etl/transform/transform_sales.py`)
- Sales Quality Transform → Store Dimension: checks store IDs (evidence: `src/retail_etl/transform/transform_sales.py`)
- Sales Quality Transform → Product Dimension: checks product IDs (evidence: `src/retail_etl/transform/transform_sales.py`)
- Sales Quality Transform → Sales Fact: writes accepted rows (evidence: `src/retail_etl/transform/transform_sales.py`)
- Sales Quality Transform → Rejected Records: quarantines invalid rows (evidence: `src/retail_etl/transform/transform_sales.py`)
- Sales Fact → Seven BI Marts: feeds sales views (evidence: `README.md`)
- Rejected Records → Seven BI Marts: feeds quality views (evidence: `README.md`)
- Seven BI Marts → Power BI Dashboard: powers reporting (evidence: `README.md`)
- Raw Source Files → Source Profiling: profiles source files (evidence: `src/retail_etl/profiling.py`)