# youssefmo7amed/logistics-etl-migration architecture

> Architecture diagram of the GitHub repository youssefmo7amed/logistics-etl-migration, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/youssefmo7amed/logistics-etl-migration
- Repository: https://github.com/youssefmo7amed/logistics-etl-migration

## Overview

This is a MongoDB-to-MySQL logistics data migration application. Its principal workflow extracts MongoDB collections, transforms them into normalized relational records, and loads them into MySQL. The pipeline processes reference data, then pickup/dropoff addresses, then orders and related payment, confirmation, and tracker records. Airflow provides an optional scheduled entry point with failure-email notification. Some extraction, loading, and DAG details are documented but their implementation was not sampled, so those links reflect documented behavior rather than inspected calls.

Main entry points: [`etl/etl_pipeline.py`](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/etl_pipeline.py) (Pipeline Runner), [`connections/sql_connector.py`](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/connections/sql_connector.py) (MySQL Connector), [`utils/sql_data_access.py`](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/utils/sql_data_access.py) (SQL Data Access), [`dags/etl_pipeline_dag.py`](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/dags/etl_pipeline_dag.py) (Airflow DAG)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_main["Local Entrypoint<br/>[main.py]"]
  node_pipeline["Pipeline Runner<br/>[etl_pipeline.py]"]
  node_airflow["Airflow DAG"]
end

subgraph group_etl["ETL Processing"]
  node_extract["Mongo Extraction<br/>[extract.py]"]
  node_transform["Collection Transforms<br/>[transform.py]"]
  node_address["Address Transform<br/>[transform.py]"]
  node_orderdata["Order Related Transforms<br/>[transform.py]"]
  node_load["Relational Loading<br/>[load.py]"]
end

subgraph group_data["Relational Data"]
  node_sql_access["SQL Data Access<br/>[sql_data_access.py]"]
  node_schema["Relational Schema<br/>[sql_models.py]"]
end

subgraph group_integrations["Database Integrations"]
  node_mongo_connector["Mongo Connector<br/>[mongo_connector.py]"]
  node_sql_connector["MySQL Connector<br/>[sql_connector.py]"]
  node_failure_email["Failure Email<br/>[helpers.py]"]
end

node_operator(("Operator"))
node_scheduler(("Airflow Scheduler"))
node_mongodb[("MongoDB")]
node_mysql[("MySQL Database")]
node_email_service(("Email Service"))

node_operator -->|"runs locally"| node_main
node_main -->|"runs pipeline"| node_pipeline
node_scheduler -.->|"triggers schedule"| node_airflow
node_airflow -.->|"invokes stages"| node_pipeline
node_airflow -.->|"notifies on failure"| node_failure_email
node_failure_email -.->|"sends email"| node_email_service
node_pipeline -->|"requests collections"| node_extract
node_pipeline -->|"transforms reference data"| node_transform
node_pipeline -->|"transforms addresses"| node_address
node_pipeline -->|"transforms related data"| node_orderdata
node_pipeline -->|"loads records"| node_load
node_extract -->|"connects through"| node_mongo_connector
node_mongo_connector -->|"reads collections"| node_mongodb
node_transform -->|"looks up identifiers"| node_sql_access
node_address -->|"resolves geography"| node_sql_access
node_orderdata -->|"resolves relationships"| node_sql_access
node_sql_access -->|"opens sessions"| node_sql_connector
node_load -->|"writes through"| node_sql_connector
node_sql_connector -->|"creates and connects"| node_mysql
node_sql_connector -->|"creates tables from"| node_schema
node_load -->|"maps records to"| node_schema

click node_main "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/main.py"
click node_pipeline "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/etl_pipeline.py"
click node_airflow "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/dags/etl_pipeline_dag.py"
click node_extract "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/extract.py"
click node_transform "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/transform.py"
click node_address "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/transform.py"
click node_orderdata "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/transform.py"
click node_load "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/load.py"
click node_mongo_connector "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/connections/mongo_connector.py"
click node_sql_access "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/utils/sql_data_access.py"
click node_sql_connector "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/connections/sql_connector.py"
click node_schema "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/models/sql/sql_models.py"
click node_failure_email "https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/utils/helpers.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_pipeline,node_airflow toneBlue
class node_extract,node_transform,node_address,node_orderdata,node_load,node_mongodb,node_mysql toneAmber
class node_sql_access,node_schema toneMint
class node_mongo_connector,node_sql_connector,node_failure_email,node_scheduler toneRose
class node_operator,node_email_service toneIndigo
```

## Components

### Orchestration

- [Local Entrypoint](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/main.py): `main.py`
- [Pipeline Runner](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/etl_pipeline.py): `etl/etl_pipeline.py`
- [Airflow DAG](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/dags/etl_pipeline_dag.py): `dags/etl_pipeline_dag.py`

### ETL Processing

- [Mongo Extraction](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/extract.py): `etl/extract.py`
- [Collection Transforms](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/transform.py): `etl/transform.py`
- [Address Transform](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/transform.py): `etl/transform.py`
- [Order Related Transforms](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/transform.py): `etl/transform.py`
- [Relational Loading](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/etl/load.py): `etl/load.py`

### Relational Data

- [SQL Data Access](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/utils/sql_data_access.py): `utils/sql_data_access.py`
- [Relational Schema](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/models/sql/sql_models.py): `models/sql/sql_models.py`

### Database Integrations

- [Mongo Connector](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/connections/mongo_connector.py): `connections/mongo_connector.py`
- [MySQL Connector](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/connections/sql_connector.py): `connections/sql_connector.py`
- [Failure Email](https://github.com/youssefmo7amed/logistics-etl-migration/blob/main/utils/helpers.py): `utils/helpers.py`

### Other components

- **Operator**
- **Airflow Scheduler**
- **MongoDB**
- **MySQL Database**
- **Email Service**

## Connections

- Operator → Local Entrypoint: runs locally
- Local Entrypoint → Pipeline Runner: runs pipeline
- Airflow Scheduler → Airflow DAG: triggers schedule
- Airflow DAG → Pipeline Runner: invokes stages
- Airflow DAG → Failure Email: notifies on failure
- Failure Email → Email Service: sends email
- Pipeline Runner → Mongo Extraction: requests collections
- Pipeline Runner → Collection Transforms: transforms reference data
- Pipeline Runner → Address Transform: transforms addresses
- Pipeline Runner → Order Related Transforms: transforms related data
- Pipeline Runner → Relational Loading: loads records
- Mongo Extraction → Mongo Connector: connects through
- Mongo Connector → MongoDB: reads collections
- Collection Transforms → SQL Data Access: looks up identifiers
- Address Transform → SQL Data Access: resolves geography
- Order Related Transforms → SQL Data Access: resolves relationships
- SQL Data Access → MySQL Connector: opens sessions
- Relational Loading → MySQL Connector: writes through
- MySQL Connector → MySQL Database: creates and connects
- MySQL Connector → Relational Schema: creates tables from
- Relational Loading → Relational Schema: maps records to