How youssefmo7amed/logistics-etl-migration works
This is a MongoDB-to-MySQL logistics data migration application. Its principal workflow extracts MongoDB collections, transforms them into normalized relational records, and loads them into MySQL. The pipeline processes reference data, then pickup/dropoff addresses, then orders and related payment, confirmation, and tracker records. Airflow provides an optional scheduled entry point with failure-email notification. Some extraction, loading, and DAG details are documented but their implementation was not sampled, so those links reflect documented behavior rather than inspected calls.
Main entry points: etl_pipeline.py Pipeline Runner · sql_connector.py MySQL Connector · sql_data_access.py SQL Data Access · etl_pipeline_dag.py Airflow DAG
Components 18
Orchestration
- Local Entrypoint
main.py - Pipeline Runner
etl/etl_pipeline.py - Airflow DAG
dags/etl_pipeline_dag.py
ETL Processing
- Mongo Extraction
etl/extract.py - Collection Transforms
etl/transform.py - Address Transform
etl/transform.py - Order Related Transforms
etl/transform.py - Relational Loading
etl/load.py
Relational Data
- SQL Data Access
utils/sql_data_access.py - Relational Schema
models/sql/sql_models.py
Database Integrations
- Mongo Connector
connections/mongo_connector.py - MySQL Connector
connections/sql_connector.py - Failure Email
utils/helpers.py
Other components
- Operator
- Airflow Scheduler
- MongoDB
- MySQL Database
- Email Service
Connections 21
- Operator → Local Entrypoint: runs locally
- Local Entrypoint → Pipeline Runner: runs pipeline
- Airflow Scheduler → Airflow DAG: triggers schedule
- Airflow DAG → Pipeline Runner: invokes stages
- Airflow DAG → Failure Email: notifies on failure
- Failure Email → Email Service: sends email
- Pipeline Runner → Mongo Extraction: requests collections
- Pipeline Runner → Collection Transforms: transforms reference data
- Pipeline Runner → Address Transform: transforms addresses
- Pipeline Runner → Order Related Transforms: transforms related data
- Pipeline Runner → Relational Loading: loads records
- Mongo Extraction → Mongo Connector: connects through
- Mongo Connector → MongoDB: reads collections
- Collection Transforms → SQL Data Access: looks up identifiers
- Address Transform → SQL Data Access: resolves geography
- Order Related Transforms → SQL Data Access: resolves relationships
- SQL Data Access → MySQL Connector: opens sessions
- Relational Loading → MySQL Connector: writes through
- MySQL Connector → MySQL Database: creates and connects
- MySQL Connector → Relational Schema: creates tables from
- Relational Loading → Relational Schema: maps records to