How shubhamdalvi1999/scheduled_data_migration_airflow works
Project type and purpose
• This is a containerized data‐engineering pipeline built on Apache Airflow.
• Its purpose is to orchestrate a daily ETL workflow:
• Ingest raw CSVs from a volume
• Clean and transform the data via Python
• Create/query/load data into a MySQL database
• Post‐process output files (move/rename)
• Send notification emails with the results
File structure
• Top level
• Dockerfile & docker-compose-LocalExecutor.yml ⇒ containerized Airflow + MySQL
• config/airflow.cfg & mysql.cnf ⇒ service configuration
• dags/ ⇒ Python DAG definitions (datacleaner.py, store_DAG.py)
• script/entrypoint.sh ⇒ container entrypoint logic
• sql_files/ ⇒ DDL/DML scripts and helper shell scripts
• store_files/ ⇒ mounted volume for raw and output CSVs
• Shows an Airflow LocalExecutor deployment: one container runs scheduler, webserver, worker; a second runs MySQL
Readme insights
• The DAG “store_DAG” defines nine tasks in sequence:
1) check for raw CSV
2) clean CSV (PythonOperator)
3) create MySQL table
4) insert cleaned data
5) select/query data
6–7) move/rename output CSVs
8) send email with attachments
9) run shell script to rename raw files
• Dependencies are strict linear (t1 → t2 → … → t9)