# shubhamdalvi1999/scheduled_data_migration_airflow architecture

> Architecture diagram of the GitHub repository shubhamdalvi1999/scheduled_data_migration_airflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shubhamdalvi1999/scheduled_data_migration_airflow
- Repository: https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow

## Overview

##### Project type and purpose
- This is a containerized data‐engineering pipeline built on Apache Airflow.
- Its purpose is to orchestrate a daily ETL workflow:
- Ingest raw CSVs from a volume
- Clean and transform the data via Python
- Create/query/load data into a MySQL database
- Post‐process output files (move/rename)
- Send notification emails with the results

##### File structure
- Top level
- Dockerfile & docker-compose-LocalExecutor.yml ⇒ containerized Airflow + MySQL
- config/airflow.cfg & mysql.cnf ⇒ service configuration
- dags/ ⇒ Python DAG definitions (datacleaner.py, store_DAG.py)
- script/entrypoint.sh ⇒ container entrypoint logic
- sql_files/ ⇒ DDL/DML scripts and helper shell scripts
- store_files/ ⇒ mounted volume for raw and output CSVs
- Shows an Airflow LocalExecutor deployment: one container runs scheduler, webserver, worker; a second runs MySQL

##### Readme insights
- The DAG “store_DAG” defines nine tasks in sequence:
1) check for raw CSV
2) clean CSV (PythonOperator)
3) create MySQL table
4) insert cleaned data
5) select/query data
6–7) move/rename output CSVs
8) send email with attachments
9) run shell script to rename raw files
- Dependencies are strict linear (t1 → t2 → … → t9)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Docker Compose Deployment
    subgraph "Docker Compose Deployment"
        direction TB
        subgraph "Airflow Container"
            direction TB
            AF_Config["Airflow Config"]:::config
            AF_Scheduler["Scheduler"]:::orchestrator
            AF_Webserver["Webserver"]:::orchestrator
            AF_Worker["Worker"]:::orchestrator

            AF_Config --> AF_Scheduler
            AF_Config --> AF_Webserver
            AF_Config --> AF_Worker

            entrypoint["Entrypoint & Shell Env"]:::shell
            AF_Worker --> entrypoint

            subgraph "ETL DAG: store_DAG" 
                direction TB
                t1_Check["1) Check Raw CSV"]:::task
                t2_Clean["2) Clean CSV (Python)"]:::task
                t3_Create["3) Create Table"]:::task
                t4_Insert["4) Insert Data"]:::task
                t5_Select["5) Select Query"]:::task
                t6_Move["6) Move Output CSV"]:::task
                t7_Rename["7) Rename Output CSV"]:::task
                t8_Email["8) Send Email"]:::task
                t9_RawRename["9) Rename Raw Files"]:::task

                t1_Check --> t2_Clean --> t3_Create --> t4_Insert --> t5_Select --> t6_Move --> t7_Rename --> t8_Email --> t9_RawRename
            end

            entrypoint --> t9_RawRename
        end

        subgraph "MySQL Container"
            direction TB
            MySQL["MySQL Service"]:::database
        end

        AF_Worker -.->|runs SQL tasks| MySQL
    end

    %% Storage and External
    StoreFiles["store_files Volume"]:::storage
    SMTP["SMTP / Email Service"]:::external

    StoreFiles --> t1_Check
    t6_Move --> StoreFiles
    t7_Rename --> StoreFiles
    t8_Email --> SMTP
    t5_Select --> StoreFiles

    %% MySQL data flow
    t3_Create -->|DDL| MySQL
    t4_Insert -->|DML| MySQL
    MySQL -->|Query Results| t5_Select

    %% Click Events
    click AF_Config "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/config/airflow.cfg"
    click AF_Scheduler "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/tree/main/Dockerfile"
    click AF_Scheduler "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/docker-compose-LocalExecutor.yml"
    click AF_Webserver "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/tree/main/Dockerfile"
    click AF_Webserver "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/docker-compose-LocalExecutor.yml"
    click AF_Worker "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/tree/main/Dockerfile"
    click AF_Worker "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/docker-compose-LocalExecutor.yml"
    click t2_Clean "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/dags/datacleaner.py"
    click t8_Email "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/dags/store_DAG.py"
    click StoreFiles "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/tree/main/store_files/"
    click MySQL "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/mysql.cnf"
    click MySQL "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/docker-compose-LocalExecutor.yml"
    click t3_Create "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/sql_files/create_table.sql"
    click t4_Insert "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/sql_files/insert_into_table.sql"
    click t5_Select "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/sql_files/select_from_table.sql"
    click entrypoint "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/script/entrypoint.sh"
    click entrypoint "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/sql_files/copy_shell_script.sh"
    click AF_Config "https://github.com/shubhamdalvi1999/scheduled_data_migration_airflow/blob/main/fethernet_key__file.py"

    %% Styles
    classDef orchestrator fill:#FFA500,stroke:#333,stroke-width:1px
    classDef task fill:#87CEEB,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef storage fill:#FFD700,stroke:#333,stroke-width:1px,shape: cylinder
    classDef database fill:#F0E68C,stroke:#333,stroke-width:1px,shape: cylinder
    classDef external fill:#98FB98,stroke:#333,stroke-width:1px,shape: hexagon
    classDef shell fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef config fill:#FFE4B5,stroke:#333,stroke-width:1px
```
