# hexag158/data-pipeline-with-airflow architecture

> Architecture diagram of the GitHub repository hexag158/data-pipeline-with-airflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hexag158/data-pipeline-with-airflow
- Repository: https://github.com/hexag158/data-pipeline-with-airflow

## Overview

This project is a data pipeline built using Apache Airflow to orchestrate ETL (Extract, Transform, Load) processes. The system is designed to extract data from a dummy API, save the results to CSV files, load the data into a local Oracle Database, and send out email notifications when the pipeline completes or encounters errors. The use of Docker Compose indicates that the entire Airflow environment is containerized, making it easier to deploy and manage.

##### Main Components
a. Airflow Environment (Docker):
- Docker container environment initiated by docker-compose. This will include key Airflow services such as the Scheduler, Webserver, and Workers.
b. DAGs (Workflows):
- There are two main DAG files in the “dags” folder: one for saving CSV files and another for loading data into the Oracle Database and sending email notifications.
c. Data Storage:
- “data” folder as a repository for the CSV files generated by the ETL process.
d. Logging:
- “logs” folder for capturing runtime logs of the pipeline.
e. Configuration:
- The “config” folder contains the airflow.cfg file which dictates configuration options for Airflow.
f. Plugins:
- Custom extensions available in the “plugins” folder (e.g., custom operators, sensors, hooks).

##### Relationships and Interactions
a. The Docker Compose setup (docker-compose.yaml) creates and orchestrates the Airflow environment by launching the containerized environment.
b. Within the container, the Airflow Scheduler picks up the DAG definitions from the “dags” folder and triggers the tasks.
c. The tasks within each DAG execute in a defined order:
- Data extraction from a dummy API → Data saved as CSV in the “data” folder.
- The second DAG includes steps to load data into the Oracle Database and send out email notifications.
d. The “logs” folder collects process logs that aid in troubleshooting.
e. The “config” folder is used to manage the overall configuration of Airflow, which impacts scheduling, concurrency, and execution parameters.
f. The “plugins” folder allows users to extend Airflow’s functionality if special operators or integrations are needed.

##### Architectural Patterns and Design Principles
- The design follows a modular pattern where each aspect (data capture, transformation, load, alerting) is encapsulated in separate tasks within a DAG.
- Separation of concerns: Configuration is externalized, logging is separated, and custom behavior is added via plugins.
- Containerization with Docker ensures consistency across development and production environments.

##### Relevant Technologies and Tools
- Apache Airflow as the workflow orchestrator.
- Docker and Docker Compose for container management.
- Oracle Database as the destination for data post-transformation.
- Email services configured within Airflow for notifications.
- Python scripts (and potentially notebooks as suggested by file naming) for executing ETL tasks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Components
    DC("Docker Compose"):::docker
    click DC "https://github.com/hexag158/data-pipeline-with-airflow/blob/main/docker-compose.yaml"
    
    %% Airflow Container Subgraph
    subgraph "Airflow Container"
        A1("Airflow Scheduler"):::airflow
        A2("Airflow Webserver"):::airflow
        A3("Airflow Worker"):::airflow
    end
    DC -->|"deploys"| A1
    DC --> A2
    DC --> A3
    
    %% Supporting Configuration and Extensions
    C("Configuration (airflow.cfg)"):::config
    click C "https://github.com/hexag158/data-pipeline-with-airflow/tree/main/config"
    P("Plugins"):::config
    click P "https://github.com/hexag158/data-pipeline-with-airflow/tree/main/plugins"
    
    %% Link Airflow Container with Config and Plugins
    A1 ---|"uses"| C
    A2 --- C
    A3 --- C
    A1 ---|"extends via"| P
    A2 --- P
    A3 --- P
    
    %% DAGs Subgraph
    subgraph "DAGs"
        D1("CSV Extraction DAG"):::dag
        click D1 "https://github.com/hexag158/data-pipeline-with-airflow/blob/main/dags/etl_dag.py"
        D2("DB Load & Email DAG"):::dag
        click D2 "https://github.com/hexag158/data-pipeline-with-airflow/blob/main/dags/etl_sql_dag.py"
    end
    %% Airflow Scheduler picks up DAGs
    A1 -->|"schedules"| D1
    A1 --> D2
    
    %% Tasks for CSV Extraction DAG
    subgraph "CSV Extraction Tasks"
        T1("Extract Data from Dummy API"):::dag
        T2("Save Data as CSV"):::dag
    end
    D1 -->|"triggers"| T1
    T1 --> T2
    
    %% Data Storage for CSV Files
    DS("Data Storage (CSV Files)"):::storage
    click DS "https://github.com/hexag158/data-pipeline-with-airflow/tree/main/data"
    T2 -->|"writes to"| DS
    
    %% Logging for CSV Extraction
    L("Logs"):::storage
    click L "https://github.com/hexag158/data-pipeline-with-airflow/tree/main/logs"
    T1 -->|"logs to"| L
    T2 --> L
    
    %% Tasks for DB Loading & Email Notification DAG
    subgraph "DB & Email Tasks"
        T3("Load Data to Oracle DB"):::dag
        T4("Send Email Notification"):::dag
    end
    D2 -->|"triggers"| T3
    D2 --> T4
    
    %% External Integrations
    O("Oracle Database"):::external
    E("Email Service"):::external
    T3 -->|"loads into"| O
    T4 -->|"notifies via"| E
    
    %% Logging for DB & Email Tasks
    T3 --> L
    T4 --> L

    %% Styles
    classDef docker fill:#aqua,stroke:#000,stroke-width:2px;
    classDef airflow fill:#9fdf9f,stroke:#000,stroke-width:2px;
    classDef dag fill:#ffd700,stroke:#000,stroke-width:2px;
    classDef storage fill:#ffcc99,stroke:#000,stroke-width:2px;
    classDef external fill:#ff9999,stroke:#000,stroke-width:2px;
    classDef config fill:#ccccff,stroke:#000,stroke-width:2px;
```
