# ayalaadaili/etl_airflow architecture

> Architecture diagram of the GitHub repository ayalaadaili/etl_airflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayalaadaili/etl_airflow
- Repository: https://github.com/ayalaadaili/etl_airflow

## Overview

##### Project Type and Purpose
- This is a data-engineering pipeline and dashboarding “full-stack” project. It uses Apache Airflow (Python) to orchestrate an ETL pipeline that reads raw data, transforms it, and loads it into a SQLite store, while using Dash and Streamlit for interactive visualization. All components are containerized via Docker Compose, with PostgreSQL as Airflow’s metadata database.

##### File Structure Highlights
- Top-level:
- Dockerfile, Dockerfile.streamlit – images for services
- docker-compose.yaml – orchestrates Airflow, Postgres, SQLite volume, Dash, Streamlit
- app.py, dashboard.py – entrypoints for Dash and Streamlit apps
- dags/etl.py – the pipeline definition (DAG)
- Data/data.rar – raw data source
- logs/ – Airflow task logs
- requirements.txt – Python dependencies
- Patterns:
- “dags” folder—Airflow’s DAG definitions (Extraction, Transformation, Load)
- Separate scripts for Dash (dashboard.py) and Streamlit (app.py)
- Docker Compose network linking all services

##### Key Architecture Insights from README
- Components:
- Apache Airflow (Scheduler, Webserver, Workers)
- PostgreSQL (Airflow metadata DB)
- SQLite (transformed data store)
- Dash (http://localhost:8050) for interactive charts
- Streamlit (http://localhost:8501) for exploratory UI
- Deployment:
- Single Docker Compose stack, services defined in docker-compose.yaml
- Ports exposed for Airflow UI (8080), Dash, and Streamlit
- Workflow:
- Raw data extracted from local archive (Data/)
- Airflow tasks orchestrate ETL steps defined in etl.py
- Cleaned data loaded into SQLite
- Dash and Streamlit read from SQLite for visualization

## Diagram (Mermaid)

```mermaid
graph TB
    %% User
    User["User Browser"]:::user

    subgraph "Docker Compose Network"
        direction TB
        AFweb["Airflow Webserver\n(port 8080)"]:::orchestration
        AFsched["Airflow Scheduler"]:::orchestration
        AFworker["Airflow Worker"]:::orchestration
        Postgres[(PostgreSQL\nmetadata DB)]:::database
        SQLite[(SQLite\nanalytical DB)]:::database
        RawDataFolder["Raw Data\nData/data.rar"]:::storage
        Logs["Logs Volume"]:::storage
        DashApp["Dash App\n(port 8050)"]:::dashboard
        StreamlitApp["Streamlit App\n(port 8501)"]:::dashboard
    end

    %% Data Flow
    User -->|Access UI| AFweb
    AFweb --> AFsched
    AFsched --> AFworker
    AFworker -->|reads/writes| Postgres
    AFworker -->|reads| RawDataFolder
    AFworker -->|writes| SQLite
    AFworker --> Logs
    User -->|View Charts| DashApp
    User -->|View UI| StreamlitApp
    DashApp -->|reads| SQLite
    StreamlitApp -->|reads| SQLite

    %% Click Events
    click AFweb "https://github.com/ayalaadaili/etl_airflow/blob/main/docker-compose.yaml"
    click AFsched "https://github.com/ayalaadaili/etl_airflow/blob/main/docker-compose.yaml"
    click AFworker "https://github.com/ayalaadaili/etl_airflow/blob/main/docker-compose.yaml"
    click AFworker "https://github.com/ayalaadaili/etl_airflow/blob/main/dags/etl.py"
    click AFworker "https://github.com/ayalaadaili/etl_airflow/tree/main/logs/"
    click AFworker "https://github.com/ayalaadaili/etl_airflow/blob/main/requirements.txt"
    click Postgres "https://github.com/ayalaadaili/etl_airflow/blob/main/docker-compose.yaml"
    click SQLite "https://github.com/ayalaadaili/etl_airflow/blob/main/dags/etl.py"
    click SQLite "https://github.com/ayalaadaili/etl_airflow/blob/main/docker-compose.yaml"
    click RawDataFolder "https://github.com/ayalaadaili/etl_airflow/blob/main/Data/data.rar"
    click DashApp "https://github.com/ayalaadaili/etl_airflow/blob/main/dashboard.py"
    click DashApp "https://github.com/ayalaadaili/etl_airflow/tree/main/Dockerfile"
    click DashApp "https://github.com/ayalaadaili/etl_airflow/blob/main/docker-compose.yaml"
    click StreamlitApp "https://github.com/ayalaadaili/etl_airflow/blob/main/app.py"
    click StreamlitApp "https://github.com/ayalaadaili/etl_airflow/blob/main/Dockerfile.streamlit"
    click StreamlitApp "https://github.com/ayalaadaili/etl_airflow/blob/main/docker-compose.yaml"

    %% Styles
    classDef orchestration fill:#D8BFD8,stroke:#9370DB,stroke-width:2px
    classDef database fill:#ADD8E6,stroke:#1E90FF,stroke-width:2px
    classDef dashboard fill:#90EE90,stroke:#32CD32,stroke-width:2px
    classDef storage fill:#FFE4B5,stroke:#FFA500,stroke-width:2px
    classDef user fill:#FFEFD5,stroke:#FFD700,stroke-width:2px
```
