# canhphu/data-pipeline-for-manufacturing architecture

> Architecture diagram of the GitHub repository canhphu/data-pipeline-for-manufacturing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/canhphu/data-pipeline-for-manufacturing
- Repository: https://github.com/canhphu/data-pipeline-for-manufacturing

## Overview

##### Project type and purpose
- This is a containerized, Airflow-based ETL data-pipeline project (“Data-Pipeline-for-Manufacturing”). Its job is to extract raw air-quality data (CSV), transform/clean it, and load the results.

##### File structure
- Top level:
- docker-compose.yaml under airflow/ → defines Airflow services (scheduler, webserver, metadata DB, broker, workers)
- airflow/dags/etl_pipeline.py → the master DAG orchestrating three tasks
- airflow/dags/extract.py, transform.py, load.py → the three task definitions
- data/air_quality_data.csv → raw input
- data/air_quality_cleaned.csv → transformed output
- requirements.txt → Python dependencies (likely apache-airflow, pandas, etc.)

##### README
- Minimal content, but project name confirms it’s a manufacturing-focused ETL pipeline.

a. Main components
- Data Source (Raw CSV in the data folder)
- Apache Airflow Orchestrator, which comprises:
- Webserver (UI)
- Scheduler (triggers DAG runs)
- Metadata Database (stores DAG/run/task state)
- Message Broker (if using CeleryExecutor)
- Workers (execute Python tasks)
- DAG definition (etl_pipeline.py) calling three task modules: extract, transform, load
- Data Sink (Cleaned CSV or database target)
- Supporting infrastructure: Docker Compose network

b. Relationships and data flow
1. Docker Compose brings up: Webserver, Scheduler, Metadata DB, Broker, Workers
2. Scheduler reads etl_pipeline.py → triggers a DAG run
3. Worker executes extract.py → reads data/air_quality_data.csv → writes interim data (in memory or staging)
4. transform.py picks up interim data → applies cleaning logic → writes to data/air_quality_cleaned.csv
6. Webserver/UI displays DAG status; Metadata DB/broker manage state and messaging

c. Architectural patterns and design principles
- Orchestration pattern (Airflow DAG)
- Task separation (single responsibility for extract/transform/load)
- Containerization (Docker Compose for reproducible infra)
- Modularity (each task lives in its own Python file)

d. Technologies, frameworks, libraries
- Apache Airflow (DAG engine, web UI, scheduler)
- Python (task code, pandas likely)
- Docker Compose (service orchestration)
- CSV file system (data source and sink)
- (Optional) Postgres or MySQL used by Airflow metadata DB
- (Optional) Redis/RabbitMQ as Celery broker

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Infrastructure Configurations
    subgraph "Infrastructure" 
        direction TB
        DC["Docker Compose\n(airflow/docker-compose.yaml)"]:::infra
        REQ["requirements.txt"]:::infra
        RDME["README.md"]:::infra
        ARCH["Architecture Draft"]:::infra
    end

    %% Data Sources and Sinks
    DS["Raw Data Source\n(data/air_quality_data.csv)"]:::data
    DN["Cleaned Data Sink\n(data/air_quality_cleaned.csv)"]:::data

    %% Airflow Services
    subgraph "Airflow Services" 
        direction TB
        WS["Webserver"]:::orchestration
        SCH["Scheduler"]:::orchestration
        MDB["Metadata DB"]:::orchestration
        BRK["Message Broker"]:::orchestration
        WRK["Worker"]:::orchestration
    end

    %% ETL DAG and Tasks
    subgraph "ETL DAG" 
        direction TB
        DAG["etl_pipeline.py"]:::dag
        EX["extract.py"]:::task
        TR["transform.py"]:::task
        LD["load.py"]:::task
    end

    %% Infrastructure startup
    DC --> WS
    DC --> SCH
    DC --> MDB
    DC --> BRK
    DC --> WRK

    %% DAG scheduling
    SCH -->|triggers DAG run| DAG
    WS -->|UI displays status| DAG
    DAG -->|calls extract| EX
    EX -->|reads CSV| DS
    EX -->|passes data| TR
    TR -->|writes cleaned CSV| DN
    TR -->|passes data| LD
    LD -->|loads cleaned data| DN
    LD -->|final destination| MDB

    %% Click Events
    click DC "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/airflow/docker-compose.yaml"
    click REQ "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/requirements.txt"
    click RDME "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/README.md"
    click ARCH "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/image/Untitled Diagram.drawio"
    click DS "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/data/air_quality_data.csv"
    click DN "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/data/air_quality_cleaned.csv"
    click DAG "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/airflow/dags/etl_pipeline.py"
    click EX "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/airflow/dags/extract.py"
    click TR "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/airflow/dags/transform.py"
    click LD "https://github.com/canhphu/data-pipeline-for-manufacturing/blob/main/airflow/dags/load.py"

    %% Styles
    classDef orchestration fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef task fill:#90EE90,stroke:#000,stroke-width:1px
    classDef data fill:#FFD700,stroke:#000,stroke-width:1px
    classDef infra fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef dag fill:#FFB6C1,stroke:#000,stroke-width:1px
```
