How canhphu/data-pipeline-for-manufacturing works
Project type and purpose
• This is a containerized, Airflow-based ETL data-pipeline project (“Data-Pipeline-for-Manufacturing”). Its job is to extract raw air-quality data (CSV), transform/clean it, and load the results.
File structure
• Top level:
• docker-compose.yaml under airflow/ → defines Airflow services (scheduler, webserver, metadata DB, broker, workers)
• airflow/dags/etl_pipeline.py → the master DAG orchestrating three tasks
• airflow/dags/extract.py, transform.py, load.py → the three task definitions
• data/air_quality_data.csv → raw input
• data/air_quality_cleaned.csv → transformed output
• requirements.txt → Python dependencies (likely apache-airflow, pandas, etc.)
README
• Minimal content, but project name confirms it’s a manufacturing-focused ETL pipeline.
a. Main components
• Data Source (Raw CSV in the data folder)
• Apache Airflow Orchestrator, which comprises:
• Webserver (UI)
• Scheduler (triggers DAG runs)
• Metadata Database (stores DAG/run/task state)
• Message Broker (if using CeleryExecutor)
• Workers (execute Python tasks)
• DAG definition (etl_pipeline.py) calling three task modules: extract, transform, load
• Data Sink (Cleaned CSV or database target)
• Supporting infrastructure: Docker Compose network
b. Relationships and data flow
1. Docker Compose brings up: Webserver, Scheduler, Metadata DB, Broker, Workers
2. Scheduler reads etl_pipeline.py → triggers a DAG run
3. Worker executes extract.py → reads data/air_quality_data.csv → writes interim data (in memory or staging)
4. transform.py picks up interim data → applies cleaning logic → writes to data/air_quality_cleaned.csv
6. Webserver/UI displays DAG status; Metadata DB/broker manage state and messaging
c. Architectural patterns and design principles
• Orchestration pattern (Airflow DAG)
• Task separation (single responsibility for extract/transform/load)
• Containerization (Docker Compose for reproducible infra)
• Modularity (each task lives in its own Python file)
d. Technologies, frameworks, libraries
• Apache Airflow (DAG engine, web UI, scheduler)
• Python (task code, pandas likely)
• Docker Compose (service orchestration)
• CSV file system (data source and sink)
• (Optional) Postgres or MySQL used by Airflow metadata DB
• (Optional) Redis/RabbitMQ as Celery broker