How apcpt42/intern-etl-project works
This repository implements two Airflow ETL workflows: MRFT quality-data integration and TAM/TAP trend analysis. MRFT ingests Google Sheets data and extracts operational quality data from Oracle and deviation data from MSSQL into a PostgreSQL warehouse, then transforms warehouse data for a data mart. The trend workflow extracts Oracle parameter and TAM/TAP records, combines them into the warehouse, and transfers processed data onward to the mart; the MRFT DAG can trigger it. Shared modules provide database access, sheet access, formatting, and date-range utilities. Workflow details are based on the README and sampled excerpts; some DAG task wiring was not included.
Main entry points: etl_qo_qc_mrft.py MRFT Workflow · dbConn.py Database Access · etl_qo_qc_trend_analysis.py Trend Workflow · sheetConn.py Sheet Access
Components 20
MRFT ETL
- MRFT Workflow
etl_qo_qc_mrft.py - Oracle Quality Extracts
etl_qo_qc_mrft.py - Deviation Extract
etl_qo_qc_mrft.py - Warehouse Loading
etl_qo_qc_mrft.py - Mart Preparation
etl_qo_qc_mrft.py
Trend ETL
- Trend Workflow
etl_qo_qc_trend_analysis.py - Oracle TAM/TAP Extract
etl_qo_qc_trend_analysis.py - TAM/TAP Combine
etl_qo_qc_trend_analysis.py - Warehouse Transfer
etl_qo_qc_trend_analysis.py - Mart Transfer
etl_qo_qc_trend_analysis.py
Shared Modules
- Sheet Access
CustomModule/sheetConn.py - Data Formatting
CustomModule/dataFormat.py - Date Partitioning
CustomModule/datelib_mod.py - Database Access
CustomModule/dbConn.py
Data Platforms
- MRFT Spreadsheet
- Oracle Sources
- MSSQL Source
- PostgreSQL Warehouse
- PostgreSQL Data Mart
Other components
- Airflow Scheduler
Connections 26
- Airflow Scheduler → MRFT Workflow: schedules
- Airflow Scheduler → Trend Workflow: schedules
- MRFT Workflow → Sheet Access: calls
- Sheet Access → MRFT Spreadsheet: reads
- MRFT Workflow → Data Formatting: formats
- MRFT Workflow → Oracle Quality Extracts: dispatches
- Oracle Quality Extracts → Oracle Sources: queries
- MRFT Workflow → Deviation Extract: dispatches
- Deviation Extract → MSSQL Source: queries
- MRFT Workflow → Warehouse Loading: dispatches
- MRFT Workflow → Mart Preparation: dispatches
- Warehouse Loading → Database Access: uses
- Mart Preparation → Database Access: uses
- Database Access → PostgreSQL Warehouse: reads/writes
- Database Access → PostgreSQL Data Mart: reads/writes
- MRFT Workflow → Trend Workflow: triggers
- Trend Workflow → Date Partitioning: uses
- Trend Workflow → Oracle TAM/TAP Extract: dispatches
- Oracle TAM/TAP Extract → Oracle Sources: queries
- Trend Workflow → TAM/TAP Combine: dispatches
- Trend Workflow → Warehouse Transfer: dispatches
- Trend Workflow → Mart Transfer: dispatches
- Warehouse Transfer → Database Access: uses
- Mart Transfer → Database Access: uses
- Database Access → PostgreSQL Warehouse: reads/writes
- Database Access → PostgreSQL Data Mart: reads/writes