# apcpt42/intern-etl-project architecture

> Architecture diagram of the GitHub repository apcpt42/intern-etl-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apcpt42/intern-etl-project
- Repository: https://github.com/apcpt42/intern-etl-project

## Overview

This repository implements two Airflow ETL workflows: MRFT quality-data integration and TAM/TAP trend analysis. MRFT ingests Google Sheets data and extracts operational quality data from Oracle and deviation data from MSSQL into a PostgreSQL warehouse, then transforms warehouse data for a data mart. The trend workflow extracts Oracle parameter and TAM/TAP records, combines them into the warehouse, and transfers processed data onward to the mart; the MRFT DAG can trigger it. Shared modules provide database access, sheet access, formatting, and date-range utilities. Workflow details are based on the README and sampled excerpts; some DAG task wiring was not included.

Main entry points: [`etl_qo_qc_mrft.py`](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py) (MRFT Workflow), [`CustomModule/dbConn.py`](https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/dbConn.py) (Database Access), [`etl_qo_qc_trend_analysis.py`](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py) (Trend Workflow), [`CustomModule/sheetConn.py`](https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/sheetConn.py) (Sheet Access)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_mrft["MRFT ETL"]
  node_mrft_dag["MRFT Workflow<br/>[etl_qo_qc_mrft.py]"]
  node_mrft_oracle["Oracle Quality Extracts<br/>[etl_qo_qc_mrft.py]"]
  node_mrft_mssql["Deviation Extract<br/>[etl_qo_qc_mrft.py]"]
  node_mrft_load["Warehouse Loading<br/>[etl_qo_qc_mrft.py]"]
  node_mrft_mart["Mart Preparation<br/>[etl_qo_qc_mrft.py]"]
end

subgraph group_trend["Trend ETL"]
  node_trend_dag["Trend Workflow"]
  node_trend_extract["Oracle TAM/TAP Extract"]
  node_trend_combine["TAM/TAP Combine"]
  node_trend_wh["Warehouse Transfer"]
  node_trend_dm["Mart Transfer"]
end

subgraph group_shared["Shared Modules"]
  node_sheet_access["Sheet Access<br/>[sheetConn.py]"]
  node_formatting["Data Formatting<br/>[dataFormat.py]"]
  node_date_ranges["Date Partitioning<br/>[datelib_mod.py]"]
  node_db_access["Database Access<br/>[dbConn.py]"]
end

subgraph group_platform["Data Platforms"]
  node_sheets[("MRFT Spreadsheet")]
  node_oracle[("Oracle Sources")]
  node_mssql[("MSSQL Source")]
  node_warehouse[("PostgreSQL Warehouse")]
  node_mart[("PostgreSQL Data Mart")]
end

node_airflow(("Airflow Scheduler"))

node_airflow -->|"schedules"| node_mrft_dag
node_airflow -->|"schedules"| node_trend_dag
node_mrft_dag -->|"calls"| node_sheet_access
node_sheet_access -->|"reads"| node_sheets
node_mrft_dag -->|"formats"| node_formatting
node_mrft_dag -->|"dispatches"| node_mrft_oracle
node_mrft_oracle -->|"queries"| node_oracle
node_mrft_dag -->|"dispatches"| node_mrft_mssql
node_mrft_mssql -->|"queries"| node_mssql
node_mrft_dag -->|"dispatches"| node_mrft_load
node_mrft_dag -->|"dispatches"| node_mrft_mart
node_mrft_load -->|"uses"| node_db_access
node_mrft_mart -->|"uses"| node_db_access
node_db_access -->|"reads/writes"| node_warehouse
node_db_access -->|"reads/writes"| node_mart
node_mrft_dag -.->|"triggers"| node_trend_dag
node_trend_dag -->|"uses"| node_date_ranges
node_trend_dag -->|"dispatches"| node_trend_extract
node_trend_extract -->|"queries"| node_oracle
node_trend_dag -->|"dispatches"| node_trend_combine
node_trend_dag -->|"dispatches"| node_trend_wh
node_trend_dag -->|"dispatches"| node_trend_dm
node_trend_wh -->|"uses"| node_db_access
node_trend_dm -->|"uses"| node_db_access
node_db_access -->|"reads/writes"| node_warehouse
node_db_access -->|"reads/writes"| node_mart

click node_mrft_dag "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py"
click node_sheet_access "https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/sheetConn.py"
click node_formatting "https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/dataFormat.py"
click node_mrft_oracle "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py"
click node_mrft_mssql "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py"
click node_mrft_load "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py"
click node_mrft_mart "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py"
click node_trend_dag "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py"
click node_date_ranges "https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/datelib_mod.py"
click node_trend_extract "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py"
click node_trend_combine "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py"
click node_trend_wh "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py"
click node_trend_dm "https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py"
click node_db_access "https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/dbConn.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mrft_dag,node_mrft_oracle,node_mrft_mssql,node_mrft_load,node_mrft_mart toneBlue
class node_trend_dag,node_trend_extract,node_trend_combine,node_trend_wh,node_trend_dm toneAmber
class node_sheet_access,node_formatting,node_date_ranges,node_db_access toneMint
class node_sheets,node_oracle,node_mssql,node_warehouse,node_mart,node_airflow toneRose
```

## Components

### MRFT ETL

- [MRFT Workflow](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py): `etl_qo_qc_mrft.py`
- [Oracle Quality Extracts](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py): `etl_qo_qc_mrft.py`
- [Deviation Extract](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py): `etl_qo_qc_mrft.py`
- [Warehouse Loading](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py): `etl_qo_qc_mrft.py`
- [Mart Preparation](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_mrft.py): `etl_qo_qc_mrft.py`

### Trend ETL

- [Trend Workflow](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py): `etl_qo_qc_trend_analysis.py`
- [Oracle TAM/TAP Extract](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py): `etl_qo_qc_trend_analysis.py`
- [TAM/TAP Combine](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py): `etl_qo_qc_trend_analysis.py`
- [Warehouse Transfer](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py): `etl_qo_qc_trend_analysis.py`
- [Mart Transfer](https://github.com/apcpt42/intern-etl-project/blob/main/etl_qo_qc_trend_analysis.py): `etl_qo_qc_trend_analysis.py`

### Shared Modules

- [Sheet Access](https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/sheetConn.py): `CustomModule/sheetConn.py`
- [Data Formatting](https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/dataFormat.py): `CustomModule/dataFormat.py`
- [Date Partitioning](https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/datelib_mod.py): `CustomModule/datelib_mod.py`
- [Database Access](https://github.com/apcpt42/intern-etl-project/blob/main/CustomModule/dbConn.py): `CustomModule/dbConn.py`

### Data Platforms

- **MRFT Spreadsheet**
- **Oracle Sources**
- **MSSQL Source**
- **PostgreSQL Warehouse**
- **PostgreSQL Data Mart**

### Other components

- **Airflow Scheduler**

## Connections

- Airflow Scheduler → MRFT Workflow: schedules
- Airflow Scheduler → Trend Workflow: schedules
- MRFT Workflow → Sheet Access: calls
- Sheet Access → MRFT Spreadsheet: reads
- MRFT Workflow → Data Formatting: formats
- MRFT Workflow → Oracle Quality Extracts: dispatches
- Oracle Quality Extracts → Oracle Sources: queries
- MRFT Workflow → Deviation Extract: dispatches
- Deviation Extract → MSSQL Source: queries
- MRFT Workflow → Warehouse Loading: dispatches
- MRFT Workflow → Mart Preparation: dispatches
- Warehouse Loading → Database Access: uses
- Mart Preparation → Database Access: uses
- Database Access → PostgreSQL Warehouse: reads/writes
- Database Access → PostgreSQL Data Mart: reads/writes
- MRFT Workflow → Trend Workflow: triggers
- Trend Workflow → Date Partitioning: uses
- Trend Workflow → Oracle TAM/TAP Extract: dispatches
- Oracle TAM/TAP Extract → Oracle Sources: queries
- Trend Workflow → TAM/TAP Combine: dispatches
- Trend Workflow → Warehouse Transfer: dispatches
- Trend Workflow → Mart Transfer: dispatches
- Warehouse Transfer → Database Access: uses
- Mart Transfer → Database Access: uses
- Database Access → PostgreSQL Warehouse: reads/writes
- Database Access → PostgreSQL Data Mart: reads/writes