# workivanaccount/my_project_ecology architecture

> Architecture diagram of the GitHub repository workivanaccount/my_project_ecology, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/workivanaccount/my_project_ecology
- Repository: https://github.com/workivanaccount/my_project_ecology

## Overview

This project is an Airflow-orchestrated air-quality data pipeline: it fetches daily measurements, stores Parquet in S3-compatible storage, loads them into PostgreSQL, then calculates a daily BI mart. The DAGs are the runtime entry points; shared configuration and date helpers support them. The README documents Metabase dashboards and the raw→ODS→STG→DM layers. Metabase is represented as a documented consumer, but its implementation is not shown; Docker and deployment wiring are omitted.

Main entry points: [`dags/ecology_raw_s3_to_pg.py`](https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_raw_s3_to_pg.py) (S3 to ODS DAG), [`dags/ecology_api_to_s3.py`](https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_api_to_s3.py) (API to S3 DAG), [`dags/dm_air_quality_daily.py`](https://github.com/workivanaccount/my_project_ecology/blob/main/dags/dm_air_quality_daily.py) (Daily Mart DAG), [`plugins/utils/dag_config.py`](https://github.com/workivanaccount/my_project_ecology/blob/main/plugins/utils/dag_config.py) (DAG Configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["API Ingestion"]
  node_ingest_dag["API to S3 DAG"]
  node_raw_store[("Raw S3 Parquet")]
end

subgraph group_warehouse["Warehouse Pipeline"]
  node_transfer_dag["S3 to ODS DAG"]
  node_duckdb["DuckDB ETL"]
  node_dwh[("PostgreSQL DWH")]
end

subgraph group_shared["Shared DAG Utilities"]
  node_config["DAG Configuration<br/>[dag_config.py]"]
  node_dates["Interval Dates<br/>[dates.py]"]
end

subgraph group_analytics["BI Analytics"]
  node_mart_dag["Daily Mart DAG"]
  node_metabase["Metabase BI"]
end

node_api(("Air Quality API"))

node_ingest_dag -->|"fetches JSON"| node_api
node_ingest_dag -->|"writes Parquet"| node_raw_store
node_transfer_dag -->|"waits for success"| node_ingest_dag
node_transfer_dag -->|"reads Parquet"| node_raw_store
node_transfer_dag -->|"runs ETL"| node_duckdb
node_transfer_dag -->|"loads ODS"| node_dwh
node_mart_dag -->|"waits for success"| node_transfer_dag
node_mart_dag -->|"builds STG and DM"| node_dwh
node_dwh -->|"serves BI"| node_metabase
node_ingest_dag -->|"uses settings"| node_config
node_ingest_dag -->|"gets interval"| node_dates
node_transfer_dag -->|"uses settings"| node_config
node_transfer_dag -->|"gets interval"| node_dates
node_mart_dag -->|"uses defaults"| node_config

click node_ingest_dag "https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_api_to_s3.py"
click node_transfer_dag "https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_raw_s3_to_pg.py"
click node_duckdb "https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_raw_s3_to_pg.py"
click node_mart_dag "https://github.com/workivanaccount/my_project_ecology/blob/main/dags/dm_air_quality_daily.py"
click node_metabase "https://github.com/workivanaccount/my_project_ecology/tree/main/metabase"
click node_config "https://github.com/workivanaccount/my_project_ecology/blob/main/plugins/utils/dag_config.py"
click node_dates "https://github.com/workivanaccount/my_project_ecology/blob/main/plugins/utils/dates.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_dag,node_raw_store toneBlue
class node_transfer_dag,node_duckdb,node_dwh toneAmber
class node_config,node_dates,node_api toneMint
class node_mart_dag,node_metabase toneRose
```

## Components

### API Ingestion

- [API to S3 DAG](https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_api_to_s3.py): `dags/ecology_api_to_s3.py`
- **Raw S3 Parquet**

### Warehouse Pipeline

- [S3 to ODS DAG](https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_raw_s3_to_pg.py): `dags/ecology_raw_s3_to_pg.py`
- [DuckDB ETL](https://github.com/workivanaccount/my_project_ecology/blob/main/dags/ecology_raw_s3_to_pg.py): `dags/ecology_raw_s3_to_pg.py`
- **PostgreSQL DWH**

### Shared DAG Utilities

- [DAG Configuration](https://github.com/workivanaccount/my_project_ecology/blob/main/plugins/utils/dag_config.py): `plugins/utils/dag_config.py`
- [Interval Dates](https://github.com/workivanaccount/my_project_ecology/blob/main/plugins/utils/dates.py): `plugins/utils/dates.py`

### BI Analytics

- [Daily Mart DAG](https://github.com/workivanaccount/my_project_ecology/blob/main/dags/dm_air_quality_daily.py): `dags/dm_air_quality_daily.py`
- [Metabase BI](https://github.com/workivanaccount/my_project_ecology/tree/main/metabase): `metabase`

### Other components

- **Air Quality API**

## Connections

- API to S3 DAG → Air Quality API: fetches JSON (evidence: `dags/ecology_api_to_s3.py`)
- API to S3 DAG → Raw S3 Parquet: writes Parquet (evidence: `dags/ecology_api_to_s3.py`)
- S3 to ODS DAG → API to S3 DAG: waits for success (evidence: `dags/ecology_raw_s3_to_pg.py`)
- S3 to ODS DAG → Raw S3 Parquet: reads Parquet (evidence: `dags/ecology_raw_s3_to_pg.py`)
- S3 to ODS DAG → DuckDB ETL: runs ETL (evidence: `dags/ecology_raw_s3_to_pg.py`)
- S3 to ODS DAG → PostgreSQL DWH: loads ODS (evidence: `dags/ecology_raw_s3_to_pg.py`)
- Daily Mart DAG → S3 to ODS DAG: waits for success (evidence: `dags/dm_air_quality_daily.py`)
- Daily Mart DAG → PostgreSQL DWH: builds STG and DM (evidence: `dags/dm_air_quality_daily.py`)
- PostgreSQL DWH → Metabase BI: serves BI
- API to S3 DAG → DAG Configuration: uses settings (evidence: `dags/ecology_api_to_s3.py`)
- API to S3 DAG → Interval Dates: gets interval (evidence: `dags/ecology_api_to_s3.py`)
- S3 to ODS DAG → DAG Configuration: uses settings (evidence: `dags/ecology_raw_s3_to_pg.py`)
- S3 to ODS DAG → Interval Dates: gets interval (evidence: `dags/ecology_raw_s3_to_pg.py`)
- Daily Mart DAG → DAG Configuration: uses defaults (evidence: `dags/dm_air_quality_daily.py`)