How workivanaccount/my_project_ecology works
This project is an Airflow-orchestrated air-quality data pipeline: it fetches daily measurements, stores Parquet in S3-compatible storage, loads them into PostgreSQL, then calculates a daily BI mart. The DAGs are the runtime entry points; shared configuration and date helpers support them. The README documents Metabase dashboards and the raw→ODS→STG→DM layers. Metabase is represented as a documented consumer, but its implementation is not shown; Docker and deployment wiring are omitted.
Main entry points: ecology_raw_s3_to_pg.py S3 to ODS DAG · ecology_api_to_s3.py API to S3 DAG · dm_air_quality_daily.py Daily Mart DAG · dag_config.py DAG Configuration
Components 10
API Ingestion
- API to S3 DAG
dags/ecology_api_to_s3.py - Raw S3 Parquet
Warehouse Pipeline
- S3 to ODS DAG
dags/ecology_raw_s3_to_pg.py - DuckDB ETL
dags/ecology_raw_s3_to_pg.py - PostgreSQL DWH
Shared DAG Utilities
- DAG Configuration
plugins/utils/dag_config.py - Interval Dates
plugins/utils/dates.py
BI Analytics
- Daily Mart DAG
dags/dm_air_quality_daily.py - Metabase BI
metabase
Other components
- Air Quality API
Connections 14
- API to S3 DAG → Air Quality API: fetches JSON
dags/ecology_api_to_s3.py - API to S3 DAG → Raw S3 Parquet: writes Parquet
dags/ecology_api_to_s3.py - S3 to ODS DAG → API to S3 DAG: waits for success
dags/ecology_raw_s3_to_pg.py - S3 to ODS DAG → Raw S3 Parquet: reads Parquet
dags/ecology_raw_s3_to_pg.py - S3 to ODS DAG → DuckDB ETL: runs ETL
dags/ecology_raw_s3_to_pg.py - S3 to ODS DAG → PostgreSQL DWH: loads ODS
dags/ecology_raw_s3_to_pg.py - Daily Mart DAG → S3 to ODS DAG: waits for success
dags/dm_air_quality_daily.py - Daily Mart DAG → PostgreSQL DWH: builds STG and DM
dags/dm_air_quality_daily.py - PostgreSQL DWH → Metabase BI: serves BI
- API to S3 DAG → DAG Configuration: uses settings
dags/ecology_api_to_s3.py - API to S3 DAG → Interval Dates: gets interval
dags/ecology_api_to_s3.py - S3 to ODS DAG → DAG Configuration: uses settings
dags/ecology_raw_s3_to_pg.py - S3 to ODS DAG → Interval Dates: gets interval
dags/ecology_raw_s3_to_pg.py - Daily Mart DAG → DAG Configuration: uses defaults
dags/dm_air_quality_daily.py