# sleekdata/airflow_proj architecture

> Architecture diagram of the GitHub repository sleekdata/airflow_proj, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sleekdata/airflow_proj
- Repository: https://github.com/sleekdata/airflow_proj

## Overview

This repository is a collection of Apache Airflow DAG examples, rather than a single application. The principal data workflows demonstrate scheduled extraction, transformation, validation and loading, plus integrations with S3 and Snowflake; other DAGs illustrate branching, task dependencies, trigger rules and setup/teardown. The clearest end-to-end path is the exchange-rate download and cleanup DAG. Relationships for unsampled DAGs are omitted because the README only names the project and their source was not inspected; the graph reflects only visible task dependencies and calls.

Main entry points: [`airflow/dags/conditional_branching.py`](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py) (Branch decision), [`airflow/dags/exchange_rate_pipeline.py`](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/exchange_rate_pipeline.py) (Rate download), [`airflow/dags/tradititional.py`](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py) (Transform), [`airflow/plugins/clean_data.py`](https://github.com/sleekdata/airflow_proj/blob/main/airflow/plugins/clean_data.py) (Rate cleanup)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data_workflows["Data workflows"]
  node_traditional_extract["Extract<br/>[tradititional.py]"]
  node_traditional_transform["Transform<br/>[tradititional.py]"]
  node_traditional_validate["Validate<br/>[tradititional.py]"]
  node_traditional_load["Load<br/>[tradititional.py]"]
  node_exchange_download["Rate download"]
  node_exchange_clean["Rate cleanup<br/>[clean_data.py]"]
  node_exchange_email["Completion email"]
  node_xrate_fetch["Rate fetch<br/>[xrate_to_s3.py]"]
  node_s3_upload["S3 upload<br/>[xrate_to_s3.py]"]
  node_s3_sensor["S3 file sensor<br/>[empdata_process.py]"]
  node_snowflake_load["Snowflake load<br/>[empdata_process.py]"]
end

subgraph group_orchestration_patterns["Orchestration patterns"]
  node_conditional_check["File size check"]
  node_conditional_decide["Branch decision"]
  node_serial_transform["Serial transform"]
  node_serial_load["Serial load"]
  node_parallel_transform["Parallel transform"]
  node_parallel_load["Parallel load"]
  node_latest_only_dag["Latest-only tasks<br/>[LatestOnly.py]"]
  node_setup_teardown_dag["Setup and teardown<br/>[setup_teardown.py]"]
  node_trigger_rules_dag["Trigger rules<br/>[trigger_rules.py]"]
end

subgraph group_other_dags["Other DAG examples"]
  node_custom_s3_dag["Custom S3 transfer<br/>[custom_s3_trans.py]"]
  node_depends_past_dag["Past dependencies<br/>[dependsOnPast.py]"]
  node_producer_dag["FX producer"]
  node_consumer_dag["FX consumer"]
  node_git_dag["Git repository DAG<br/>[git_repo_dag.py]"]
  node_hybrid_dag["Hybrid DAG<br/>[hybrid_dag.py]"]
  node_live_rates_dag["Live rates"]
  node_profit_uk_dag["UK profit<br/>[profit_uk.py]"]
  node_taskflow_dag["TaskFlow API<br/>[taskflow_api.py]"]
  node_welcome_dag["Welcome DAG<br/>[welcome_dag.py]"]
end

subgraph group_shared_code["Shared code"]
  node_clean_data["Data cleaner<br/>[clean_data.py]"]
end

node_traditional_extract -->|"passes XCom"| node_traditional_transform
node_traditional_transform -->|"passes XCom"| node_traditional_validate
node_traditional_validate -->|"passes XCom"| node_traditional_load
node_exchange_download -->|"runs after"| node_exchange_clean
node_exchange_clean -->|"runs before"| node_exchange_email
node_exchange_download -->|"calls"| node_clean_data
node_xrate_fetch -->|"passes filename"| node_s3_upload
node_s3_sensor -->|"triggers load"| node_snowflake_load
node_conditional_check -->|"runs before"| node_conditional_decide
node_conditional_decide -->|"selects branch"| node_serial_transform
node_serial_transform -->|"runs before"| node_serial_load
node_conditional_decide -->|"selects branch"| node_parallel_transform
node_parallel_transform -->|"runs before"| node_parallel_load

click node_traditional_extract "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py"
click node_traditional_transform "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py"
click node_traditional_validate "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py"
click node_traditional_load "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py"
click node_exchange_download "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/exchange_rate_pipeline.py"
click node_exchange_clean "https://github.com/sleekdata/airflow_proj/blob/main/airflow/plugins/clean_data.py"
click node_exchange_email "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/exchange_rate_pipeline.py"
click node_xrate_fetch "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/xrate_to_s3.py"
click node_s3_upload "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/xrate_to_s3.py"
click node_s3_sensor "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/empdata_process.py"
click node_snowflake_load "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/empdata_process.py"
click node_conditional_check "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py"
click node_conditional_decide "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py"
click node_serial_transform "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py"
click node_serial_load "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py"
click node_parallel_transform "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py"
click node_parallel_load "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py"
click node_latest_only_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/LatestOnly.py"
click node_setup_teardown_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/setup_teardown.py"
click node_trigger_rules_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/trigger_rules.py"
click node_custom_s3_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/custom_s3_trans.py"
click node_depends_past_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/dependsOnPast.py"
click node_producer_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/fx_data%20producer.py"
click node_consumer_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/fx_data_consumer.py"
click node_git_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/git_repo_dag.py"
click node_hybrid_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/hybrid_dag.py"
click node_live_rates_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/live_exchange_rates.py"
click node_profit_uk_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/profit_uk.py"
click node_taskflow_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/taskflow_api.py"
click node_welcome_dag "https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/welcome_dag.py"
click node_clean_data "https://github.com/sleekdata/airflow_proj/blob/main/airflow/plugins/clean_data.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_traditional_extract,node_traditional_transform,node_traditional_validate,node_traditional_load,node_exchange_download,node_exchange_clean,node_exchange_email,node_xrate_fetch,node_s3_upload,node_s3_sensor,node_snowflake_load toneBlue
class node_conditional_check,node_conditional_decide,node_serial_transform,node_serial_load,node_parallel_transform,node_parallel_load,node_latest_only_dag,node_setup_teardown_dag,node_trigger_rules_dag toneAmber
class node_custom_s3_dag,node_depends_past_dag,node_producer_dag,node_consumer_dag,node_git_dag,node_hybrid_dag,node_live_rates_dag,node_profit_uk_dag,node_taskflow_dag,node_welcome_dag toneMint
class node_clean_data toneRose
```

## Components

### Data workflows

- [Extract](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py): `airflow/dags/tradititional.py`
- [Transform](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py): `airflow/dags/tradititional.py`
- [Validate](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py): `airflow/dags/tradititional.py`
- [Load](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/tradititional.py): `airflow/dags/tradititional.py`
- [Rate download](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/exchange_rate_pipeline.py): `airflow/dags/exchange_rate_pipeline.py`
- [Rate cleanup](https://github.com/sleekdata/airflow_proj/blob/main/airflow/plugins/clean_data.py): `airflow/plugins/clean_data.py`
- [Completion email](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/exchange_rate_pipeline.py): `airflow/dags/exchange_rate_pipeline.py`
- [Rate fetch](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/xrate_to_s3.py): `airflow/dags/xrate_to_s3.py`
- [S3 upload](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/xrate_to_s3.py): `airflow/dags/xrate_to_s3.py`
- [S3 file sensor](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/empdata_process.py): `airflow/dags/empdata_process.py`
- [Snowflake load](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/empdata_process.py): `airflow/dags/empdata_process.py`

### Orchestration patterns

- [File size check](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py): `airflow/dags/conditional_branching.py`
- [Branch decision](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py): `airflow/dags/conditional_branching.py`
- [Serial transform](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py): `airflow/dags/conditional_branching.py`
- [Serial load](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py): `airflow/dags/conditional_branching.py`
- [Parallel transform](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py): `airflow/dags/conditional_branching.py`
- [Parallel load](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/conditional_branching.py): `airflow/dags/conditional_branching.py`
- [Latest-only tasks](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/LatestOnly.py): `airflow/dags/LatestOnly.py`
- [Setup and teardown](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/setup_teardown.py): `airflow/dags/setup_teardown.py`
- [Trigger rules](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/trigger_rules.py): `airflow/dags/trigger_rules.py`

### Other DAG examples

- [Custom S3 transfer](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/custom_s3_trans.py): `airflow/dags/custom_s3_trans.py`
- [Past dependencies](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/dependsOnPast.py): `airflow/dags/dependsOnPast.py`
- [FX producer](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/fx_data%20producer.py): `airflow/dags/fx_data producer.py`
- [FX consumer](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/fx_data_consumer.py): `airflow/dags/fx_data_consumer.py`
- [Git repository DAG](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/git_repo_dag.py): `airflow/dags/git_repo_dag.py`
- [Hybrid DAG](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/hybrid_dag.py): `airflow/dags/hybrid_dag.py`
- [Live rates](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/live_exchange_rates.py): `airflow/dags/live_exchange_rates.py`
- [UK profit](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/profit_uk.py): `airflow/dags/profit_uk.py`
- [TaskFlow API](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/taskflow_api.py): `airflow/dags/taskflow_api.py`
- [Welcome DAG](https://github.com/sleekdata/airflow_proj/blob/main/airflow/dags/welcome_dag.py): `airflow/dags/welcome_dag.py`

### Shared code

- [Data cleaner](https://github.com/sleekdata/airflow_proj/blob/main/airflow/plugins/clean_data.py): `airflow/plugins/clean_data.py`

## Connections

- Extract → Transform: passes XCom (evidence: `airflow/dags/tradititional.py`)
- Transform → Validate: passes XCom (evidence: `airflow/dags/tradititional.py`)
- Validate → Load: passes XCom (evidence: `airflow/dags/tradititional.py`)
- Rate download → Rate cleanup: runs after (evidence: `airflow/dags/exchange_rate_pipeline.py`)
- Rate cleanup → Completion email: runs before (evidence: `airflow/dags/exchange_rate_pipeline.py`)
- Rate download → Data cleaner: calls (evidence: `airflow/dags/exchange_rate_pipeline.py`)
- Rate fetch → S3 upload: passes filename (evidence: `airflow/dags/xrate_to_s3.py`)
- S3 file sensor → Snowflake load: triggers load (evidence: `airflow/dags/empdata_process.py`)
- File size check → Branch decision: runs before (evidence: `airflow/dags/conditional_branching.py`)
- Branch decision → Serial transform: selects branch (evidence: `airflow/dags/conditional_branching.py`)
- Serial transform → Serial load: runs before (evidence: `airflow/dags/conditional_branching.py`)
- Branch decision → Parallel transform: selects branch (evidence: `airflow/dags/conditional_branching.py`)
- Parallel transform → Parallel load: runs before (evidence: `airflow/dags/conditional_branching.py`)