How sleekdata/airflow_proj works
This repository is a collection of Apache Airflow DAG examples, rather than a single application. The principal data workflows demonstrate scheduled extraction, transformation, validation and loading, plus integrations with S3 and Snowflake; other DAGs illustrate branching, task dependencies, trigger rules and setup/teardown. The clearest end-to-end path is the exchange-rate download and cleanup DAG. Relationships for unsampled DAGs are omitted because the README only names the project and their source was not inspected; the graph reflects only visible task dependencies and calls.
Main entry points: conditional_branching.py Branch decision · exchange_rate_pipeline.py Rate download · tradititional.py Transform · clean_data.py Rate cleanup
Components 31
Data workflows
- Extract
airflow/dags/tradititional.py - Transform
airflow/dags/tradititional.py - Validate
airflow/dags/tradititional.py - Load
airflow/dags/tradititional.py - Rate download
airflow/dags/exchange_rate_pipeline.py - Rate cleanup
airflow/plugins/clean_data.py - Completion email
airflow/dags/exchange_rate_pipeline.py - Rate fetch
airflow/dags/xrate_to_s3.py - S3 upload
airflow/dags/xrate_to_s3.py - S3 file sensor
airflow/dags/empdata_process.py - Snowflake load
airflow/dags/empdata_process.py
Orchestration patterns
- File size check
airflow/dags/conditional_branching.py - Branch decision
airflow/dags/conditional_branching.py - Serial transform
airflow/dags/conditional_branching.py - Serial load
airflow/dags/conditional_branching.py - Parallel transform
airflow/dags/conditional_branching.py - Parallel load
airflow/dags/conditional_branching.py - Latest-only tasks
airflow/dags/LatestOnly.py - Setup and teardown
airflow/dags/setup_teardown.py - Trigger rules
airflow/dags/trigger_rules.py
Other DAG examples
- Custom S3 transfer
airflow/dags/custom_s3_trans.py - Past dependencies
airflow/dags/dependsOnPast.py - FX producer
airflow/dags/fx_data producer.py - FX consumer
airflow/dags/fx_data_consumer.py - Git repository DAG
airflow/dags/git_repo_dag.py - Hybrid DAG
airflow/dags/hybrid_dag.py - Live rates
airflow/dags/live_exchange_rates.py - UK profit
airflow/dags/profit_uk.py - TaskFlow API
airflow/dags/taskflow_api.py - Welcome DAG
airflow/dags/welcome_dag.py
Shared code
- Data cleaner
airflow/plugins/clean_data.py
Connections 13
- Extract → Transform: passes XCom
airflow/dags/tradititional.py - Transform → Validate: passes XCom
airflow/dags/tradititional.py - Validate → Load: passes XCom
airflow/dags/tradititional.py - Rate download → Rate cleanup: runs after
airflow/dags/exchange_rate_pipeline.py - Rate cleanup → Completion email: runs before
airflow/dags/exchange_rate_pipeline.py - Rate download → Data cleaner: calls
airflow/dags/exchange_rate_pipeline.py - Rate fetch → S3 upload: passes filename
airflow/dags/xrate_to_s3.py - S3 file sensor → Snowflake load: triggers load
airflow/dags/empdata_process.py - File size check → Branch decision: runs before
airflow/dags/conditional_branching.py - Branch decision → Serial transform: selects branch
airflow/dags/conditional_branching.py - Serial transform → Serial load: runs before
airflow/dags/conditional_branching.py - Branch decision → Parallel transform: selects branch
airflow/dags/conditional_branching.py - Parallel transform → Parallel load: runs before
airflow/dags/conditional_branching.py