How harsh199323/marketing-attribution-etl-framework--maef works
Project type and purpose
• This is an Apache Airflow–driven ETL pipeline in Python.
• It automates extraction of raw data, transformation, loading into a database, API integration, and reporting as CSV.
File structure
• Top-level:
– main.py (entrypoint for triggering run outside Airflow)
– webserver_config.py (Airflow UI config)
– requirements.txt, README.md
• dirs:
– dags/ihc_api_dag.py → defines the DAG and task dependencies
– src/etl (extract.py, transform.py, load.py, reporting.py) → ETL stages
– src/api (ihc_api.py, api_utils.py) → external API client
– src/database (schema.py, db_utils.py) → DB schema & utilities
– src/utils → date handling
• data/ holds raw SQL, outputs, formatted JSON/CSV.
README for additional insights
• Tech stack: Python, Apache Airflow, SQLite (or other RDBMS), external HTTP API.
Detail without overcomplexity
• Essential components and their interactions.
• Do not break modules into sub-functions—group at the task level.
• Flow mirrors the DAG’s order exactly.