How hsg15/airflow_etl_pipeline works
Project Type & Purpose
• This is a data-engineering ETL pipeline built with Apache Airflow.
• It fetches JSON from NASA’s APOD HTTP API, transforms selected fields, and loads the results into a PostgreSQL database.
• The pipeline runs locally in Docker Compose during development and is deployed in the cloud via Astronomer on AWS (using RDS for Postgres).
File-Structure Analysis
• Top‐level:
• Dockerfile & packages.txt / requirements.txt → custom Airflow image + Python deps
• docker-compose.yml → local Airflow + Postgres services and networks
• airflow_settings.yaml → Astronomer/Airflow config
• dags/etl.py & exampledag.py → DAG definitions (extract, transform, load tasks)
• No front-end; it’s strictly backend/orchestration.
• Key patterns: DAG orchestration (Airflow), containerization (Docker), external API calls, database interactions.
README Insights
• Local workflow: astro dev init → Docker Compose brings up Airflow scheduler, webserver, worker, triggerer + Postgres.
• Airflow Connections: HTTP hook to NASA API, Postgres hook to DB.
• Cloud workflow: code in Git → astro deploy → Astronomer Cloud runs Airflow → connects to AWS RDS.
a. Main Components
• External Services:
• NASA APOD HTTP API
• Astronomer Cloud (Airflow managed service)
• AWS RDS (Postgres)
• Local Dev Environment (Docker Compose):
• Airflow Scheduler, Webserver, Worker, Triggerer (each a container)
• Postgres container
• VS Code / Astro CLI (developer workstation)
• Production Environment (Astronomer/AWS):
• Astronomer-managed Airflow nodes
• RDS Postgres instance
b. Relationships & Data Flow
1. Airflow Scheduler → triggers DAG according to schedule.
2. “extract_apod” task uses HTTP connection → calls NASA API → retrieves JSON.
3. “transform_apod” task processes JSON → extracts relevant fields.
4. “load_data” task uses Postgres connection → inserts/updates data in nasa_apod_data.
5. For local dev: all components communicate over Docker network.
6. For cloud: Git push → Astro CLI deploys code → Astronomer Airflow pulls DAGs from repo → connects to AWS RDS.
c. Architectural Patterns & Principles
• ETL Pipeline as code (DAGs)
• Separation of concerns: extract, transform, load tasks are modular.
• Containerization & immutable infrastructure for consistency.
• IaC / GitOps via Astro CLI for deployment.
d. Key Technologies
• Apache Airflow (Python, DAGs, Operators, Hooks)
• Docker & Docker Compose
• PostgreSQL
• Astronomer (Airflow Cloud orchestration)
• AWS RDS