How geetanjalidevadula/data-engineering-foundations-astronomer works
Project type and purpose
• This repository is not a single deployed application but a collection of learning artifacts—code samples and challenge scripts—for the “Data Engineering Foundations” course by Astronomer.
• Its purpose is to demonstrate core data-engineering workflows: SQL programming, Python-based transformations, orchestration with Apache Airflow, data‐warehousing concepts, and governance principles.
File structure
• Top level:
• .gitignore, LICENSE, README.md – standard repo metadata
• Python/ – contains five “Challenge” scripts (First through Fifth) implementing hands‐on exercises
• No production DAGs or warehouse schemas in code; the learning artifacts map directly to course modules.
README for additional insights
• Six course modules aligned to key stages of a data pipeline:
1) SQL programming (data modeling & querying)
2) Python for transformations
3) Hands-on ingestion/transformation
4) Apache Airflow orchestration
5) Data-warehouse design
6) Data‐governance controls
Main components
a. Data Sources – e.g., OLTP databases, CSVs, APIs (the inputs used in your SQL/Python challenges)
b. Ingestion & Orchestration – Apache Airflow scheduling and coordinating tasks
c. Data Transformation – Python scripts (the five Challenge .py files) and SQL queries
d. Data Warehouse – staging and analytical storage layer (conceptual)
e. Governance & Monitoring – lineage tracking, quality checks, metadata management
Relationships and interactions
• Data Sources feeding into Airflow
• Airflow DAG nodes invoking Python scripts and SQL jobs
• Transformed data landing in the Data Warehouse schema
• Governance processes querying metadata stores and dashboards to enforce quality
Architectural patterns and technologies
• Orchestration pattern: directed acyclic graph (DAG) of tasks
• Batch processing model for data extraction, transformation, load (ETL)
• Modular code: each Challenge script represents a transformation stage
• Technologies: Python (for transformations), SQL engines (e.g., Postgres/BigQuery), Apache Airflow
Detail level
• Essential stage even if it’s conceptual: ingestion, transform, load, governance
• Don’t overpopulate with every file; group the Python challenges under one “Data Transformation” node