How androchentw/airflow-playground works
• Running a standalone Airflow hello-world,
• Setting up a Great Expectations hello-world,
• Combining Great Expectations with Airflow via DAGs, and
• Integrating Great Expectations with GCP BigQuery.
Main Components
• Airflow Environment: Represent the Apache Airflow orchestration system with key subcomponents like the scheduler, webserver, worker(s), and the Docker container(s) these run in (given the docker-compose.yaml).
• DAGs: Show a specific component for the DAGs (located in the “dags” directory). Mark the “great_expectations_airflow.py” DAG that integrates Great Expectations validations within Airflow.
• Great Expectations: Illustrate the role of Great Expectations, as demonstrated by the examples in the “src” directory (both the notebook and Python script). This component handles data quality validations.
• GCP BigQuery (optional): Since one of the topics is Great Expectations + GCP BigQuery, include an external component for the BigQuery data warehouse that may serve as a target or source of data validations.
• Docker Environment: Since docker-compose.yaml is used, include a Docker container layer to represent how the system environment is spun up.
• Repository/Source Files: Mention sample notebooks and scripts from “src” and configuration files that support the orchestration and validations.
Relationships and Interactions
• Data flow paths: From ingestion (if applicable) into Airflow, moving through DAG tasks to data validations, and then possibly outputting to BigQuery or storing results.
• Docker as a foundational layer that encapsulates the Airflow and Great Expectations components.
Architectural Patterns and Design Principles
• A modular separation between orchestration (Airflow) and data validation (Great Expectations), each with its own clearly defined responsibilities.
• The system is containerized (using docker-compose), so represent this abstraction to underline how environment consistency is maintained.
Relevant Technologies, Frameworks, and Libraries
• Apache Airflow – for orchestration (scheduler, webserver, workers)
• Great Expectations – for data validation
• Docker & Docker Compose – for containerization and environment setup
• (Optional) GCP BigQuery – for the external data warehouse
• Python scripts, Jupyter notebooks, and possibly the airflow-provider-great-expectations package, as mentioned in the README.