How gtoonstra/etl-with-airflow works
Project Type and Purpose
• This repository is dedicated to showcasing best practices for building ETL pipelines using Apache Airflow. The multitude of examples (e.g., datavault, datavault2, etl-example, hive-example, mssql-example, file-ingest) indicate that the repository serves as both a documentation source and a code repository for different ETL patterns and integration strategies.
• The presence of multiple docker-compose files (for different configurations like LocalExecutor, BigData, dbt, hive, MSSQL) underscores that the project aims to demonstrate how to set up various ETL environments using containerized services.
File Structure
• Top-level directories:
• "docs": Contains documentation files, tutorials, and perhaps deployment guides.
• "examples": Houses numerous directories with complete end-to-end examples, each illustrating different ETL patterns, integrations, or data modeling approaches (like Data Vault, Hive-based pipelines, and MSSQL examples).
• "jdbc": Contains external jar dependencies (for connectors like PostgreSQL).
• Numerous docker-compose YAML files exist at the project root for different deployment scenarios, suggesting that each example or environment may require a slightly different orchestration of components.
• Various shell scripts (run_datavault.sh, run_dv2.sh, etc.) indicate start and stop commands used for quickly testing or standing up the environments.
README for Additional Insights
• The README emphasizes that this is a repository of examples and usage patterns built around the Airflow ecosystem, not the official Airflow documentation.
• The purpose is to provide guidance for implementing ETL pipelines using Airflow, hinting that the architecture should clearly delineate the orchestration layer (Airflow) from the data processing and storage components.
a. Main Components:
• Airflow Core: Include components such as the Airflow Scheduler, Web Server, Worker(s), and Metadata Database. Indicate any separation if using a LocalExecutor versus a Celery or Kubernetes Executor based on the provided docker-compose files.
• ETL DAGs/Operators: Represent the various DAG examples (e.g., datavault, etl-example, hive-example, mssql-example) that implement ETL best practices.
• Data Sources and Targets: Illustrate connections to multiple external data systems, such as relational databases (MSSQL, PostgreSQL), Hadoop ecosystems (Hive, HDFS), cloud data warehouses (BigQuery in some sql examples), and file ingestion systems.
• Infrastructure/Orchestration: Show how docker-compose orchestrates the deployment and integration of these components. You might include containers for services like the database for Airflow, message queues (if applicable) and other required middleware.
• External Libraries/Connectors: Note the presence of JDBC connectors (the jar file) and any additional middleware (e.g., Sqoop, file operators) used to communicate with external systems.
b. Relationships and Interactions:
• Airflow as the central orchestrator that triggers the DAGs.
• Interaction between individual ETL tasks (extraction, transformation, and load) and the respective data stores.
• Integration with external systems and dashboards, such as reference documentation or monitoring components mentioned in the docs.
c. Architectural Patterns or Principles:
• A modular design. Each example (datavault, etl, hive, mssql) is its own module, reflecting a pattern where core ETL processes are abstracted as operators and hooks.
• Use of containerization (with docker-compose) to manage environment-specific configurations.
• ETL lifecycle stages (ingest, stage, transform, load) and the use of clear separation of duties between orchestration (Airflow) and data operations.
d. Relevant Technologies, Frameworks, or Libraries:
• Apache Airflow (DAGs, Operators, Hooks)
• Docker and docker-compose (for container orchestration)
• Databases such as MSSQL, PostgreSQL, Hive (for sample data sources and targets)
• Additional frameworks like dbt (as indicated by the datavault2-dbt example) and connectors like JDBC