How hexag158/data-pipeline-with-airflow works
This project is a data pipeline built using Apache Airflow to orchestrate ETL (Extract, Transform, Load) processes. The system is designed to extract data from a dummy API, save the results to CSV files, load the data into a local Oracle Database, and send out email notifications when the pipeline completes or encounters errors. The use of Docker Compose indicates that the entire Airflow environment is containerized, making it easier to deploy and manage.
Main Components
a. Airflow Environment (Docker):
• Docker container environment initiated by docker-compose. This will include key Airflow services such as the Scheduler, Webserver, and Workers.
b. DAGs (Workflows):
• There are two main DAG files in the “dags” folder: one for saving CSV files and another for loading data into the Oracle Database and sending email notifications.
c. Data Storage:
• “data” folder as a repository for the CSV files generated by the ETL process.
d. Logging:
• “logs” folder for capturing runtime logs of the pipeline.
e. Configuration:
• The “config” folder contains the airflow.cfg file which dictates configuration options for Airflow.
f. Plugins:
• Custom extensions available in the “plugins” folder (e.g., custom operators, sensors, hooks).
Relationships and Interactions
a. The Docker Compose setup (docker-compose.yaml) creates and orchestrates the Airflow environment by launching the containerized environment.
b. Within the container, the Airflow Scheduler picks up the DAG definitions from the “dags” folder and triggers the tasks.
c. The tasks within each DAG execute in a defined order:
• Data extraction from a dummy API → Data saved as CSV in the “data” folder.
• The second DAG includes steps to load data into the Oracle Database and send out email notifications.
d. The “logs” folder collects process logs that aid in troubleshooting.
e. The “config” folder is used to manage the overall configuration of Airflow, which impacts scheduling, concurrency, and execution parameters.
f. The “plugins” folder allows users to extend Airflow’s functionality if special operators or integrations are needed.
Architectural Patterns and Design Principles
• The design follows a modular pattern where each aspect (data capture, transformation, load, alerting) is encapsulated in separate tasks within a DAG.
• Separation of concerns: Configuration is externalized, logging is separated, and custom behavior is added via plugins.
• Containerization with Docker ensures consistency across development and production environments.
Relevant Technologies and Tools
• Apache Airflow as the workflow orchestrator.
• Docker and Docker Compose for container management.
• Oracle Database as the destination for data post-transformation.
• Email services configured within Airflow for notifications.
• Python scripts (and potentially notebooks as suggested by file naming) for executing ETL tasks.