How dogukannulu/docker-airflow works
Main Components
• Dockerized Airflow Container:
• This is the heart of the project. The Dockerfile builds upon the official Python 3.7-slim image and installs Apache Airflow along with any additional dependencies.
• Within this container, Airflow components such as the webserver, scheduler, and (when applicable) the worker processes run.
• Executor Variants:
• Database Service (PostgreSQL):
• Message Broker (Redis):
• For setups using the CeleryExecutor, a Redis container is used as the Celery broker. This should be depicted as a separate component that communicates with both the scheduler and worker containers.
• DAGs and Scripts:
• The “dags” directory holds sample DAG files (like tuto.py). This is where the user’s workflow definitions reside. Represent this as a volume mounted into the Airflow container.
• Configuration:
• The “config/airflow.cfg” and environment variable settings (as noted in the README) are critical. Show these as either a separate configuration layer or annotate the Airflow container indicating that it is driven by custom configurations.
• CI/CD and Build Components:
Relationships and Interactions
• That the Airflow webserver provides the UI and API endpoints, and highlight the connections between the scheduler and webserver components.
Architectural Patterns
• The system design leverages containerization and microservice-oriented patterns even though it’s packaged as a single image by default. The separation between configuration, storage (Postgres), and message brokering (Redis) is essential in making Airflow scalable (especially using the CeleryExecutor).
• Use of environment variables for configuration overrides (AIRFLOW__<SECTION>__<KEY>), which allows the system to be flexible and easily re-deployable.