How airflow-plugins/airflow-dag-creation-manager-plugin works
Project Overview and Purpose
• This project is an Apache Airflow plugin called “DAG Creation Manager Plugin.” Its purpose is to allow users to create and manage DAGs (Directed Acyclic Graphs) through a dedicated web UI.
• It extends Airflow by adding new web routes (accessible through Airflow’s Admin tab), additional configuration options (set via airflow.cfg), and custom authentication/backends as needed.
Main Components
a. Airflow Ecosystem (External Component)
– Airflow Web Server: The plugin integrates with Airflow’s web interface.
– Airflow Scheduler and Executor: While not a direct part of the plugin, understanding the plugin’s placement in the overall Airflow architecture is key.
b. DAG Creation Manager Plugin (Internal Plugin Component)
– Plugin Core:
• Contains main plugin logic (e.g., plugins/dcmp/dag_creation_manager_plugin.py).
• Holds core functionality for DAG creation, management, and conversion (e.g., dcmp/dag_converter.py).
– Authentication Module:
• Found under plugins/dcmp/auth.
• Contains authentication backends (e.g., password_auth.py) and views/models to support login functionalities, which are optionally used if Airflow’s authentication is enabled.
– UI Components:
• Templates: HTML files under plugins/dcmp/templates (e.g., approve.html, compare.html, edit.html, etc.) used to render the plugin’s web pages.
• Static Assets: CSS and JavaScript files under plugins/dcmp/static. Notice various JS files (like ace.js, edit.js, etc.) and CSS files used for the web UI.
– Utility and Tools:
• Utility Modules: Such as plugins/dcmp/utils.py providing common helper functions.
• Admin/Upgrade Scripts: Under the tools folder (e.g., upgradedb.py, update_superuser.py) which are used to manage database upgrades and other tasks.
– Configuration Module:
• plugins/dcmp/settings.py holds settings specific to the plugin.
• README instructions detail additional configuration required within airflow.cfg for customized behavior.
Relationships and Data Flow
• The Airflow Web Server loads the plugin from the configured plugins folder. This plugin registers additional web endpoints.
• User interactions with the web UI (rendered using the templates and static assets) trigger the plugin’s backend logic to create or manage DAG definitions.
• The authentication module (if enabled) intercepts and validates user credentials.
• Utility scripts and command-line tools are used for maintenance tasks, such as database upgrades.
• The configuration settings provided in airflow.cfg guide how the plugin behaves, especially around UI options, auth, and DAG line interpolation.
Architectural Patterns and Design Principles
• Plugin Architecture: The design leverages Airflow’s plugin system. It is modular and integrates tightly with an external system.
• MVC-like Separation: Although not a full MVC framework, you have clear separation between:
– Models (plugins/dcmp/models.py, auth/models.py)
– Views/Templates (HTML files in plugins/dcmp/templates)
– Controller Logic (views in plugins/dcmp/auth/views.py and plugin core code)
• Extensibility: The structure supports custom authentication backends and configurable DAG properties through settings.