How damavis/airflow-pentaho-plugin works
Main Components
• Airflow System: Represent the Airflow infrastructure (scheduler, webserver, and workers) which is the host for this plugin. Emphasize that this plugin must be deployed on all nodes (scheduler, webserver, and worker) that execute tasks.
• Pentaho Systems: Illustrate external Pentaho components, including the Carte servers (for remote execution) and local PDI installations (for running jobs/transformations using Kitchen and Pan). Show elements such as:
– PDI Repository (with configuration details like rep, pentaho_home, etc.)
– Carte server (with its host, port, credentials, and optional JSON extras for repository info).
• Plugin Components: Display the core package, “airflow_pentaho”, which contains:
– Hooks: Modules (e.g., kettle.py, carte.py) that encapsulate the connections and communication patterns. They are the abstraction layer to interact with Pentaho systems.
– Operators: Different Airflow Operator implementations (CarteJobOperator, CarteTransOperator, KitchenOperator, PanOperator) which use the hooks to trigger jobs/transformations.
• Supporting Elements: Include configuration files (setup.py, requirements.txt, pre-commit and CI/CD pipelines via .travis.yml and workflows in .github), test suites, and sample DAGs from “sample_dags” to show how the plugin is integrated and validated.
Architectural Patterns and Design Principles
• Plugin-Based Extension: Clearly show that this project extends Airflow’s capabilities with custom operators and hooks to seamlessly integrate Pentaho functionality.
• Separation of Concerns: Differentiate between the “orchestration” (Airflow DAGs and scheduling) and the “execution” (the actual execution on Pentaho systems).
• Connectivity and Abstraction: Emphasize that hooks abstract the connection details, allowing operators to call them without dealing with low-level protocol or authentication issues.