# damavis/airflow-pentaho-plugin architecture

> Architecture diagram of the GitHub repository damavis/airflow-pentaho-plugin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/damavis/airflow-pentaho-plugin
- Repository: https://github.com/damavis/airflow-pentaho-plugin

## Overview

##### Main Components
- Airflow System: Represent the Airflow infrastructure (scheduler, webserver, and workers) which is the host for this plugin. Emphasize that this plugin must be deployed on all nodes (scheduler, webserver, and worker) that execute tasks.
- Pentaho Systems: Illustrate external Pentaho components, including the Carte servers (for remote execution) and local PDI installations (for running jobs/transformations using Kitchen and Pan). Show elements such as:
– PDI Repository (with configuration details like rep, pentaho_home, etc.)
– Carte server (with its host, port, credentials, and optional JSON extras for repository info).
- Plugin Components: Display the core package, “airflow_pentaho”, which contains:
– Hooks: Modules (e.g., kettle.py, carte.py) that encapsulate the connections and communication patterns. They are the abstraction layer to interact with Pentaho systems.
– Operators: Different Airflow Operator implementations (CarteJobOperator, CarteTransOperator, KitchenOperator, PanOperator) which use the hooks to trigger jobs/transformations.
- Supporting Elements: Include configuration files (setup.py, requirements.txt, pre-commit and CI/CD pipelines via .travis.yml and workflows in .github), test suites, and sample DAGs from “sample_dags” to show how the plugin is integrated and validated.

##### Architectural Patterns and Design Principles
- Plugin-Based Extension: Clearly show that this project extends Airflow’s capabilities with custom operators and hooks to seamlessly integrate Pentaho functionality.
- Separation of Concerns: Differentiate between the “orchestration” (Airflow DAGs and scheduling) and the “execution” (the actual execution on Pentaho systems).
- Connectivity and Abstraction: Emphasize that hooks abstract the connection details, allowing operators to call them without dealing with low-level protocol or authentication issues.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Airflow Cluster
    subgraph "Apache Airflow Cluster"
        AS("Airflow Scheduler"):::airflowService
        AW("Airflow Webserver"):::airflowService
        AWorker("Airflow Worker (with Pentaho Plugin)"):::airflowService
    end

    %% Airflow Pentaho Plugin Layer
    subgraph "Airflow Pentaho Plugin"
        PI("Plugin Integration (plugin.py)"):::integration
        subgraph "Hooks"
            H_C("Carte Hook (carte.py)"):::hook
            H_K("Kettle Hook (kettle.py)"):::hook
        end
        subgraph "Operators"
            O_CJ("CarteJobOperator (CarteJobOperator.py)"):::operator
            O_CT("CarteTransOperator (CarteTransOperator.py)"):::operator
            O_K("KitchenOperator (KitchenOperator.py)"):::operator
            O_P("PanOperator (PanOperator.py)"):::operator
        end
        P("Plugin Package"):::plugin
    end

    %% External Pentaho Systems
    subgraph "External Pentaho Systems"
        Carte("Carte Server"):::pentaho
        PDI("PDI Repository / Local PDI (Kitchen & Pan)"):::pentaho
    end

    %% Supporting Elements
    subgraph "Supporting Elements"
        CF("Configuration Files (setup.py, requirements.txt, .travis.yml, .github/workflows)"):::supporting
        SD("Sample DAG (pdi_flow.py)"):::supporting
    end

    %% Configuration / Metadata
    Config("Airflow Connections (pdi_default, JSON extras)"):::config

    %% Connections between Airflow Cluster and Plugin
    AS -->|"deploys Plugin"| PI
    AW -->|"deploys Plugin"| PI
    AWorker -->|"executes Plugin"| PI
    PI -->|"integrates with"| P

    %% Plugin Internal Dependencies: Operators use Hooks
    O_CJ -->|"calls"| H_C
    O_CT -->|"calls"| H_C
    O_K -->|"calls"| H_K
    O_P -->|"calls"| H_K

    %% Plugin Operators communicate with External Pentaho Systems
    O_CJ -->|"triggers"| Carte
    O_CT -->|"triggers"| Carte
    O_K -->|"triggers"| PDI
    O_P -->|"triggers"| PDI

    %% Configuration feeds Plugin Integration
    Config -->|"configures"| PI

    %% Supporting Elements providing configuration and examples
    CF -->|"configures system"| PI
    SD -->|"demonstrates usage"| PI

    %% Click Events for Plugin Components
    click P "https://github.com/damavis/airflow-pentaho-plugin/tree/master/airflow_pentaho/"
    click H_C "https://github.com/damavis/airflow-pentaho-plugin/blob/master/airflow_pentaho/hooks/carte.py"
    click H_K "https://github.com/damavis/airflow-pentaho-plugin/blob/master/airflow_pentaho/hooks/kettle.py"
    click O_CJ "https://github.com/damavis/airflow-pentaho-plugin/blob/master/airflow_pentaho/operators/CarteJobOperator.py"
    click O_CT "https://github.com/damavis/airflow-pentaho-plugin/blob/master/airflow_pentaho/operators/CarteTransOperator.py"
    click O_K "https://github.com/damavis/airflow-pentaho-plugin/blob/master/airflow_pentaho/operators/KitchenOperator.py"
    click O_P "https://github.com/damavis/airflow-pentaho-plugin/blob/master/airflow_pentaho/operators/PanOperator.py"
    click PI "https://github.com/damavis/airflow-pentaho-plugin/blob/master/airflow_pentaho/plugin.py"
    click CF "https://github.com/damavis/airflow-pentaho-plugin/blob/master/setup.py"
    click SD "https://github.com/damavis/airflow-pentaho-plugin/blob/master/sample_dags/pdi_flow.py"

    %% Styles
    classDef airflowService fill:#aacef9,stroke:#333,stroke-width:2px;
    classDef plugin fill:#f9c0d9,stroke:#333,stroke-width:2px;
    classDef integration fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef hook fill:#d4efdf,stroke:#333,stroke-width:2px;
    classDef operator fill:#fcf3cf,stroke:#333,stroke-width:2px;
    classDef pentaho fill:#f5b7b1,stroke:#333,stroke-width:2px;
    classDef config fill:#d6eaf8,stroke:#333,stroke-width:2px;
    classDef supporting fill:#fad7a0,stroke:#333,stroke-width:2px;
```
