# geoedf/engine architecture

> Architecture diagram of the GitHub repository geoedf/engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geoedf/engine
- Repository: https://github.com/geoedf/engine

## Overview

##### Main Components
- Workflow Engine Core – Represented by the GeoEDFWorkflow and WorkflowEngine classes, which are the heart of task planning and execution.
- Configuration Module – Show the geoedf.cfg file (from the cfg directory) that drives the engine’s site-specific settings such as execution targets, HUBzero submit wrappers, and paths to critical executables.
- Helper Components – These include modules in geoedfengine/helper:
- GeoEDFConnector: likely handles connections (e.g., to data sources or remote services).
- GeoEDFProcessor: processes individual tasks.
- PegasusBroker: interfaces with the Pegasus WMS to plan and manage workflow tasks.
- SubmitBroker: works with HUBzero’s submit tool to actually dispatch workflows.
- Other utilities (WorkflowBuilder, WorkflowDBHelper, WorkflowUtils) that assist in constructing and managing workflows.
- Build Scripts – The binaries in the bin/ directory indicate support scripts for building various components of the workflow, such as “build-conn-plugin-subdax”, “build-final-subdax”, etc. They should be represented as operational utilities that package or prepare parts of the workflow engine.
- User/Application Interaction – Illustrate how users (or other systems) instantiate the workflow engine via Python (as shown in the README code snippet) and trigger both execution and monitoring phases.

##### Relationships and Interactions
- The User/Application imports and instantiates the GeoEDFWorkflow class using a YAML workflow file as input.
- The GeoEDFWorkflow makes use of configuration (via geoedf.cfg) to set up the environment.
- The core engine (WorkflowEngine) then leverages helper modules:
- The PegasusBroker and SubmitBroker modules interact with external systems (Pegasus WMS and HUBzero submit, respectively).
- Other helper modules ensure that errors are handled (GeoEDFError) and tasks are built and checked (WorkflowBuilder, WorkflowDBHelper, etc.).
- After planning, tasks are executed asynchronously – the design should show a flow from task definition, to submission, to execution, and finally to monitoring, where the workflow directory is used to track progress.

##### Architectural Patterns and Design Principles
- Encapsulation & Modularity – The project is broken into clear modules: core workflow engine, configuration, and a set of helper components. Emphasize that the design leverages separation of concerns (the GeoEDFWorkflow class hides the complexity of the execution logic).
- Asynchronous Execution – Show that the execution process is not synchronous but handed off to external systems via brokers.
- Extensibility – Through multiple helper modules, the system is made extensible to support additional connectors or new types of execution targets if need be.
- Plug-and-play Design – Since the engine aims to provide a plug-and-play experience for workflows described in YAML, indicate that components are loosely coupled via a configuration file, enabling easy updates or extensions.

##### Relevant Technologies, Frameworks, and Libraries
- Python – As the entire engine is a Python library/module.
- Pegasus WMS – Represented by the PegasusBroker component.
- HUBzero Submit – Represented by the SubmitBroker component.
- YAML – The engine is designed to consume YAML workflow definitions.
- Build and Deployment Tools – The scripts residing in the bin/ directory and configuration management (manifest, setup.py) are part of the overall toolchain.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A("User/Application"):::user
    B("YAML Workflow Definition"):::input

    A --> B
    B --> E("GeoEDFWorkflow"):::core

    subgraph "GeoEDFWorkflow Engine"
        D("geoedf.cfg"):::config
        subgraph "Core Components"
            E("GeoEDFWorkflow"):::core
            F("WorkflowEngine"):::core
        end
    end

    D --> E
    D --> F
    E --> F

    subgraph "Helper Modules"
        H1("GeoEDFConnector"):::helper
        H2("GeoEDFProcessor"):::helper
        H3("WorkflowBuilder"):::helper
        H4("WorkflowDBHelper"):::helper
        H5("WorkflowUtils"):::helper
        H6("GeoEDFError"):::helper
        H7("PegasusBroker"):::helper
        H8("SubmitBroker"):::helper
    end

    F --> H1
    F --> H2
    F --> H3
    F --> H4
    F --> H5
    F --> H6
    F --> H7
    F --> H8

    H7 -->|"submits to"| G("Pegasus WMS"):::external
    H8 -->|"submits to"| I("HUBzero Submit System"):::external

    subgraph "Build Scripts"
        J("build-conn-plugin-subdax"):::build
        K("build-final-subdax"):::build
        L("build-proc-plugin-subdax"):::build
    end

    J --> F
    K --> F
    L --> F

    %% Click Events
    click E "https://github.com/geoedf/engine/blob/master/geoedfengine/GeoEDFWorkflow.py"
    click F "https://github.com/geoedf/engine/blob/master/geoedfengine/WorkflowEngine.py"
    click D "https://github.com/geoedf/engine/blob/master/cfg/geoedf.cfg"
    click H1 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/GeoEDFConnector.py"
    click H2 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/GeoEDFProcessor.py"
    click H3 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/WorkflowBuilder.py"
    click H4 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/WorkflowDBHelper.py"
    click H5 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/WorkflowUtils.py"
    click H6 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/GeoEDFError.py"
    click H7 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/PegasusBroker.py"
    click H8 "https://github.com/geoedf/engine/blob/master/geoedfengine/helper/SubmitBroker.py"
    click J "https://github.com/geoedf/engine/tree/master/bin/build-conn-plugin-subdax"
    click K "https://github.com/geoedf/engine/tree/master/bin/build-final-subdax"
    click L "https://github.com/geoedf/engine/tree/master/bin/build-proc-plugin-subdax"

    %% Styles
    classDef user fill:#fde2e2,stroke:#b30000,stroke-width:2px;
    classDef input fill:#fff2cc,stroke:#b36b00,stroke-width:2px;
    classDef config fill:#ffebcc,stroke:#b38f00,stroke-width:2px;
    classDef core fill:#cce5ff,stroke:#00509e,stroke-width:2px;
    classDef helper fill:#e6ccff,stroke:#6a0dad,stroke-width:2px;
    classDef external fill:#d0f0c0,stroke:#008000,stroke-width:2px;
    classDef build fill:#ffe6f2,stroke:#d1005d,stroke-width:2px;
```
