How geoedf/engine works
Main Components
• Workflow Engine Core – Represented by the GeoEDFWorkflow and WorkflowEngine classes, which are the heart of task planning and execution.
• Configuration Module – Show the geoedf.cfg file (from the cfg directory) that drives the engine’s site-specific settings such as execution targets, HUBzero submit wrappers, and paths to critical executables.
• Helper Components – These include modules in geoedfengine/helper:
• GeoEDFConnector: likely handles connections (e.g., to data sources or remote services).
• GeoEDFProcessor: processes individual tasks.
• PegasusBroker: interfaces with the Pegasus WMS to plan and manage workflow tasks.
• SubmitBroker: works with HUBzero’s submit tool to actually dispatch workflows.
• Other utilities (WorkflowBuilder, WorkflowDBHelper, WorkflowUtils) that assist in constructing and managing workflows.
• Build Scripts – The binaries in the bin/ directory indicate support scripts for building various components of the workflow, such as “build-conn-plugin-subdax”, “build-final-subdax”, etc. They should be represented as operational utilities that package or prepare parts of the workflow engine.
• User/Application Interaction – Illustrate how users (or other systems) instantiate the workflow engine via Python (as shown in the README code snippet) and trigger both execution and monitoring phases.
Relationships and Interactions
• The User/Application imports and instantiates the GeoEDFWorkflow class using a YAML workflow file as input.
• The GeoEDFWorkflow makes use of configuration (via geoedf.cfg) to set up the environment.
• The core engine (WorkflowEngine) then leverages helper modules:
• The PegasusBroker and SubmitBroker modules interact with external systems (Pegasus WMS and HUBzero submit, respectively).
• Other helper modules ensure that errors are handled (GeoEDFError) and tasks are built and checked (WorkflowBuilder, WorkflowDBHelper, etc.).
• After planning, tasks are executed asynchronously – the design should show a flow from task definition, to submission, to execution, and finally to monitoring, where the workflow directory is used to track progress.
Architectural Patterns and Design Principles
• Encapsulation & Modularity – The project is broken into clear modules: core workflow engine, configuration, and a set of helper components. Emphasize that the design leverages separation of concerns (the GeoEDFWorkflow class hides the complexity of the execution logic).
• Asynchronous Execution – Show that the execution process is not synchronous but handed off to external systems via brokers.
• Extensibility – Through multiple helper modules, the system is made extensible to support additional connectors or new types of execution targets if need be.
• Plug-and-play Design – Since the engine aims to provide a plug-and-play experience for workflows described in YAML, indicate that components are loosely coupled via a configuration file, enabling easy updates or extensions.
Relevant Technologies, Frameworks, and Libraries
• Python – As the entire engine is a Python library/module.
• Pegasus WMS – Represented by the PegasusBroker component.
• HUBzero Submit – Represented by the SubmitBroker component.
• YAML – The engine is designed to consume YAML workflow definitions.
• Build and Deployment Tools – The scripts residing in the bin/ directory and configuration management (manifest, setup.py) are part of the overall toolchain.