How sachapoulet/daliuge works
DALiuGE is a workflow-graph development, translation, deployment, and execution framework for large data-processing workloads. The documented user journey begins with EAGLE submitting a graph to the Translator; the resulting workflow can then be deployed and run by the Engine across managers and nodes. Engine managers build and deploy sessions, while node-side runtime components execute graph tasks and manage data. The graph retains major documented translation, scheduling, deployment, and execution capabilities; unsampled subsystem links are marked dashed where their exact call paths are not evidenced.
Main entry points: composite_manager.py Composite manager · node_manager.py Node manager · translator_rest.py Translator API · pipeline.py Translation pipeline
Components 24
Graph translation
- Translator API
daliuge-translator/dlg/translator/web/translator_rest.py - Translation pipeline
daliuge-translator/dlg/translator/pipeline.py - Graph preparation
daliuge-translator/dlg/translator/stages/prepare - Graph unrolling
daliuge-translator/dlg/translator/stages/unroll - Graph partitioning
daliuge-translator/dlg/translator/stages/partition/scheduler.py - Physical graph manager
daliuge-translator/dlg/translator/web/pg_manager.py - Translator UI
daliuge-translator/dlg/translator/web/main.js
Deployment
- Deployment clients
daliuge-engine/dlg/deploy - Helm deployment
daliuge-engine/dlg/deploy/helm_client.py - SLURM deployment
daliuge-engine/dlg/deploy/slurm_client.py
Execution
- Composite manager
daliuge-engine/dlg/manager/composite_manager.py - Island manager
daliuge-engine/dlg/manager/composite_manager.py - Node manager
daliuge-engine/dlg/manager/node_manager.py - Session lifecycle
daliuge-engine/dlg/manager/session.py - Graph loader
daliuge-engine/dlg/graph_loader.py - Data and applications
daliuge-engine/dlg/data/io.py - Data lifecycle
daliuge-engine/dlg/lifecycle/dlm.py - RPC transport
daliuge-engine/dlg/rpc.py - Runtime errors
daliuge-engine/dlg/runtime/error_management.py
Shared services
- Reproducibility
daliuge-common/dlg/common/reproducibility/reproducibility.py - REST utilities
daliuge-common/dlg/restutils.py - Manager protocol
daliuge-engine/dlg/ddap_protocol.py - Domain exceptions
daliuge-common/dlg/exceptions.py
Other components
- EAGLE user
Connections 26
- EAGLE user → Translator API: submits graph
- Translator UI → Translator API: uses API
- Translator API → Translation pipeline: translates graph
- Translation pipeline → Graph preparation: runs stage
- Translation pipeline → Graph unrolling: runs stage
- Translation pipeline → Graph partitioning: runs stage
- Translation pipeline → Reproducibility: stamps output
daliuge-translator/dlg/translator/pipeline.py - Translator API → Physical graph manager: manages graph
daliuge-translator/dlg/translator/web/pg_manager.py - Translator API → Graph partitioning: uses scheduler
- Deployment clients → Composite manager: starts managers
- Helm deployment → Composite manager: starts managers
daliuge-engine/dlg/deploy/helm_client.py - SLURM deployment → Composite manager: submits job
daliuge-engine/dlg/deploy/slurm_client.py - Composite manager → Island manager: coordinates partitions
daliuge-engine/dlg/manager/composite_manager.py - Composite manager → Session lifecycle: creates sessions
daliuge-engine/dlg/manager/composite_manager.py - Composite manager → Graph loader: loads graph
daliuge-engine/dlg/manager/composite_manager.py - Composite manager → Reproducibility: uses metadata
daliuge-engine/dlg/manager/composite_manager.py - Island manager → Node manager: dispatches work
- Node manager → Session lifecycle: manages sessions
daliuge-engine/dlg/manager/node_manager.py - Node manager → RPC transport: uses transport
daliuge-engine/dlg/manager/node_manager.py - Node manager → Data and applications: executes drops
daliuge-engine/dlg/manager/node_manager.py - Node manager → Data lifecycle: starts lifecycle
daliuge-engine/dlg/manager/node_manager.py - Node manager → Runtime errors: handles failures
daliuge-engine/dlg/manager/node_manager.py - Data and applications → Reproducibility: records provenance
- Translator API → REST utilities: serves API
- Composite manager → Manager protocol: uses protocol
daliuge-engine/dlg/manager/composite_manager.py - Node manager → Domain exceptions: raises errors
daliuge-engine/dlg/manager/node_manager.py