# sachapoulet/daliuge architecture

> Architecture diagram of the GitHub repository sachapoulet/daliuge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sachapoulet/daliuge
- Repository: https://github.com/sachapoulet/daliuge

## Overview

DALiuGE is a workflow-graph development, translation, deployment, and execution framework for large data-processing workloads. The documented user journey begins with EAGLE submitting a graph to the Translator; the resulting workflow can then be deployed and run by the Engine across managers and nodes. Engine managers build and deploy sessions, while node-side runtime components execute graph tasks and manage data. The graph retains major documented translation, scheduling, deployment, and execution capabilities; unsampled subsystem links are marked dashed where their exact call paths are not evidenced.

Main entry points: [`daliuge-engine/dlg/manager/composite_manager.py`](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/composite_manager.py) (Composite manager), [`daliuge-engine/dlg/manager/node_manager.py`](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/node_manager.py) (Node manager), [`daliuge-translator/dlg/translator/web/translator_rest.py`](https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/web/translator_rest.py) (Translator API), [`daliuge-translator/dlg/translator/pipeline.py`](https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/pipeline.py) (Translation pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_translation["Graph translation"]
  node_translator_web["Translator API<br/>[translator_rest.py]"]
  node_translator_pipeline["Translation pipeline<br/>[pipeline.py]"]
  node_prepare["Graph preparation"]
  node_unroll["Graph unrolling"]
  node_partition["Graph partitioning<br/>[scheduler.py]"]
  node_physical_graph[("Physical graph manager<br/>[pg_manager.py]")]
  node_translator_ui["Translator UI<br/>[main.js]"]
end

subgraph group_deployment["Deployment"]
  node_deployment["Deployment clients"]
  node_helm["Helm deployment<br/>[helm_client.py]"]
  node_slurm["SLURM deployment<br/>[slurm_client.py]"]
end

subgraph group_execution["Execution"]
  node_composite["Composite manager"]
  node_island["Island manager"]
  node_node_manager["Node manager<br/>[node_manager.py]"]
  node_session["Session lifecycle<br/>[session.py]"]
  node_graph_loader["Graph loader<br/>[graph_loader.py]"]
  node_drops["Data and applications<br/>[io.py]"]
  node_lifecycle["Data lifecycle<br/>[dlm.py]"]
  node_rpc["RPC transport<br/>[rpc.py]"]
  node_errors["Runtime errors"]
end

subgraph group_shared["Shared services"]
  node_repro["Reproducibility<br/>[reproducibility.py]"]
  node_rest["REST utilities<br/>[restutils.py]"]
  node_protocol["Manager protocol<br/>[ddap_protocol.py]"]
  node_exceptions["Domain exceptions<br/>[exceptions.py]"]
end

node_eagle(("EAGLE user"))

node_eagle -->|"submits graph"| node_translator_web
node_translator_ui -.->|"uses API"| node_translator_web
node_translator_web -.->|"translates graph"| node_translator_pipeline
node_translator_pipeline -.->|"runs stage"| node_prepare
node_translator_pipeline -.->|"runs stage"| node_unroll
node_translator_pipeline -.->|"runs stage"| node_partition
node_translator_pipeline -.->|"stamps output"| node_repro
node_translator_web -.->|"manages graph"| node_physical_graph
node_translator_web -->|"uses scheduler"| node_partition
node_deployment -.->|"starts managers"| node_composite
node_helm -->|"starts managers"| node_composite
node_slurm -.->|"submits job"| node_composite
node_composite -->|"coordinates partitions"| node_island
node_composite -->|"creates sessions"| node_session
node_composite -->|"loads graph"| node_graph_loader
node_composite -->|"uses metadata"| node_repro
node_island -.->|"dispatches work"| node_node_manager
node_node_manager -->|"manages sessions"| node_session
node_node_manager -->|"uses transport"| node_rpc
node_node_manager -->|"executes drops"| node_drops
node_node_manager -->|"starts lifecycle"| node_lifecycle
node_node_manager -->|"handles failures"| node_errors
node_drops -.->|"records provenance"| node_repro
node_translator_web -.->|"serves API"| node_rest
node_composite -->|"uses protocol"| node_protocol
node_node_manager -->|"raises errors"| node_exceptions

click node_translator_web "https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/web/translator_rest.py"
click node_translator_pipeline "https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/pipeline.py"
click node_prepare "https://github.com/sachapoulet/daliuge/tree/master/daliuge-translator/dlg/translator/stages/prepare"
click node_unroll "https://github.com/sachapoulet/daliuge/tree/master/daliuge-translator/dlg/translator/stages/unroll"
click node_partition "https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/stages/partition/scheduler.py"
click node_physical_graph "https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/web/pg_manager.py"
click node_translator_ui "https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/web/main.js"
click node_deployment "https://github.com/sachapoulet/daliuge/tree/master/daliuge-engine/dlg/deploy"
click node_helm "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/deploy/helm_client.py"
click node_slurm "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/deploy/slurm_client.py"
click node_composite "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/composite_manager.py"
click node_island "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/composite_manager.py"
click node_node_manager "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/node_manager.py"
click node_session "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/session.py"
click node_graph_loader "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/graph_loader.py"
click node_drops "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/data/io.py"
click node_lifecycle "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/lifecycle/dlm.py"
click node_rpc "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/rpc.py"
click node_errors "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/runtime/error_management.py"
click node_repro "https://github.com/sachapoulet/daliuge/blob/master/daliuge-common/dlg/common/reproducibility/reproducibility.py"
click node_rest "https://github.com/sachapoulet/daliuge/blob/master/daliuge-common/dlg/restutils.py"
click node_protocol "https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/ddap_protocol.py"
click node_exceptions "https://github.com/sachapoulet/daliuge/blob/master/daliuge-common/dlg/exceptions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_translator_web,node_translator_pipeline,node_prepare,node_unroll,node_partition,node_physical_graph,node_translator_ui,node_eagle toneBlue
class node_deployment,node_helm,node_slurm toneAmber
class node_composite,node_island,node_node_manager,node_session,node_graph_loader,node_drops,node_lifecycle,node_rpc,node_errors toneMint
class node_repro,node_rest,node_protocol,node_exceptions toneRose
```

## Components

### Graph translation

- [Translator API](https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/web/translator_rest.py): `daliuge-translator/dlg/translator/web/translator_rest.py`
- [Translation pipeline](https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/pipeline.py): `daliuge-translator/dlg/translator/pipeline.py`
- [Graph preparation](https://github.com/sachapoulet/daliuge/tree/master/daliuge-translator/dlg/translator/stages/prepare): `daliuge-translator/dlg/translator/stages/prepare`
- [Graph unrolling](https://github.com/sachapoulet/daliuge/tree/master/daliuge-translator/dlg/translator/stages/unroll): `daliuge-translator/dlg/translator/stages/unroll`
- [Graph partitioning](https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/stages/partition/scheduler.py): `daliuge-translator/dlg/translator/stages/partition/scheduler.py`
- [Physical graph manager](https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/web/pg_manager.py): `daliuge-translator/dlg/translator/web/pg_manager.py`
- [Translator UI](https://github.com/sachapoulet/daliuge/blob/master/daliuge-translator/dlg/translator/web/main.js): `daliuge-translator/dlg/translator/web/main.js`

### Deployment

- [Deployment clients](https://github.com/sachapoulet/daliuge/tree/master/daliuge-engine/dlg/deploy): `daliuge-engine/dlg/deploy`
- [Helm deployment](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/deploy/helm_client.py): `daliuge-engine/dlg/deploy/helm_client.py`
- [SLURM deployment](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/deploy/slurm_client.py): `daliuge-engine/dlg/deploy/slurm_client.py`

### Execution

- [Composite manager](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/composite_manager.py): `daliuge-engine/dlg/manager/composite_manager.py`
- [Island manager](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/composite_manager.py): `daliuge-engine/dlg/manager/composite_manager.py`
- [Node manager](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/node_manager.py): `daliuge-engine/dlg/manager/node_manager.py`
- [Session lifecycle](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/manager/session.py): `daliuge-engine/dlg/manager/session.py`
- [Graph loader](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/graph_loader.py): `daliuge-engine/dlg/graph_loader.py`
- [Data and applications](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/data/io.py): `daliuge-engine/dlg/data/io.py`
- [Data lifecycle](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/lifecycle/dlm.py): `daliuge-engine/dlg/lifecycle/dlm.py`
- [RPC transport](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/rpc.py): `daliuge-engine/dlg/rpc.py`
- [Runtime errors](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/runtime/error_management.py): `daliuge-engine/dlg/runtime/error_management.py`

### Shared services

- [Reproducibility](https://github.com/sachapoulet/daliuge/blob/master/daliuge-common/dlg/common/reproducibility/reproducibility.py): `daliuge-common/dlg/common/reproducibility/reproducibility.py`
- [REST utilities](https://github.com/sachapoulet/daliuge/blob/master/daliuge-common/dlg/restutils.py): `daliuge-common/dlg/restutils.py`
- [Manager protocol](https://github.com/sachapoulet/daliuge/blob/master/daliuge-engine/dlg/ddap_protocol.py): `daliuge-engine/dlg/ddap_protocol.py`
- [Domain exceptions](https://github.com/sachapoulet/daliuge/blob/master/daliuge-common/dlg/exceptions.py): `daliuge-common/dlg/exceptions.py`

### Other components

- **EAGLE user**

## Connections

- EAGLE user → Translator API: submits graph
- Translator UI → Translator API: uses API
- Translator API → Translation pipeline: translates graph
- Translation pipeline → Graph preparation: runs stage
- Translation pipeline → Graph unrolling: runs stage
- Translation pipeline → Graph partitioning: runs stage
- Translation pipeline → Reproducibility: stamps output (evidence: `daliuge-translator/dlg/translator/pipeline.py`)
- Translator API → Physical graph manager: manages graph (evidence: `daliuge-translator/dlg/translator/web/pg_manager.py`)
- Translator API → Graph partitioning: uses scheduler
- Deployment clients → Composite manager: starts managers
- Helm deployment → Composite manager: starts managers (evidence: `daliuge-engine/dlg/deploy/helm_client.py`)
- SLURM deployment → Composite manager: submits job (evidence: `daliuge-engine/dlg/deploy/slurm_client.py`)
- Composite manager → Island manager: coordinates partitions (evidence: `daliuge-engine/dlg/manager/composite_manager.py`)
- Composite manager → Session lifecycle: creates sessions (evidence: `daliuge-engine/dlg/manager/composite_manager.py`)
- Composite manager → Graph loader: loads graph (evidence: `daliuge-engine/dlg/manager/composite_manager.py`)
- Composite manager → Reproducibility: uses metadata (evidence: `daliuge-engine/dlg/manager/composite_manager.py`)
- Island manager → Node manager: dispatches work
- Node manager → Session lifecycle: manages sessions (evidence: `daliuge-engine/dlg/manager/node_manager.py`)
- Node manager → RPC transport: uses transport (evidence: `daliuge-engine/dlg/manager/node_manager.py`)
- Node manager → Data and applications: executes drops (evidence: `daliuge-engine/dlg/manager/node_manager.py`)
- Node manager → Data lifecycle: starts lifecycle (evidence: `daliuge-engine/dlg/manager/node_manager.py`)
- Node manager → Runtime errors: handles failures (evidence: `daliuge-engine/dlg/manager/node_manager.py`)
- Data and applications → Reproducibility: records provenance
- Translator API → REST utilities: serves API
- Composite manager → Manager protocol: uses protocol (evidence: `daliuge-engine/dlg/manager/composite_manager.py`)
- Node manager → Domain exceptions: raises errors (evidence: `daliuge-engine/dlg/manager/node_manager.py`)