# finos/opengris-scaler architecture

> Architecture diagram of the GitHub repository finos/opengris-scaler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/finos/opengris-scaler
- Repository: https://github.com/finos/opengris-scaler

## Overview

OpenGRIS Scaler distributes Python task execution across local and remote infrastructure. A client submits tasks; the scheduler tracks task state, applies allocation and scaling policy, and dispatches work through worker managers. Workers fetch serialized inputs and return results, while object storage holds task data. The repository also includes DAG handling, monitoring/UI, and multiple infrastructure adapters. The README documents these capabilities; unsampled subsystem links are shown only where the documented architecture supports them.

Main entry points: [`src/scaler/scheduler/controllers/task_controller.py`](https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/controllers/task_controller.py) (Task controller), [`src/scaler/worker_manager_adapter/worker_manager_runner.py`](https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker_manager_adapter/worker_manager_runner.py) (Worker manager), [`src/scaler/scheduler/scheduler.py`](https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/scheduler.py) (Scheduler), [`src/scaler/worker_manager_adapter/task_manager.py`](https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker_manager_adapter/task_manager.py) (Task manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Client API"]
  node_client_api["Client API<br/>[client.py]"]
  node_client_agent["Client agent<br/>[client_agent.py]"]
end

subgraph group_coordination["Scheduling"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_task_control["Task controller<br/>[task_controller.py]"]
  node_graph_control["Graph controller"]
  node_policies["Allocation policies"]
  node_worker_control["Worker controller"]
end

subgraph group_execution["Worker Execution"]
  node_worker_manager["Worker manager"]
  node_capacity_adapters["Capacity adapters"]
  node_worker["Worker runtime<br/>[worker.py]"]
  node_task_manager["Task manager<br/>[task_manager.py]"]
  node_processor["Task processor<br/>[processor.py]"]
end

subgraph group_storage["Object Storage"]
  node_object_storage["Object storage server"]
  node_object_manager[("Object manager<br/>[object_manager.cpp]")]
  node_storage_connector["Storage connectors"]
end

subgraph group_operations["Operations"]
  node_transport["YMQ transport"]
  node_ui["Monitoring UI<br/>[app.py]"]
  node_config["Configuration"]
end

node_user(("Python caller"))

node_user -->|"submits tasks"| node_client_api
node_client_api -->|"routes requests"| node_client_agent
node_client_agent -->|"sends tasks"| node_scheduler
node_scheduler -->|"routes events"| node_task_control
node_task_control -->|"tracks graph"| node_graph_control
node_task_control -->|"requests allocation"| node_policies
node_task_control -->|"dispatches tasks"| node_worker_control
node_worker_control -->|"requests capacity"| node_worker_manager
node_worker_manager -->|"provisions workers"| node_capacity_adapters
node_worker_manager -->|"starts workers"| node_worker
node_worker -->|"manages execution"| node_task_manager
node_task_manager -->|"executes tasks"| node_processor
node_task_manager -->|"fetches and stores"| node_storage_connector
node_storage_connector -->|"sends requests"| node_object_storage
node_object_storage -->|"reads and writes"| node_object_manager
node_object_storage -->|"binds socket"| node_transport
node_storage_connector -->|"connects through"| node_transport
node_scheduler -->|"uses settings"| node_config
node_worker_manager -->|"uses settings"| node_config

click node_client_api "https://github.com/finos/opengris-scaler/blob/main/src/scaler/client/client.py"
click node_client_agent "https://github.com/finos/opengris-scaler/blob/main/src/scaler/client/agent/client_agent.py"
click node_scheduler "https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/scheduler.py"
click node_task_control "https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/controllers/task_controller.py"
click node_graph_control "https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/controllers/graph_controller.py"
click node_policies "https://github.com/finos/opengris-scaler/tree/main/src/scaler/scheduler/controllers/policies"
click node_worker_control "https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/controllers/worker_controller.py"
click node_worker_manager "https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker_manager_adapter/worker_manager_runner.py"
click node_capacity_adapters "https://github.com/finos/opengris-scaler/tree/main/src/scaler/worker_manager_adapter"
click node_worker "https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker/worker.py"
click node_task_manager "https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker_manager_adapter/task_manager.py"
click node_processor "https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker/agent/processor/processor.py"
click node_object_storage "https://github.com/finos/opengris-scaler/blob/main/src/cpp/scaler/object_storage/object_storage_server.cpp"
click node_object_manager "https://github.com/finos/opengris-scaler/blob/main/src/cpp/scaler/object_storage/object_manager.cpp"
click node_storage_connector "https://github.com/finos/opengris-scaler/blob/main/src/scaler/io/ymq_async_object_storage_connector.py"
click node_transport "https://github.com/finos/opengris-scaler/tree/main/src/scaler/io/ymq"
click node_ui "https://github.com/finos/opengris-scaler/blob/main/src/scaler/ui/app.py"
click node_config "https://github.com/finos/opengris-scaler/tree/main/src/scaler/config"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client_api,node_client_agent toneBlue
class node_scheduler,node_task_control,node_graph_control,node_policies,node_worker_control toneAmber
class node_worker_manager,node_capacity_adapters,node_worker,node_task_manager,node_processor toneMint
class node_object_storage,node_object_manager,node_storage_connector toneRose
class node_transport,node_ui,node_config,node_user toneIndigo
```

## Components

### Client API

- [Client API](https://github.com/finos/opengris-scaler/blob/main/src/scaler/client/client.py): `src/scaler/client/client.py`
- [Client agent](https://github.com/finos/opengris-scaler/blob/main/src/scaler/client/agent/client_agent.py): `src/scaler/client/agent/client_agent.py`

### Scheduling

- [Scheduler](https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/scheduler.py): `src/scaler/scheduler/scheduler.py`
- [Task controller](https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/controllers/task_controller.py): `src/scaler/scheduler/controllers/task_controller.py`
- [Graph controller](https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/controllers/graph_controller.py): `src/scaler/scheduler/controllers/graph_controller.py`
- [Allocation policies](https://github.com/finos/opengris-scaler/tree/main/src/scaler/scheduler/controllers/policies): `src/scaler/scheduler/controllers/policies`
- [Worker controller](https://github.com/finos/opengris-scaler/blob/main/src/scaler/scheduler/controllers/worker_controller.py): `src/scaler/scheduler/controllers/worker_controller.py`

### Worker Execution

- [Worker manager](https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker_manager_adapter/worker_manager_runner.py): `src/scaler/worker_manager_adapter/worker_manager_runner.py`
- [Capacity adapters](https://github.com/finos/opengris-scaler/tree/main/src/scaler/worker_manager_adapter): `src/scaler/worker_manager_adapter`
- [Worker runtime](https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker/worker.py): `src/scaler/worker/worker.py`
- [Task manager](https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker_manager_adapter/task_manager.py): `src/scaler/worker_manager_adapter/task_manager.py`
- [Task processor](https://github.com/finos/opengris-scaler/blob/main/src/scaler/worker/agent/processor/processor.py): `src/scaler/worker/agent/processor/processor.py`

### Object Storage

- [Object storage server](https://github.com/finos/opengris-scaler/blob/main/src/cpp/scaler/object_storage/object_storage_server.cpp): `src/cpp/scaler/object_storage/object_storage_server.cpp`
- [Object manager](https://github.com/finos/opengris-scaler/blob/main/src/cpp/scaler/object_storage/object_manager.cpp): `src/cpp/scaler/object_storage/object_manager.cpp`
- [Storage connectors](https://github.com/finos/opengris-scaler/blob/main/src/scaler/io/ymq_async_object_storage_connector.py): `src/scaler/io/ymq_async_object_storage_connector.py`

### Operations

- [YMQ transport](https://github.com/finos/opengris-scaler/tree/main/src/scaler/io/ymq): `src/scaler/io/ymq`
- [Monitoring UI](https://github.com/finos/opengris-scaler/blob/main/src/scaler/ui/app.py): `src/scaler/ui/app.py`
- [Configuration](https://github.com/finos/opengris-scaler/tree/main/src/scaler/config): `src/scaler/config`

### Other components

- **Python caller**

## Connections

- Python caller → Client API: submits tasks
- Client API → Client agent: routes requests
- Client agent → Scheduler: sends tasks
- Scheduler → Task controller: routes events
- Task controller → Graph controller: tracks graph
- Task controller → Allocation policies: requests allocation
- Task controller → Worker controller: dispatches tasks
- Worker controller → Worker manager: requests capacity
- Worker manager → Capacity adapters: provisions workers
- Worker manager → Worker runtime: starts workers
- Worker runtime → Task manager: manages execution
- Task manager → Task processor: executes tasks
- Task manager → Storage connectors: fetches and stores
- Storage connectors → Object storage server: sends requests
- Object storage server → Object manager: reads and writes
- Object storage server → YMQ transport: binds socket
- Storage connectors → YMQ transport: connects through
- Scheduler → Configuration: uses settings
- Worker manager → Configuration: uses settings