How finos/opengris-scaler works
OpenGRIS Scaler distributes Python task execution across local and remote infrastructure. A client submits tasks; the scheduler tracks task state, applies allocation and scaling policy, and dispatches work through worker managers. Workers fetch serialized inputs and return results, while object storage holds task data. The repository also includes DAG handling, monitoring/UI, and multiple infrastructure adapters. The README documents these capabilities; unsampled subsystem links are shown only where the documented architecture supports them.
Main entry points: task_controller.py Task controller · worker_manager_runner.py Worker manager · scheduler.py Scheduler · task_manager.py Task manager
Components 19
Client API
- Client API
src/scaler/client/client.py - Client agent
src/scaler/client/agent/client_agent.py
Scheduling
- Scheduler
src/scaler/scheduler/scheduler.py - Task controller
src/scaler/scheduler/controllers/task_controller.py - Graph controller
src/scaler/scheduler/controllers/graph_controller.py - Allocation policies
src/scaler/scheduler/controllers/policies - Worker controller
src/scaler/scheduler/controllers/worker_controller.py
Worker Execution
- Worker manager
src/scaler/worker_manager_adapter/worker_manager_runner.py - Capacity adapters
src/scaler/worker_manager_adapter - Worker runtime
src/scaler/worker/worker.py - Task manager
src/scaler/worker_manager_adapter/task_manager.py - Task processor
src/scaler/worker/agent/processor/processor.py
Object Storage
- Object storage server
src/cpp/scaler/object_storage/object_storage_server.cpp - Object manager
src/cpp/scaler/object_storage/object_manager.cpp - Storage connectors
src/scaler/io/ymq_async_object_storage_connector.py
Operations
- YMQ transport
src/scaler/io/ymq - Monitoring UI
src/scaler/ui/app.py - Configuration
src/scaler/config
Other components
- Python caller
Connections 19
- Python caller → Client API: submits tasks
- Client API → Client agent: routes requests
- Client agent → Scheduler: sends tasks
- Scheduler → Task controller: routes events
- Task controller → Graph controller: tracks graph
- Task controller → Allocation policies: requests allocation
- Task controller → Worker controller: dispatches tasks
- Worker controller → Worker manager: requests capacity
- Worker manager → Capacity adapters: provisions workers
- Worker manager → Worker runtime: starts workers
- Worker runtime → Task manager: manages execution
- Task manager → Task processor: executes tasks
- Task manager → Storage connectors: fetches and stores
- Storage connectors → Object storage server: sends requests
- Object storage server → Object manager: reads and writes
- Object storage server → YMQ transport: binds socket
- Storage connectors → YMQ transport: connects through
- Scheduler → Configuration: uses settings
- Worker manager → Configuration: uses settings