# mpaggio/distributed-compute-cluster architecture

> Architecture diagram of the GitHub repository mpaggio/distributed-compute-cluster, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mpaggio/distributed-compute-cluster
- Repository: https://github.com/mpaggio/distributed-compute-cluster

## Overview

This is a Python distributed-compute cluster prototype. Its visible runtime workflow is coordinator-led: a worker connects, requests work, executes an assigned task, reports completion, and requests again. The coordinator tracks task and worker state, monitors heartbeats, and requeues work from failed workers. The README also describes job submission, DAG workflows, replicated coordinators, leader election, and observability, but the shown runtime does not establish those capabilities; no submission interface or coordinator-to-coordinator subsystem appears in the tree. Arrows below reflect excerpted code and SOURCE INDEX references.

Main entry points: [`cluster/worker/worker.py`](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/worker/worker.py) (Worker), [`cluster/coordinator/coordinator.py`](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/coordinator/coordinator.py) (Coordinator), [`cluster/dispatcher/event_dispatcher.py`](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/dispatcher/event_dispatcher.py) (Event dispatcher), [`cluster/serializer/serializer.py`](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/serializer/serializer.py) (JSON serializer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_coordination["Coordination"]
  node_coordinator["Coordinator<br/>[coordinator.py]"]
  node_task_state["Task state<br/>[task.py]"]
  node_task_types["Task states<br/>[task_type.py]"]
end

subgraph group_worker_execution["Worker execution"]
  node_worker["Worker<br/>[worker.py]"]
end

subgraph group_messaging["Messaging"]
  node_event_dispatcher["Event dispatcher"]
  node_event_model["Event model<br/>[event.py]"]
  node_event_types["Event types<br/>[event_type.py]"]
  node_serializer["JSON serializer<br/>[serializer.py]"]
  node_message_sender["Message sender<br/>[message_sender.py]"]
end

node_worker -->|"connects, requests"| node_coordinator
node_worker -->|"sends events"| node_message_sender
node_coordinator -->|"sends assignments"| node_message_sender
node_coordinator -->|"dispatches events"| node_event_dispatcher
node_worker -->|"dispatches received"| node_event_dispatcher
node_event_dispatcher -->|"handles"| node_event_model
node_event_dispatcher -->|"routes by"| node_event_types
node_coordinator -->|"tracks and updates"| node_task_state
node_task_state -->|"uses states"| node_task_types
node_worker -->|"serializes task"| node_task_state
node_message_sender -->|"serializes events"| node_serializer
node_serializer -->|"encodes and decodes"| node_event_model
node_serializer -->|"maps event types"| node_event_types

click node_worker "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/worker/worker.py"
click node_coordinator "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/coordinator/coordinator.py"
click node_task_state "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/task.py"
click node_event_dispatcher "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/dispatcher/event_dispatcher.py"
click node_event_model "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/event.py"
click node_event_types "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/event_type.py"
click node_task_types "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/task_type.py"
click node_serializer "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/serializer/serializer.py"
click node_message_sender "https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/sender/message_sender.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_coordinator,node_task_state,node_task_types toneBlue
class node_worker toneAmber
class node_event_dispatcher,node_event_model,node_event_types,node_serializer,node_message_sender toneMint
```

## Components

### Coordination

- [Coordinator](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/coordinator/coordinator.py): `cluster/coordinator/coordinator.py`
- [Task state](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/task.py): `cluster/common/task.py`
- [Task states](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/task_type.py): `cluster/common/task_type.py`

### Worker execution

- [Worker](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/worker/worker.py): `cluster/worker/worker.py`

### Messaging

- [Event dispatcher](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/dispatcher/event_dispatcher.py): `cluster/dispatcher/event_dispatcher.py`
- [Event model](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/event.py): `cluster/common/event.py`
- [Event types](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/common/event_type.py): `cluster/common/event_type.py`
- [JSON serializer](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/serializer/serializer.py): `cluster/serializer/serializer.py`
- [Message sender](https://github.com/mpaggio/distributed-compute-cluster/blob/main/cluster/sender/message_sender.py): `cluster/sender/message_sender.py`

## Connections

- Worker → Coordinator: connects, requests (evidence: `cluster/worker/worker.py`)
- Worker → Message sender: sends events (evidence: `cluster/worker/worker.py`)
- Coordinator → Message sender: sends assignments (evidence: `cluster/coordinator/coordinator.py`)
- Coordinator → Event dispatcher: dispatches events (evidence: `cluster/coordinator/coordinator.py`)
- Worker → Event dispatcher: dispatches received (evidence: `cluster/worker/worker.py`)
- Event dispatcher → Event model: handles (evidence: `cluster/dispatcher/event_dispatcher.py`)
- Event dispatcher → Event types: routes by (evidence: `cluster/dispatcher/event_dispatcher.py`)
- Coordinator → Task state: tracks and updates (evidence: `cluster/coordinator/coordinator.py`)
- Task state → Task states: uses states (evidence: `cluster/common/task.py`)
- Worker → Task state: serializes task (evidence: `cluster/worker/worker.py`)
- Message sender → JSON serializer: serializes events (evidence: `cluster/sender/message_sender.py`)
- JSON serializer → Event model: encodes and decodes (evidence: `cluster/serializer/serializer.py`)
- JSON serializer → Event types: maps event types (evidence: `cluster/serializer/serializer.py`)