How mpaggio/distributed-compute-cluster works
This is a Python distributed-compute cluster prototype. Its visible runtime workflow is coordinator-led: a worker connects, requests work, executes an assigned task, reports completion, and requests again. The coordinator tracks task and worker state, monitors heartbeats, and requeues work from failed workers. The README also describes job submission, DAG workflows, replicated coordinators, leader election, and observability, but the shown runtime does not establish those capabilities; no submission interface or coordinator-to-coordinator subsystem appears in the tree. Arrows below reflect excerpted code and SOURCE INDEX references.
Main entry points: worker.py Worker · coordinator.py Coordinator · event_dispatcher.py Event dispatcher · serializer.py JSON serializer
Components 9
Coordination
- Coordinator
cluster/coordinator/coordinator.py - Task state
cluster/common/task.py - Task states
cluster/common/task_type.py
Worker execution
- Worker
cluster/worker/worker.py
Messaging
- Event dispatcher
cluster/dispatcher/event_dispatcher.py - Event model
cluster/common/event.py - Event types
cluster/common/event_type.py - JSON serializer
cluster/serializer/serializer.py - Message sender
cluster/sender/message_sender.py
Connections 13
- Worker → Coordinator: connects, requests
cluster/worker/worker.py - Worker → Message sender: sends events
cluster/worker/worker.py - Coordinator → Message sender: sends assignments
cluster/coordinator/coordinator.py - Coordinator → Event dispatcher: dispatches events
cluster/coordinator/coordinator.py - Worker → Event dispatcher: dispatches received
cluster/worker/worker.py - Event dispatcher → Event model: handles
cluster/dispatcher/event_dispatcher.py - Event dispatcher → Event types: routes by
cluster/dispatcher/event_dispatcher.py - Coordinator → Task state: tracks and updates
cluster/coordinator/coordinator.py - Task state → Task states: uses states
cluster/common/task.py - Worker → Task state: serializes task
cluster/worker/worker.py - Message sender → JSON serializer: serializes events
cluster/sender/message_sender.py - JSON serializer → Event model: encodes and decodes
cluster/serializer/serializer.py - JSON serializer → Event types: maps event types
cluster/serializer/serializer.py