How hardy-c/adacomp works
AdaComp distributes MNIST stochastic-gradient training across workers and a parameter server, with an optional supervisor that periodically evaluates the shared model. The runtime starts by dispatching a node role; worker data is loaded locally, gradients are computed and compressed, and the PS applies updates while tracking staleness. The CNN and TensorFlow update operations are shared internal layers. The sampled code supports the principal paths below; connections between distributed processes are documented and visible in socket handling, but the full protocol is not independently traced end-to-end.
Main entry points: Worker.py Worker trainer · PS.py Parameter server · Supervisor.py Supervisor evaluator · main.py Role dispatcher
Components 13
Runtime roles
- Role dispatcher
main.py - Worker trainer
Worker.py - Parameter server
PS.py - Supervisor evaluator
Supervisor.py
Training and model
- CNN model
Model_DNN.py - Gradient computation
Worker.py - Sparse updates
Tf_op.py - Shared parameters
Model_DNN.py
Parameter coordination
- Message encoding
Communication.py - Staleness scaling
PS.py
Data and evaluation
- MNIST input
mnist.py - Accuracy and loss
Supervisor.py
Other components
- Node operator
Connections 20
- Node operator → Role dispatcher: starts role
README.md - Role dispatcher → MNIST input: loads data
main.py - Role dispatcher → Worker trainer: dispatches
main.py - Role dispatcher → Supervisor evaluator: dispatches
main.py - Role dispatcher → Parameter server: dispatches
main.py - Worker trainer → CNN model: builds model
Worker.py - Worker trainer → Gradient computation: computes gradients
Worker.py - Gradient computation → Message encoding: encodes updates
Worker.py - Worker trainer → Message encoding: requests parameters
Worker.py - Worker trainer → Parameter server: sends updates
README.md - Parameter server → Message encoding: decodes messages
PS.py - Parameter server → Staleness scaling: scales gradients
PS.py - Parameter server → Sparse updates: builds update ops
PS.py - Sparse updates → Shared parameters: updates parameters
Tf_op.py - Parameter server → Worker trainer: returns parameters
README.md - Supervisor evaluator → CNN model: builds model
Supervisor.py - Supervisor evaluator → Message encoding: requests parameters
Supervisor.py - Supervisor evaluator → Sparse updates: builds fetch ops
Supervisor.py - Supervisor evaluator → Accuracy and loss: computes metrics
Supervisor.py - MNIST input → Supervisor evaluator: provides test batches
main.py