# abeenoch/distributedtraining architecture

> Architecture diagram of the GitHub repository abeenoch/distributedtraining, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abeenoch/distributedtraining
- Repository: https://github.com/abeenoch/distributedtraining

## Overview

This repository is a distributed deep-learning training framework. Its main workflow configures a run, loads and shards data, then trains through either a parameter-server worker/RPC path or a DDP path; training produces checkpoints and metrics that support recovery and performance comparison. The README documents baseline training, sync/async aggregation, compression, fault tolerance, and reporting, so those branches are represented. Arrows are limited to relationships shown by excerpts, SOURCE INDEX references, or README statements; unsampled details are omitted.

Main entry points: [`src/worker.py`](https://github.com/abeenoch/distributedtraining/blob/main/src/worker.py) (Training Worker), [`run_final_checkpoint_validation.py`](https://github.com/abeenoch/distributedtraining/blob/main/run_final_checkpoint_validation.py) (Checkpoint Validation), [`src/parameter_server.py`](https://github.com/abeenoch/distributedtraining/blob/main/src/parameter_server.py) (Parameter Server), [`src/metrics.py`](https://github.com/abeenoch/distributedtraining/blob/main/src/metrics.py) (Metrics Logger)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_setup["Configuration and Data"]
  node_config["Run Configuration<br/>[config.py]"]
  node_data["Dataset Loading<br/>[dataset.py]"]
end

subgraph group_training["Training Architectures"]
  node_baseline["Baseline Trainer<br/>[trainer.py]"]
  node_ddp["DDP Trainer<br/>[ddp_trainer.py]"]
  node_baseline_entry["Baseline Runner"]
  node_ddp_entry["DDP Runner"]
end

subgraph group_distributed["Distributed Runtime"]
  node_ps["Parameter Server"]
  node_worker["Training Worker<br/>[worker.py]"]
  node_coordinator["Fault Coordinator<br/>[coordinator.py]"]
  node_rpc["RPC Services<br/>[services.py]"]
  node_serialization["Tensor Serialization<br/>[serialization.py]"]
  node_retry["RPC Retry<br/>[retry.py]"]
  node_compression["Gradient Compression<br/>[compression.py]"]
  node_proto["RPC Protocol<br/>[distributed.proto]"]
  node_ps_entry["Parameter Server Runner"]
end

subgraph group_observability["Metrics and Reporting"]
  node_metrics["Metrics Logger<br/>[metrics.py]"]
  node_performance["Performance Analysis<br/>[performance.py]"]
  node_reporting["Comparison Reports<br/>[reporting.py]"]
  node_validation["Checkpoint Validation"]
end

node_user(("Run Operator"))

node_user -->|"starts"| node_baseline_entry
node_user -->|"starts"| node_ddp_entry
node_user -->|"starts"| node_ps_entry
node_user -->|"starts"| node_validation
node_baseline_entry -->|"loads"| node_config
node_baseline_entry -->|"runs"| node_baseline
node_ddp_entry -->|"loads"| node_config
node_ddp_entry -->|"loads"| node_data
node_ddp_entry -->|"launches"| node_ddp
node_ps_entry -->|"loads"| node_config
node_ps_entry -->|"loads"| node_data
node_ps_entry -->|"starts"| node_coordinator
node_ps_entry -->|"starts"| node_ps
node_validation -->|"tests recovery"| node_coordinator
node_validation -->|"validates"| node_ps
node_validation -->|"validates"| node_ddp
node_validation -->|"compares"| node_baseline
node_baseline -->|"uses"| node_config
node_baseline -->|"loads"| node_data
node_ddp -->|"uses"| node_config
node_ddp -->|"logs"| node_metrics
node_coordinator -->|"manages"| node_worker
node_coordinator -->|"registers"| node_rpc
node_worker -->|"sends gradients"| node_ps
node_worker -->|"calls"| node_rpc
node_worker -->|"serializes tensors"| node_serialization
node_worker -->|"compresses gradients"| node_compression
node_worker -->|"uses shards"| node_data
node_worker -->|"reports"| node_metrics
node_ps -->|"implements"| node_rpc
node_ps -->|"decodes tensors"| node_serialization
node_ps -->|"decompresses gradients"| node_compression
node_rpc -->|"uses messages"| node_proto
node_serialization -->|"encodes messages"| node_proto
node_coordinator -->|"records metrics"| node_metrics
node_metrics -->|"feeds reports"| node_reporting
node_metrics -.->|"supports analysis"| node_performance
node_retry -.->|"retries calls"| node_rpc
node_validation -->|"measures runs"| node_performance
node_validation -->|"collects metrics"| node_metrics

click node_config "https://github.com/abeenoch/distributedtraining/blob/main/src/config.py"
click node_data "https://github.com/abeenoch/distributedtraining/blob/main/src/dataset.py"
click node_baseline "https://github.com/abeenoch/distributedtraining/blob/main/src/trainer.py"
click node_ddp "https://github.com/abeenoch/distributedtraining/blob/main/src/ddp_trainer.py"
click node_ps "https://github.com/abeenoch/distributedtraining/blob/main/src/parameter_server.py"
click node_worker "https://github.com/abeenoch/distributedtraining/blob/main/src/worker.py"
click node_coordinator "https://github.com/abeenoch/distributedtraining/blob/main/src/coordinator.py"
click node_rpc "https://github.com/abeenoch/distributedtraining/blob/main/src/communication/services.py"
click node_serialization "https://github.com/abeenoch/distributedtraining/blob/main/src/communication/serialization.py"
click node_retry "https://github.com/abeenoch/distributedtraining/blob/main/src/communication/retry.py"
click node_compression "https://github.com/abeenoch/distributedtraining/blob/main/src/compression.py"
click node_proto "https://github.com/abeenoch/distributedtraining/blob/main/src/communication/protos/distributed.proto"
click node_metrics "https://github.com/abeenoch/distributedtraining/blob/main/src/metrics.py"
click node_performance "https://github.com/abeenoch/distributedtraining/blob/main/src/performance.py"
click node_reporting "https://github.com/abeenoch/distributedtraining/blob/main/src/reporting.py"
click node_baseline_entry "https://github.com/abeenoch/distributedtraining/blob/main/run_baseline_training.py"
click node_ddp_entry "https://github.com/abeenoch/distributedtraining/blob/main/run_ddp_training.py"
click node_ps_entry "https://github.com/abeenoch/distributedtraining/blob/main/run_parameter_server_training.py"
click node_validation "https://github.com/abeenoch/distributedtraining/blob/main/run_final_checkpoint_validation.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_data toneBlue
class node_baseline,node_ddp,node_baseline_entry,node_ddp_entry toneAmber
class node_ps,node_worker,node_coordinator,node_rpc,node_serialization,node_retry,node_compression,node_proto,node_ps_entry toneMint
class node_metrics,node_performance,node_reporting,node_validation toneRose
class node_user toneIndigo
```

## Components

### Configuration and Data

- [Run Configuration](https://github.com/abeenoch/distributedtraining/blob/main/src/config.py): `src/config.py`
- [Dataset Loading](https://github.com/abeenoch/distributedtraining/blob/main/src/dataset.py): `src/dataset.py`

### Training Architectures

- [Baseline Trainer](https://github.com/abeenoch/distributedtraining/blob/main/src/trainer.py): `src/trainer.py`
- [DDP Trainer](https://github.com/abeenoch/distributedtraining/blob/main/src/ddp_trainer.py): `src/ddp_trainer.py`
- [Baseline Runner](https://github.com/abeenoch/distributedtraining/blob/main/run_baseline_training.py): `run_baseline_training.py`
- [DDP Runner](https://github.com/abeenoch/distributedtraining/blob/main/run_ddp_training.py): `run_ddp_training.py`

### Distributed Runtime

- [Parameter Server](https://github.com/abeenoch/distributedtraining/blob/main/src/parameter_server.py): `src/parameter_server.py`
- [Training Worker](https://github.com/abeenoch/distributedtraining/blob/main/src/worker.py): `src/worker.py`
- [Fault Coordinator](https://github.com/abeenoch/distributedtraining/blob/main/src/coordinator.py): `src/coordinator.py`
- [RPC Services](https://github.com/abeenoch/distributedtraining/blob/main/src/communication/services.py): `src/communication/services.py`
- [Tensor Serialization](https://github.com/abeenoch/distributedtraining/blob/main/src/communication/serialization.py): `src/communication/serialization.py`
- [RPC Retry](https://github.com/abeenoch/distributedtraining/blob/main/src/communication/retry.py): `src/communication/retry.py`
- [Gradient Compression](https://github.com/abeenoch/distributedtraining/blob/main/src/compression.py): `src/compression.py`
- [RPC Protocol](https://github.com/abeenoch/distributedtraining/blob/main/src/communication/protos/distributed.proto): `src/communication/protos/distributed.proto`
- [Parameter Server Runner](https://github.com/abeenoch/distributedtraining/blob/main/run_parameter_server_training.py): `run_parameter_server_training.py`

### Metrics and Reporting

- [Metrics Logger](https://github.com/abeenoch/distributedtraining/blob/main/src/metrics.py): `src/metrics.py`
- [Performance Analysis](https://github.com/abeenoch/distributedtraining/blob/main/src/performance.py): `src/performance.py`
- [Comparison Reports](https://github.com/abeenoch/distributedtraining/blob/main/src/reporting.py): `src/reporting.py`
- [Checkpoint Validation](https://github.com/abeenoch/distributedtraining/blob/main/run_final_checkpoint_validation.py): `run_final_checkpoint_validation.py`

### Other components

- **Run Operator**

## Connections

- Run Operator → Baseline Runner: starts (evidence: `README.md`)
- Run Operator → DDP Runner: starts (evidence: `README.md`)
- Run Operator → Parameter Server Runner: starts (evidence: `README.md`)
- Run Operator → Checkpoint Validation: starts (evidence: `README.md`)
- Baseline Runner → Run Configuration: loads (evidence: `run_baseline_training.py`)
- Baseline Runner → Baseline Trainer: runs (evidence: `run_baseline_training.py`)
- DDP Runner → Run Configuration: loads (evidence: `run_ddp_training.py`)
- DDP Runner → Dataset Loading: loads (evidence: `run_ddp_training.py`)
- DDP Runner → DDP Trainer: launches (evidence: `run_ddp_training.py`)
- Parameter Server Runner → Run Configuration: loads (evidence: `run_parameter_server_training.py`)
- Parameter Server Runner → Dataset Loading: loads (evidence: `run_parameter_server_training.py`)
- Parameter Server Runner → Fault Coordinator: starts (evidence: `run_parameter_server_training.py`)
- Parameter Server Runner → Parameter Server: starts (evidence: `run_parameter_server_training.py`)
- Checkpoint Validation → Fault Coordinator: tests recovery (evidence: `run_final_checkpoint_validation.py`)
- Checkpoint Validation → Parameter Server: validates (evidence: `run_final_checkpoint_validation.py`)
- Checkpoint Validation → DDP Trainer: validates (evidence: `run_final_checkpoint_validation.py`)
- Checkpoint Validation → Baseline Trainer: compares (evidence: `run_final_checkpoint_validation.py`)
- Baseline Trainer → Run Configuration: uses (evidence: `src/trainer.py`)
- Baseline Trainer → Dataset Loading: loads (evidence: `src/trainer.py`)
- DDP Trainer → Run Configuration: uses (evidence: `src/ddp_trainer.py`)
- DDP Trainer → Metrics Logger: logs (evidence: `src/ddp_trainer.py`)
- Fault Coordinator → Training Worker: manages (evidence: `src/coordinator.py`)
- Fault Coordinator → RPC Services: registers (evidence: `src/coordinator.py`)
- Training Worker → Parameter Server: sends gradients (evidence: `src/worker.py`)
- Training Worker → RPC Services: calls (evidence: `src/worker.py`)
- Training Worker → Tensor Serialization: serializes tensors (evidence: `src/worker.py`)
- Training Worker → Gradient Compression: compresses gradients (evidence: `src/worker.py`)
- Training Worker → Dataset Loading: uses shards (evidence: `src/worker.py`)
- Training Worker → Metrics Logger: reports (evidence: `src/worker.py`)
- Parameter Server → RPC Services: implements (evidence: `src/parameter_server.py`)
- Parameter Server → Tensor Serialization: decodes tensors (evidence: `src/parameter_server.py`)
- Parameter Server → Gradient Compression: decompresses gradients (evidence: `src/parameter_server.py`)
- RPC Services → RPC Protocol: uses messages (evidence: `src/communication/services.py`)
- Tensor Serialization → RPC Protocol: encodes messages (evidence: `src/communication/serialization.py`)
- Fault Coordinator → Metrics Logger: records metrics (evidence: `src/coordinator.py`)
- Metrics Logger → Comparison Reports: feeds reports (evidence: `generate_performance_report.py`)
- Metrics Logger → Performance Analysis: supports analysis
- RPC Retry → RPC Services: retries calls
- Checkpoint Validation → Performance Analysis: measures runs (evidence: `run_final_checkpoint_validation.py`)
- Checkpoint Validation → Metrics Logger: collects metrics (evidence: `run_final_checkpoint_validation.py`)