How pytorch/torchft works
Main Components
• Lighthouse Server – This is the central coordination entity used to manage replica groups and coordinate fault tolerance actions. It is implemented in Rust as seen from the “src/lighthouse.rs” file and is invoked externally (e.g., via the CLI “torchft_lighthouse”).
• Fault Tolerance Library (Manager and associated modules) – This library implements key routines (in torchft directory) for handling checkpointing, retries, and error recovery on a per-step basis. It includes modules such as:
• Coordination (torchft/coordination.py)
• Manager (torchft/manager.py)
• Process Group (torchft/process_group.py)
• Checkpointing (torchft/checkpointing/*)
• Data handling, futures, and optim (torchft/data.py, torchft/futures.py, torchft/optim.py)
• Training Worker and PyTorch Training Loop – The integration into the user’s training loop is showcased in the “train_ddp.py” example. Workers are enhanced for fault tolerance by wrapping model and optimizer updates with torchft’s manager and fault tolerance logic.
• Parameter Server (optional component) – There is also a fault tolerant parameter server implementation (torchft/parameter_server.py) that follows a similar reconfigurable ProcessGroup model.
• Network Communication & Protobuf Interfaces – The “proto/torchft.proto” indicates use of Protocol Buffers for defining network messages to facilitate communication between the lighthouse server, managers, and the training workers. There are specific modules in torchft for HTTP transport and PG (process group) transport.
• Rust-Python Integration – With build scripts (Cargo.toml, build.rs) and the usage of pyo3/maturin, there is a strong interoperation element between Rust backend components and Python’s torchft library.
Relationships and Interactions
• When using the Parameter Server mode, a separate flow is seen where the workers interact with the “Parameter Server” instead of coordinating directly with a lighthouse.
Architectural Patterns and Technical Stacks
• That this is a hybrid system design with a clear separation between:
• The coordination layer implemented in Rust (for performance and reliability in network communication)
• The fault tolerance logic integrated within the PyTorch training loops (Python modules)
• The service-oriented and loosely coupled structure supports pluggable modules (e.g., different transport protocols: HTTP vs. PG).
• The use of Protocol Buffers to define communication interfaces reinforces clear contracts between components.
• Fault tolerance is achieved on a per-step granularity, meaning that the system is architected to enable minimal downtime – represented as fast communication loops between Manager and Lighthouse Server.
Final Checklist for the Principal Software Engineer
▪ Rust-based components (e.g., Lighthouse Server, underlying network transport)
▪ Python-based components (e.g., Manager, training loop integration)
▪ External services (e.g., External client calls or parameter server)
• Interactions (such as “sends heartbeat”, “requests recovery”, “shares checkpoint data”).