How nvidia/nvidia-resiliency-ext works
NVRx is a modular PyTorch resilience toolkit. Training workloads integrate fault detection and restart, asynchronous or local checkpointing, straggler monitoring, and framework callbacks. A separate attribution capability accepts log-analysis requests, runs log and optional flight-recorder analysis, and can publish results. The README documents the core training capabilities; sampled attribution code supports the service-to-analysis and result-posting relationships shown. The tree and excerpts do not establish detailed wiring among most training components, so those relationships are kept at capability level.
Main entry points: analysis_pipeline.py Log/Trace Analysis · fault_tolerance Fault Tolerance · inprocess In-Process Restart · local Local Checkpointing
Components 20
Fault Recovery
- Fault Tolerance
src/nvidia_resiliency_ext/fault_tolerance - In-Process Restart
src/nvidia_resiliency_ext/inprocess - Rank Monitoring
src/nvidia_resiliency_ext/fault_tolerance/rank_monitor_server.py
Checkpointing
- Async Checkpointing
src/nvidia_resiliency_ext/checkpointing/async_ckpt - Local Checkpointing
src/nvidia_resiliency_ext/checkpointing/local - Checkpoint Manager
src/nvidia_resiliency_ext/checkpointing/local/ckpt_managers/base_manager.py - Checkpoint Storage
Monitoring Integration
- Straggler Detection
src/nvidia_resiliency_ext/attribution/straggler - Lightning Callbacks
src/nvidia_resiliency_ext/ptl_resiliency
Attribution
- Analysis Client
src/nvidia_resiliency_ext/attribution/orchestration/http_api.py - Attribution Service
src/nvidia_resiliency_ext/services/attrsvc - Attribution Controller
src/nvidia_resiliency_ext/attribution/legacy_logsage/controller.py - Analysis Engine
src/nvidia_resiliency_ext/attribution/legacy_logsage/analyzer/engine.py - Log/Trace Analysis
src/nvidia_resiliency_ext/attribution/orchestration/analysis_pipeline.py - Training Logs
- Flight Recorder Analysis
src/nvidia_resiliency_ext/attribution/trace_analyzer/trace_analyzer.py - Restart Agent
src/nvidia_resiliency_ext/attribution/restart_agent/pipeline.py - Result Posting
src/nvidia_resiliency_ext/attribution/postprocessing/pipeline.py
Other components
- PyTorch Workload
- Posting Backend
Connections 19
- PyTorch Workload → Fault Tolerance: integrates
- Fault Tolerance → Rank Monitoring: detects hung ranks
- Fault Tolerance → PyTorch Workload: restarts in-job
- PyTorch Workload → In-Process Restart: integrates
- In-Process Restart → PyTorch Workload: recovers failures
- PyTorch Workload → Async Checkpointing: saves asynchronously
- PyTorch Workload → Local Checkpointing: saves locally
- Local Checkpointing → Checkpoint Manager: manages checkpoints
- Checkpoint Manager → Checkpoint Storage: reads/writes
- PyTorch Workload → Straggler Detection: monitors performance
- PyTorch Workload → Lightning Callbacks: integrates callbacks
- Analysis Client → Attribution Service: submits requests
- Attribution Service → Attribution Controller: serves analysis
- Attribution Controller → Analysis Engine: delegates analysis
- Analysis Engine → Log/Trace Analysis: runs pipeline
- Training Logs → Log/Trace Analysis: provides logs
- Log/Trace Analysis → Flight Recorder Analysis: runs optional trace analysis
- Log/Trace Analysis → Result Posting: provides results
- Result Posting → Posting Backend: posts results