# nvidia/nvidia-resiliency-ext architecture

> Architecture diagram of the GitHub repository nvidia/nvidia-resiliency-ext, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/nvidia-resiliency-ext
- Repository: https://github.com/nvidia/nvidia-resiliency-ext

## Overview

NVRx is a modular PyTorch resilience toolkit. Training workloads integrate fault detection and restart, asynchronous or local checkpointing, straggler monitoring, and framework callbacks. A separate attribution capability accepts log-analysis requests, runs log and optional flight-recorder analysis, and can publish results. The README documents the core training capabilities; sampled attribution code supports the service-to-analysis and result-posting relationships shown. The tree and excerpts do not establish detailed wiring among most training components, so those relationships are kept at capability level.

Main entry points: [`src/nvidia_resiliency_ext/attribution/orchestration/analysis_pipeline.py`](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/orchestration/analysis_pipeline.py) (Log/Trace Analysis), [`src/nvidia_resiliency_ext/fault_tolerance`](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/fault_tolerance) (Fault Tolerance), [`src/nvidia_resiliency_ext/inprocess`](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/inprocess) (In-Process Restart), [`src/nvidia_resiliency_ext/checkpointing/local`](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/checkpointing/local) (Local Checkpointing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_recovery["Fault Recovery"]
  node_fault_tolerance["Fault Tolerance"]
  node_inprocess["In-Process Restart"]
  node_rank_monitor["Rank Monitoring"]
end

subgraph group_checkpoint["Checkpointing"]
  node_async_checkpoint["Async Checkpointing"]
  node_local_checkpoint["Local Checkpointing"]
  node_checkpoint_manager["Checkpoint Manager<br/>[base_manager.py]"]
  node_checkpoint_storage[("Checkpoint Storage")]
end

subgraph group_monitoring["Monitoring Integration"]
  node_straggler["Straggler Detection"]
  node_ptl["Lightning Callbacks"]
end

subgraph group_attribution["Attribution"]
  node_attr_client["Analysis Client<br/>[http_api.py]"]
  node_attr_service["Attribution Service"]
  node_controller["Attribution Controller<br/>[controller.py]"]
  node_analyzer["Analysis Engine<br/>[engine.py]"]
  node_analysis_pipeline["Log/Trace Analysis"]
  node_log_input["Training Logs"]
  node_trace_analysis["Flight Recorder Analysis<br/>[trace_analyzer.py]"]
  node_restart_agent["Restart Agent<br/>[pipeline.py]"]
  node_postprocessing["Result Posting<br/>[pipeline.py]"]
end

node_workload(("PyTorch Workload"))
node_result_backend(("Posting Backend"))

node_workload -->|"integrates"| node_fault_tolerance
node_fault_tolerance -->|"detects hung ranks"| node_rank_monitor
node_fault_tolerance -->|"restarts in-job"| node_workload
node_workload -->|"integrates"| node_inprocess
node_inprocess -->|"recovers failures"| node_workload
node_workload -->|"saves asynchronously"| node_async_checkpoint
node_workload -->|"saves locally"| node_local_checkpoint
node_local_checkpoint -->|"manages checkpoints"| node_checkpoint_manager
node_checkpoint_manager -->|"reads/writes"| node_checkpoint_storage
node_workload -->|"monitors performance"| node_straggler
node_workload -->|"integrates callbacks"| node_ptl
node_attr_client -->|"submits requests"| node_attr_service
node_attr_service -->|"serves analysis"| node_controller
node_controller -->|"delegates analysis"| node_analyzer
node_analyzer -->|"runs pipeline"| node_analysis_pipeline
node_log_input -->|"provides logs"| node_analysis_pipeline
node_analysis_pipeline -.->|"runs optional trace analysis"| node_trace_analysis
node_analysis_pipeline -->|"provides results"| node_postprocessing
node_postprocessing -->|"posts results"| node_result_backend

click node_fault_tolerance "https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/fault_tolerance"
click node_inprocess "https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/inprocess"
click node_rank_monitor "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/fault_tolerance/rank_monitor_server.py"
click node_async_checkpoint "https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/checkpointing/async_ckpt"
click node_local_checkpoint "https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/checkpointing/local"
click node_checkpoint_manager "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/checkpointing/local/ckpt_managers/base_manager.py"
click node_straggler "https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/attribution/straggler"
click node_ptl "https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/ptl_resiliency"
click node_attr_client "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/orchestration/http_api.py"
click node_attr_service "https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/services/attrsvc"
click node_controller "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/legacy_logsage/controller.py"
click node_analyzer "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/legacy_logsage/analyzer/engine.py"
click node_analysis_pipeline "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/orchestration/analysis_pipeline.py"
click node_trace_analysis "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/trace_analyzer/trace_analyzer.py"
click node_restart_agent "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/restart_agent/pipeline.py"
click node_postprocessing "https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/postprocessing/pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fault_tolerance,node_inprocess,node_rank_monitor toneBlue
class node_async_checkpoint,node_local_checkpoint,node_checkpoint_manager,node_checkpoint_storage toneAmber
class node_straggler,node_ptl toneMint
class node_attr_client,node_attr_service,node_controller,node_analyzer,node_analysis_pipeline,node_log_input,node_trace_analysis,node_restart_agent,node_postprocessing toneRose
class node_workload,node_result_backend toneIndigo
```

## Components

### Fault Recovery

- [Fault Tolerance](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/fault_tolerance): `src/nvidia_resiliency_ext/fault_tolerance`
- [In-Process Restart](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/inprocess): `src/nvidia_resiliency_ext/inprocess`
- [Rank Monitoring](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/fault_tolerance/rank_monitor_server.py): `src/nvidia_resiliency_ext/fault_tolerance/rank_monitor_server.py`

### Checkpointing

- [Async Checkpointing](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/checkpointing/async_ckpt): `src/nvidia_resiliency_ext/checkpointing/async_ckpt`
- [Local Checkpointing](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/checkpointing/local): `src/nvidia_resiliency_ext/checkpointing/local`
- [Checkpoint Manager](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/checkpointing/local/ckpt_managers/base_manager.py): `src/nvidia_resiliency_ext/checkpointing/local/ckpt_managers/base_manager.py`
- **Checkpoint Storage**

### Monitoring Integration

- [Straggler Detection](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/attribution/straggler): `src/nvidia_resiliency_ext/attribution/straggler`
- [Lightning Callbacks](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/ptl_resiliency): `src/nvidia_resiliency_ext/ptl_resiliency`

### Attribution

- [Analysis Client](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/orchestration/http_api.py): `src/nvidia_resiliency_ext/attribution/orchestration/http_api.py`
- [Attribution Service](https://github.com/nvidia/nvidia-resiliency-ext/tree/main/src/nvidia_resiliency_ext/services/attrsvc): `src/nvidia_resiliency_ext/services/attrsvc`
- [Attribution Controller](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/legacy_logsage/controller.py): `src/nvidia_resiliency_ext/attribution/legacy_logsage/controller.py`
- [Analysis Engine](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/legacy_logsage/analyzer/engine.py): `src/nvidia_resiliency_ext/attribution/legacy_logsage/analyzer/engine.py`
- [Log/Trace Analysis](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/orchestration/analysis_pipeline.py): `src/nvidia_resiliency_ext/attribution/orchestration/analysis_pipeline.py`
- **Training Logs**
- [Flight Recorder Analysis](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/trace_analyzer/trace_analyzer.py): `src/nvidia_resiliency_ext/attribution/trace_analyzer/trace_analyzer.py`
- [Restart Agent](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/restart_agent/pipeline.py): `src/nvidia_resiliency_ext/attribution/restart_agent/pipeline.py`
- [Result Posting](https://github.com/nvidia/nvidia-resiliency-ext/blob/main/src/nvidia_resiliency_ext/attribution/postprocessing/pipeline.py): `src/nvidia_resiliency_ext/attribution/postprocessing/pipeline.py`

### Other components

- **PyTorch Workload**
- **Posting Backend**

## Connections

- PyTorch Workload → Fault Tolerance: integrates
- Fault Tolerance → Rank Monitoring: detects hung ranks
- Fault Tolerance → PyTorch Workload: restarts in-job
- PyTorch Workload → In-Process Restart: integrates
- In-Process Restart → PyTorch Workload: recovers failures
- PyTorch Workload → Async Checkpointing: saves asynchronously
- PyTorch Workload → Local Checkpointing: saves locally
- Local Checkpointing → Checkpoint Manager: manages checkpoints
- Checkpoint Manager → Checkpoint Storage: reads/writes
- PyTorch Workload → Straggler Detection: monitors performance
- PyTorch Workload → Lightning Callbacks: integrates callbacks
- Analysis Client → Attribution Service: submits requests
- Attribution Service → Attribution Controller: serves analysis
- Attribution Controller → Analysis Engine: delegates analysis
- Analysis Engine → Log/Trace Analysis: runs pipeline
- Training Logs → Log/Trace Analysis: provides logs
- Log/Trace Analysis → Flight Recorder Analysis: runs optional trace analysis
- Log/Trace Analysis → Result Posting: provides results
- Result Posting → Posting Backend: posts results