# amazon-far/residual-offpolicy-rl architecture

> Architecture diagram of the GitHub repository amazon-far/residual-offpolicy-rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amazon-far/residual-offpolicy-rl
- Repository: https://github.com/amazon-far/residual-offpolicy-rl

## Overview

ResFiT is a robotics-learning codebase for training behavior-cloning (BC) policies and fine-tuning them with residual off-policy reinforcement learning. The main workflow is BC training on DexMG demonstrations, configuring the resulting policy checkpoint, then residual-TD3 training and evaluation in DexMG environments. The repository also includes RLPD training, ACT and diffusion policy implementations, and a Robomimic-to-LeRobot dataset converter. Edges follow README statements or source-index references; unsampled implementation details and unverified links are omitted.

Main entry points: [`resfit/rl_finetuning/scripts/train_residual_td3.py`](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/scripts/train_residual_td3.py) (Residual TD3 Training), [`resfit/lerobot/scripts/train_bc_dexmg.py`](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/scripts/train_bc_dexmg.py) (BC Training), [`resfit/lerobot/policies/factory.py`](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/factory.py) (Policy Factory), [`resfit/rl_finetuning/off_policy/rl/q_agent.py`](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/rl/q_agent.py) (Q Agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Demonstration Data"]
  node_dataset_converter["Dataset Conversion"]
end

subgraph group_bc["BC Policies"]
  node_bc_training["BC Training<br/>[train_bc_dexmg.py]"]
  node_policy_config["Policy Configurations<br/>[policies.py]"]
  node_act_config["ACT Configuration"]
  node_act_model["ACT Policy<br/>[modeling_act.py]"]
  node_diff_config["Diffusion Configuration"]
  node_diff_model["Diffusion Policy"]
  node_policy_factory["Policy Factory<br/>[factory.py]"]
  node_pretrained["Pretrained Policies<br/>[pretrained.py]"]
  node_policy_loader["Policy Checkpoint Loader<br/>[load_policy.py]"]
end

subgraph group_rl["Residual RL"]
  node_residual_config["Residual TD3 Config<br/>[residual_td3.py]"]
  node_rlpd_config["RLPD Config<br/>[rlpd.py]"]
  node_rl_encoder["Visual Encoder<br/>[encoder.py]"]
  node_actor["Actor Network<br/>[actor.py]"]
  node_critic["Critic Network<br/>[critic.py]"]
  node_agent["Q Agent<br/>[q_agent.py]"]
  node_residual_training["Residual TD3 Training"]
  node_rlpd_training["RLPD Training"]
end

subgraph group_support["Environment and Evaluation"]
  node_dexmg["DexMG Environments<br/>[dexmg.py]"]
  node_residual_wrapper["Residual Environment"]
  node_evaluation["DexMG Evaluation<br/>[evaluate_dexmg.py]"]
end

node_engineer(("Researcher"))
node_demos[("Demonstrations")]
node_hf_data[("Hugging Face Hub")]
node_wandb[("Weights &amp; Biases")]

node_engineer -->|"starts training"| node_bc_training
node_demos -->|"converts data"| node_dataset_converter
node_bc_training -->|"evaluates in"| node_dexmg
node_bc_training -->|"creates policy"| node_policy_factory
node_policy_factory -->|"builds config"| node_policy_config
node_policy_factory -->|"selects config"| node_act_config
node_policy_factory -->|"selects config"| node_diff_config
node_policy_factory -->|"instantiates"| node_act_model
node_policy_factory -->|"instantiates"| node_diff_model
node_act_model -->|"uses config"| node_act_config
node_diff_model -->|"uses config"| node_diff_config
node_act_model -->|"extends policy base"| node_pretrained
node_diff_model -->|"extends policy base"| node_pretrained
node_bc_training -->|"saves policy"| node_pretrained
node_bc_training -->|"loads policy"| node_policy_loader
node_dataset_converter -.->|"publishes dataset"| node_hf_data
node_engineer -->|"starts fine-tuning"| node_residual_training
node_bc_training -->|"checkpoint configured"| node_residual_config
node_residual_training -->|"loads config"| node_residual_config
node_residual_training -->|"creates environment"| node_dexmg
node_residual_training -->|"loads BC policy"| node_policy_loader
node_residual_training -->|"wraps environment"| node_residual_wrapper
node_residual_wrapper -->|"wraps environment"| node_dexmg
node_residual_wrapper -->|"uses BC policy"| node_act_model
node_residual_training -->|"constructs agent"| node_agent
node_agent -->|"uses encoder"| node_rl_encoder
node_agent -->|"creates actor"| node_actor
node_agent -->|"creates critic"| node_critic
node_residual_training -->|"evaluates policy"| node_evaluation
node_evaluation -->|"runs episodes"| node_dexmg
node_residual_training -->|"logs artifacts"| node_wandb
node_engineer -.->|"starts RLPD"| node_rlpd_training
node_rlpd_training -->|"loads config"| node_rlpd_config
node_rlpd_training -->|"constructs agent"| node_agent
node_rlpd_training -->|"creates environment"| node_dexmg

click node_dataset_converter "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/dataset/convert_robomimic_to_lerobot.py"
click node_bc_training "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/scripts/train_bc_dexmg.py"
click node_policy_config "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/configs/policies.py"
click node_act_config "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/act/configuration_act.py"
click node_act_model "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/act/modeling_act.py"
click node_diff_config "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/diffusion/configuration_diffusion.py"
click node_diff_model "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/diffusion/modeling_diffusion.py"
click node_policy_factory "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/factory.py"
click node_pretrained "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/pretrained.py"
click node_policy_loader "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/utils/load_policy.py"
click node_dexmg "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/dexmg/environments/dexmg.py"
click node_residual_config "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/config/residual_td3.py"
click node_rlpd_config "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/config/rlpd.py"
click node_residual_wrapper "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/wrappers/residual_env_wrapper.py"
click node_rl_encoder "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/networks/encoder.py"
click node_actor "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/rl/actor.py"
click node_critic "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/rl/critic.py"
click node_agent "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/rl/q_agent.py"
click node_residual_training "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/scripts/train_residual_td3.py"
click node_rlpd_training "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/scripts/train_rlpd_dexmg.py"
click node_evaluation "https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/utils/evaluate_dexmg.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset_converter toneBlue
class node_bc_training,node_policy_config,node_act_config,node_act_model,node_diff_config,node_diff_model,node_policy_factory,node_pretrained,node_policy_loader,node_demos,node_hf_data,node_wandb toneAmber
class node_residual_config,node_rlpd_config,node_rl_encoder,node_actor,node_critic,node_agent,node_residual_training,node_rlpd_training toneMint
class node_dexmg,node_residual_wrapper,node_evaluation toneRose
class node_engineer toneIndigo
```

## Components

### Demonstration Data

- [Dataset Conversion](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/dataset/convert_robomimic_to_lerobot.py): `resfit/lerobot/dataset/convert_robomimic_to_lerobot.py`

### BC Policies

- [BC Training](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/scripts/train_bc_dexmg.py): `resfit/lerobot/scripts/train_bc_dexmg.py`
- [Policy Configurations](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/configs/policies.py): `resfit/lerobot/configs/policies.py`
- [ACT Configuration](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/act/configuration_act.py): `resfit/lerobot/policies/act/configuration_act.py`
- [ACT Policy](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/act/modeling_act.py): `resfit/lerobot/policies/act/modeling_act.py`
- [Diffusion Configuration](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/diffusion/configuration_diffusion.py): `resfit/lerobot/policies/diffusion/configuration_diffusion.py`
- [Diffusion Policy](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/diffusion/modeling_diffusion.py): `resfit/lerobot/policies/diffusion/modeling_diffusion.py`
- [Policy Factory](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/factory.py): `resfit/lerobot/policies/factory.py`
- [Pretrained Policies](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/policies/pretrained.py): `resfit/lerobot/policies/pretrained.py`
- [Policy Checkpoint Loader](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/lerobot/utils/load_policy.py): `resfit/lerobot/utils/load_policy.py`

### Residual RL

- [Residual TD3 Config](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/config/residual_td3.py): `resfit/rl_finetuning/config/residual_td3.py`
- [RLPD Config](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/config/rlpd.py): `resfit/rl_finetuning/config/rlpd.py`
- [Visual Encoder](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/networks/encoder.py): `resfit/rl_finetuning/off_policy/networks/encoder.py`
- [Actor Network](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/rl/actor.py): `resfit/rl_finetuning/off_policy/rl/actor.py`
- [Critic Network](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/rl/critic.py): `resfit/rl_finetuning/off_policy/rl/critic.py`
- [Q Agent](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/off_policy/rl/q_agent.py): `resfit/rl_finetuning/off_policy/rl/q_agent.py`
- [Residual TD3 Training](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/scripts/train_residual_td3.py): `resfit/rl_finetuning/scripts/train_residual_td3.py`
- [RLPD Training](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/scripts/train_rlpd_dexmg.py): `resfit/rl_finetuning/scripts/train_rlpd_dexmg.py`

### Environment and Evaluation

- [DexMG Environments](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/dexmg/environments/dexmg.py): `resfit/dexmg/environments/dexmg.py`
- [Residual Environment](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/wrappers/residual_env_wrapper.py): `resfit/rl_finetuning/wrappers/residual_env_wrapper.py`
- [DexMG Evaluation](https://github.com/amazon-far/residual-offpolicy-rl/blob/main/resfit/rl_finetuning/utils/evaluate_dexmg.py): `resfit/rl_finetuning/utils/evaluate_dexmg.py`

### Other components

- **Researcher**
- **Demonstrations**
- **Hugging Face Hub**
- **Weights & Biases**

## Connections

- Researcher → BC Training: starts training (evidence: `README.md`)
- Demonstrations → Dataset Conversion: converts data (evidence: `resfit/lerobot/dataset/convert_robomimic_to_lerobot.py`)
- BC Training → DexMG Environments: evaluates in (evidence: `resfit/lerobot/scripts/train_bc_dexmg.py`)
- BC Training → Policy Factory: creates policy (evidence: `resfit/lerobot/scripts/train_bc_dexmg.py`)
- Policy Factory → Policy Configurations: builds config (evidence: `resfit/lerobot/policies/factory.py`)
- Policy Factory → ACT Configuration: selects config (evidence: `resfit/lerobot/policies/factory.py`)
- Policy Factory → Diffusion Configuration: selects config (evidence: `resfit/lerobot/policies/factory.py`)
- Policy Factory → ACT Policy: instantiates (evidence: `resfit/lerobot/policies/factory.py`)
- Policy Factory → Diffusion Policy: instantiates (evidence: `resfit/lerobot/policies/factory.py`)
- ACT Policy → ACT Configuration: uses config (evidence: `resfit/lerobot/policies/act/modeling_act.py`)
- Diffusion Policy → Diffusion Configuration: uses config (evidence: `resfit/lerobot/policies/diffusion/modeling_diffusion.py`)
- ACT Policy → Pretrained Policies: extends policy base (evidence: `resfit/lerobot/policies/act/modeling_act.py`)
- Diffusion Policy → Pretrained Policies: extends policy base (evidence: `resfit/lerobot/policies/diffusion/modeling_diffusion.py`)
- BC Training → Pretrained Policies: saves policy (evidence: `resfit/lerobot/scripts/train_bc_dexmg.py`)
- BC Training → Policy Checkpoint Loader: loads policy (evidence: `resfit/lerobot/scripts/train_bc_dexmg.py`)
- Dataset Conversion → Hugging Face Hub: publishes dataset
- Researcher → Residual TD3 Training: starts fine-tuning (evidence: `README.md`)
- BC Training → Residual TD3 Config: checkpoint configured (evidence: `README.md`)
- Residual TD3 Training → Residual TD3 Config: loads config (evidence: `resfit/rl_finetuning/scripts/train_residual_td3.py`)
- Residual TD3 Training → DexMG Environments: creates environment (evidence: `resfit/rl_finetuning/scripts/train_residual_td3.py`)
- Residual TD3 Training → Policy Checkpoint Loader: loads BC policy (evidence: `resfit/rl_finetuning/scripts/train_residual_td3.py`)
- Residual TD3 Training → Residual Environment: wraps environment (evidence: `resfit/rl_finetuning/scripts/train_residual_td3.py`)
- Residual Environment → DexMG Environments: wraps environment (evidence: `resfit/rl_finetuning/wrappers/residual_env_wrapper.py`)
- Residual Environment → ACT Policy: uses BC policy (evidence: `resfit/rl_finetuning/wrappers/residual_env_wrapper.py`)
- Residual TD3 Training → Q Agent: constructs agent (evidence: `resfit/rl_finetuning/scripts/train_residual_td3.py`)
- Q Agent → Visual Encoder: uses encoder (evidence: `resfit/rl_finetuning/off_policy/rl/q_agent.py`)
- Q Agent → Actor Network: creates actor (evidence: `resfit/rl_finetuning/off_policy/rl/q_agent.py`)
- Q Agent → Critic Network: creates critic (evidence: `resfit/rl_finetuning/off_policy/rl/q_agent.py`)
- Residual TD3 Training → DexMG Evaluation: evaluates policy (evidence: `resfit/rl_finetuning/scripts/train_residual_td3.py`)
- DexMG Evaluation → DexMG Environments: runs episodes (evidence: `resfit/rl_finetuning/utils/evaluate_dexmg.py`)
- Residual TD3 Training → Weights & Biases: logs artifacts (evidence: `README.md`)
- Researcher → RLPD Training: starts RLPD
- RLPD Training → RLPD Config: loads config (evidence: `resfit/rl_finetuning/scripts/train_rlpd_dexmg.py`)
- RLPD Training → Q Agent: constructs agent (evidence: `resfit/rl_finetuning/scripts/train_rlpd_dexmg.py`)
- RLPD Training → DexMG Environments: creates environment (evidence: `resfit/rl_finetuning/scripts/train_rlpd_dexmg.py`)