How amazon-far/residual-offpolicy-rl works
ResFiT is a robotics-learning codebase for training behavior-cloning (BC) policies and fine-tuning them with residual off-policy reinforcement learning. The main workflow is BC training on DexMG demonstrations, configuring the resulting policy checkpoint, then residual-TD3 training and evaluation in DexMG environments. The repository also includes RLPD training, ACT and diffusion policy implementations, and a Robomimic-to-LeRobot dataset converter. Edges follow README statements or source-index references; unsampled implementation details and unverified links are omitted.
Main entry points: train_residual_td3.py Residual TD3 Training · train_bc_dexmg.py BC Training · factory.py Policy Factory · q_agent.py Q Agent
Components 25
Demonstration Data
- Dataset Conversion
resfit/lerobot/dataset/convert_robomimic_to_lerobot.py
BC Policies
- BC Training
resfit/lerobot/scripts/train_bc_dexmg.py - Policy Configurations
resfit/lerobot/configs/policies.py - ACT Configuration
resfit/lerobot/policies/act/configuration_act.py - ACT Policy
resfit/lerobot/policies/act/modeling_act.py - Diffusion Configuration
resfit/lerobot/policies/diffusion/configuration_diffusion.py - Diffusion Policy
resfit/lerobot/policies/diffusion/modeling_diffusion.py - Policy Factory
resfit/lerobot/policies/factory.py - Pretrained Policies
resfit/lerobot/policies/pretrained.py - Policy Checkpoint Loader
resfit/lerobot/utils/load_policy.py
Residual RL
- Residual TD3 Config
resfit/rl_finetuning/config/residual_td3.py - RLPD Config
resfit/rl_finetuning/config/rlpd.py - Visual Encoder
resfit/rl_finetuning/off_policy/networks/encoder.py - Actor Network
resfit/rl_finetuning/off_policy/rl/actor.py - Critic Network
resfit/rl_finetuning/off_policy/rl/critic.py - Q Agent
resfit/rl_finetuning/off_policy/rl/q_agent.py - Residual TD3 Training
resfit/rl_finetuning/scripts/train_residual_td3.py - RLPD Training
resfit/rl_finetuning/scripts/train_rlpd_dexmg.py
Environment and Evaluation
- DexMG Environments
resfit/dexmg/environments/dexmg.py - Residual Environment
resfit/rl_finetuning/wrappers/residual_env_wrapper.py - DexMG Evaluation
resfit/rl_finetuning/utils/evaluate_dexmg.py
Other components
- Researcher
- Demonstrations
- Hugging Face Hub
- Weights & Biases
Connections 35
- Researcher → BC Training: starts training
README.md - Demonstrations → Dataset Conversion: converts data
resfit/lerobot/dataset/convert_robomimic_to_lerobot.py - BC Training → DexMG Environments: evaluates in
resfit/lerobot/scripts/train_bc_dexmg.py - BC Training → Policy Factory: creates policy
resfit/lerobot/scripts/train_bc_dexmg.py - Policy Factory → Policy Configurations: builds config
resfit/lerobot/policies/factory.py - Policy Factory → ACT Configuration: selects config
resfit/lerobot/policies/factory.py - Policy Factory → Diffusion Configuration: selects config
resfit/lerobot/policies/factory.py - Policy Factory → ACT Policy: instantiates
resfit/lerobot/policies/factory.py - Policy Factory → Diffusion Policy: instantiates
resfit/lerobot/policies/factory.py - ACT Policy → ACT Configuration: uses config
resfit/lerobot/policies/act/modeling_act.py - Diffusion Policy → Diffusion Configuration: uses config
resfit/lerobot/policies/diffusion/modeling_diffusion.py - ACT Policy → Pretrained Policies: extends policy base
resfit/lerobot/policies/act/modeling_act.py - Diffusion Policy → Pretrained Policies: extends policy base
resfit/lerobot/policies/diffusion/modeling_diffusion.py - BC Training → Pretrained Policies: saves policy
resfit/lerobot/scripts/train_bc_dexmg.py - BC Training → Policy Checkpoint Loader: loads policy
resfit/lerobot/scripts/train_bc_dexmg.py - Dataset Conversion → Hugging Face Hub: publishes dataset
- Researcher → Residual TD3 Training: starts fine-tuning
README.md - BC Training → Residual TD3 Config: checkpoint configured
README.md - Residual TD3 Training → Residual TD3 Config: loads config
resfit/rl_finetuning/scripts/train_residual_td3.py - Residual TD3 Training → DexMG Environments: creates environment
resfit/rl_finetuning/scripts/train_residual_td3.py - Residual TD3 Training → Policy Checkpoint Loader: loads BC policy
resfit/rl_finetuning/scripts/train_residual_td3.py - Residual TD3 Training → Residual Environment: wraps environment
resfit/rl_finetuning/scripts/train_residual_td3.py - Residual Environment → DexMG Environments: wraps environment
resfit/rl_finetuning/wrappers/residual_env_wrapper.py - Residual Environment → ACT Policy: uses BC policy
resfit/rl_finetuning/wrappers/residual_env_wrapper.py - Residual TD3 Training → Q Agent: constructs agent
resfit/rl_finetuning/scripts/train_residual_td3.py - Q Agent → Visual Encoder: uses encoder
resfit/rl_finetuning/off_policy/rl/q_agent.py - Q Agent → Actor Network: creates actor
resfit/rl_finetuning/off_policy/rl/q_agent.py - Q Agent → Critic Network: creates critic
resfit/rl_finetuning/off_policy/rl/q_agent.py - Residual TD3 Training → DexMG Evaluation: evaluates policy
resfit/rl_finetuning/scripts/train_residual_td3.py - DexMG Evaluation → DexMG Environments: runs episodes
resfit/rl_finetuning/utils/evaluate_dexmg.py - Residual TD3 Training → Weights & Biases: logs artifacts
README.md - Researcher → RLPD Training: starts RLPD
- RLPD Training → RLPD Config: loads config
resfit/rl_finetuning/scripts/train_rlpd_dexmg.py - RLPD Training → Q Agent: constructs agent
resfit/rl_finetuning/scripts/train_rlpd_dexmg.py - RLPD Training → DexMG Environments: creates environment
resfit/rl_finetuning/scripts/train_rlpd_dexmg.py