How uscresl/dmfd works
Overall Project Purpose and Scope
• This repository implements a deep reinforcement learning (RL) framework for learning deformable object manipulation from expert demonstrations. It provides a simulation environment (SoftGym integrated with PyFlex), demonstration generation, and multiple RL training/evaluation pipelines (including DMfD, SAC variants, AWAC, BC, SAC-LfD, etc.).
• The project is “full-stack” in that it spans environment simulation, expert demonstration collection, training of agents with different RL algorithms (both state-based and image-based), and evaluation scripts.
Main Components
• Environment & Simulation:
– SoftGym main folder with its “softgym/softgym/envs” package, which contains the definitions for different tasks (like RopeFlatten, ClothFold, ClothFoldRobot, etc).
– PyFlex engine integration (found in softgym/PyFlex) that provides the underlying physics simulation. This is the “simulation backend.”
• Expert Demonstration Generation:
– The experiments/generate_expert_trajs.py script’s role is to collect expert trajectories. This forms the dataset used for imitation-based approaches.
• RL Agent / Training Modules:
– A number of directories (such as “dmfd”, “awac”, “rlpyt_cloth”, “drq”, “sb3”, “curl”) are included that contain various RL algorithms or training pipelines.
– Scripts in “scripts/train” and “scripts/eval” serve as orchestration for both training and evaluation of policies (state-based and image-based) over multiple tasks.
• Data and Experiment Management:
– Environment configuration files (like environment.yml, conda environments, Dockerfile in softgym/docker, etc) and preparation scripts (prepare_1.0.sh) ensure reproducibility.
– There are also directories for experiment outputs and cached data (e.g., “dmfd/” and data folders under dmfd), plus logging utilities.
• Software Infrastructure:
– Docker and Conda are used to package and deploy the environment.
– Documentation (under docs) and helper utilities (like logger, visualization, utils) support debugging and understanding results.
Interaction and Data Flow
• The RL training modules (DMfD, SAC variants, AWAC, etc.) request environment interactions from the SoftGym simulation module (backed by PyFlex).
• Expert demonstration generation (via the experiments scripts) produces datasets that are either directly fed to or used to pretrain and guide the RL agents.
• Evaluation scripts trigger trained agents to run on simulation tasks; outputs (videos, scores, logs) are produced.
Capturing Architectural Patterns
• Modularity and separation of concerns—the simulation environment is clearly decoupled from the RL algorithm implementations.
• The use of multiple training and evaluation scripts indicates a plug-and-play design, where switching from one RL method to another is straightforward.
• The integration of physical simulation (through PyFlex/SoftGym) with learning components shows an architecture that aligns closely with standard reinforcement learning systems but with an added layer for demonstration-based learning.
• Flow: Environment Setup (Docker/Conda) → Simulation Environment → Expert Demonstration data / RL trainer → Evaluation & Visualization.
• With technologies: Python, PyTorch, NVIDIA Flex (via PyFlex), Docker, Conda, etc.
– The simulation environment layer (SoftGym + PyFlex as the physics/graphics engine)
– The expert demonstration generation module
– The various reinforcement learning training pipelines (with multiple algorithm implementations)
– The evaluation and visualization modules for benchmarking experiments
– The underlying infrastructure (environment management via Docker/Conda).