# uscresl/dmfd architecture

> Architecture diagram of the GitHub repository uscresl/dmfd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uscresl/dmfd
- Repository: https://github.com/uscresl/dmfd

## Overview

##### Overall Project Purpose and Scope
- This repository implements a deep reinforcement learning (RL) framework for learning deformable object manipulation from expert demonstrations. It provides a simulation environment (SoftGym integrated with PyFlex), demonstration generation, and multiple RL training/evaluation pipelines (including DMfD, SAC variants, AWAC, BC, SAC-LfD, etc.).
- The project is “full-stack” in that it spans environment simulation, expert demonstration collection, training of agents with different RL algorithms (both state-based and image-based), and evaluation scripts.

##### Main Components
- Environment & Simulation:
– SoftGym main folder with its “softgym/softgym/envs” package, which contains the definitions for different tasks (like RopeFlatten, ClothFold, ClothFoldRobot, etc).
– PyFlex engine integration (found in softgym/PyFlex) that provides the underlying physics simulation. This is the “simulation backend.”
- Expert Demonstration Generation:
– The experiments/generate_expert_trajs.py script’s role is to collect expert trajectories. This forms the dataset used for imitation-based approaches.
- RL Agent / Training Modules:
– A number of directories (such as “dmfd”, “awac”, “rlpyt_cloth”, “drq”, “sb3”, “curl”) are included that contain various RL algorithms or training pipelines.
– Scripts in “scripts/train” and “scripts/eval” serve as orchestration for both training and evaluation of policies (state-based and image-based) over multiple tasks.
- Data and Experiment Management:
– Environment configuration files (like environment.yml, conda environments, Dockerfile in softgym/docker, etc) and preparation scripts (prepare_1.0.sh) ensure reproducibility.
– There are also directories for experiment outputs and cached data (e.g., “dmfd/” and data folders under dmfd), plus logging utilities.
- Software Infrastructure:
– Docker and Conda are used to package and deploy the environment.
– Documentation (under docs) and helper utilities (like logger, visualization, utils) support debugging and understanding results.

##### Interaction and Data Flow
- The RL training modules (DMfD, SAC variants, AWAC, etc.) request environment interactions from the SoftGym simulation module (backed by PyFlex).
- Expert demonstration generation (via the experiments scripts) produces datasets that are either directly fed to or used to pretrain and guide the RL agents.
- Evaluation scripts trigger trained agents to run on simulation tasks; outputs (videos, scores, logs) are produced.

##### Capturing Architectural Patterns
- Modularity and separation of concerns—the simulation environment is clearly decoupled from the RL algorithm implementations.
- The use of multiple training and evaluation scripts indicates a plug-and-play design, where switching from one RL method to another is straightforward.
- The integration of physical simulation (through PyFlex/SoftGym) with learning components shows an architecture that aligns closely with standard reinforcement learning systems but with an added layer for demonstration-based learning.

- Flow: Environment Setup (Docker/Conda) → Simulation Environment → Expert Demonstration data / RL trainer → Evaluation & Visualization.
- With technologies: Python, PyTorch, NVIDIA Flex (via PyFlex), Docker, Conda, etc.

– The simulation environment layer (SoftGym + PyFlex as the physics/graphics engine)
– The expert demonstration generation module
– The various reinforcement learning training pipelines (with multiple algorithm implementations)
– The evaluation and visualization modules for benchmarking experiments
– The underlying infrastructure (environment management via Docker/Conda).

## Diagram (Mermaid)

```mermaid
flowchart TD
    U["User/Researcher"]:::user
    SI["System Infrastructure (Docker, Conda, Config)"]:::infra
    SE["Simulation Environment (SoftGym + PyFlex)"]:::simulation
    ED["SoftGym Env Definitions"]:::simulation
    PF["PyFlex Physics Engine"]:::simulation
    ED --> PF

    ED --- SE
    PF --- SE

    ED2["Expert Demonstration Generation"]:::expert
    RT["RL Training Pipelines"]:::rl
    AWAC["AWAC"]:::rl
    DRQ["DRQ"]:::rl
    SB3["SB3 (SAC/BC)"]:::rl
    CURL["CURL"]:::rl
    RLPYT["RLPYT Cloth"]:::rl
    CEM["CEM"]:::rl
    PLANET["PLANET"]:::rl

    EV["Evaluation & Visualization"]:::eval
    DOC["Documentation & Utilities"]:::docs

    %% High-level flow
    U -->|"initiates"| SI
    SI -->|"sets up"| SE
    U -->|"collects"| ED2
    SE -->|"provides obs/acts"| RT
    ED2 -->|"provides expert traj"| RT
    SI -->|"provides deps"| RT
    RT -->|"produces outputs"| EV
    U -->|"references"| DOC
    EV -->|"logs & analyzes"| DOC

    %% RL Pipelines breakdown
    RT --> AWAC
    RT --> DRQ
    RT --> SB3
    RT --> CURL
    RT --> RLPYT
    RT --> CEM
    RT --> PLANET

    %% Click events
    click ED "https://github.com/uscresl/dmfd/tree/main/softgym/softgym/envs"
    click PF "https://github.com/uscresl/dmfd/tree/main/softgym/PyFlex"
    click ED2 "https://github.com/uscresl/dmfd/blob/main/experiments/generate_expert_trajs.py"
    click AWAC "https://github.com/uscresl/dmfd/tree/main/awac/"
    click DRQ "https://github.com/uscresl/dmfd/tree/main/drq/"
    click SB3 "https://github.com/uscresl/dmfd/tree/main/sb3/"
    click CURL "https://github.com/uscresl/dmfd/tree/main/curl/"
    click RLPYT "https://github.com/uscresl/dmfd/tree/main/rlpyt_cloth/"
    click CEM "https://github.com/uscresl/dmfd/tree/main/cem/"
    click PLANET "https://github.com/uscresl/dmfd/tree/main/planet/"
    click EV "https://github.com/uscresl/dmfd/tree/main/scripts/eval/"
    click SI "https://github.com/uscresl/dmfd/tree/main/softgym/docker/"
    click DOC "https://github.com/uscresl/dmfd/tree/main/docs/"

    %% Styling
    classDef user fill:#ffd1dc,stroke:#ff69b4,stroke-width:2px;
    classDef infra fill:#c7ecee,stroke:#0984e3,stroke-width:2px;
    classDef simulation fill:#badc58,stroke:#6ab04c,stroke-width:2px;
    classDef expert fill:#f9ca24,stroke:#e1b12c,stroke-width:2px;
    classDef rl fill:#dff9fb,stroke:#130f40,stroke-width:2px;
    classDef eval fill:#f6e58d,stroke:#e58e26,stroke-width:2px;
    classDef docs fill:#c8d6e5,stroke:#576574,stroke-width:2px;
```
