# jamesd33064/diffusion-policies-for-offline-rl architecture

> Architecture diagram of the GitHub repository jamesd33064/diffusion-policies-for-offline-rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jamesd33064/diffusion-policies-for-offline-rl
- Repository: https://github.com/jamesd33064/diffusion-policies-for-offline-rl

## Overview

##### Project type and purpose
- This is an open-source research codebase for offline reinforcement learning (RL) that implements diffusion-based policies in PyTorch.
- It provides implementations of behavior cloning (BC), Diffusion-QL (offline policy improvement using a diffusion model), and toy experiments illustrating these algorithms.
- Key indicators: “Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning” in the README, the presence of MuJoCo and D4RL dependencies, and multiple “agents” implementations.

##### File structure
- Top level
- main.py – primary entrypoint for full experiments
- run_toy*.py and run_toy.sh – lightweight scripts for toy experiments
- requirements.txt, LICENSE, .gitignore
- tabulate.py – likely for formatting results
- agents/
- bc_diffusion.py, diffusion.py, ql_diffusion.py – core algorithm implementations
- model.py – neural network definitions (diffusion model, Q-networks)
- helpers.py – shared support routines (e.g., computing losses)
- toy_experiments/
- Multiple scripts implementing toy variants of BC, diffusion, QL using different objectives (MLE, MMD, CVAE)
- toy_helpers.py – utilities specific to toy runs
- utils/
- data_sampler.py – loading and batching D4RL datasets or toy data
- logger.py – experiment logging, metrics tracking
- pytorch_util.py – helper functions for model saving, weight initialization, GPU management
- utils.py – miscellaneous utilities

##### README for additional insights
- Describes core approach: behavior cloning + policy improvement via conditional diffusion model.
- Dependencies: PyTorch, MuJoCo, D4RL.
- Running instructions: “python main.py --env_name …” for full experiments, toy scripts for simple 2D bandit.

##### Main components
a. Data ingestion
- D4RL dataset loader (utils/data_sampler.py)
- MuJoCo environment interface (via gym + mujoco backend)
b. Core algorithm layer
- Agents module
· Diffusion policy network (agents/model.py)
· Q-network (agents/model.py)
· Loss computation & scheduler (agents/helpers.py)
- Algorithm orchestrators
· BC (agents/bc_diffusion.py)
· Diffusion-QL (agents/ql_diffusion.py)
c. Experiment layer
- main.py – full benchmark experiments (D4RL, online/offline model selection)
- toy_experiments/ – simplified experiments for rapid prototyping
- run_toy*.py – convenience wrappers
d. Utilities & support
- Logging & checkpointing (utils/logger.py)
- PyTorch helpers (utils/pytorch_util.py)
- Generic utilities (utils/utils.py, tabulate.py)
e. External services
- PyTorch hardware acceleration (CUDA)
- D4RL datasets
- MuJoCo simulator

##### Relationships and interactions
- Data flow: D4RL dataset → data_sampler → agent training loop
- Control flow: main.py invokes agent setup → training loop → periodic evaluation via MuJoCo → logger records results
- Toy flows: run_toy scripts → toy_experiments → agents → toy_helpers

##### Architectural patterns and design principles
- Modular “agent” layer separating policy representations from training logic
- Clear separation of concerns: data loading (utils), model definitions (agents/model.py), experiment orchestration (main.py/toy_experiments)
- Plug-and-play: swapping algorithms (BC vs. QL vs. diffusion) by swapping agent classes
- Experiment reproducibility: hard-coded hyperparameters in main.py, logging, checkpointing

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Env Layer
    subgraph "Data & Env" 
        direction TB
        D4RL["D4RL Dataset"]:::external
        DataSampler["DataSampler"]:::data
        GymMuJoCo["Gym/MuJoCo Env"]:::data
    end

    %% Agent Layer
    subgraph "Core Algorithm (Agents)" 
        direction TB
        ModelDefs["Model Definitions\n(diffusion & Q-networks)"]:::core
        LossHelpers["Loss Computation & Training Helpers"]:::core
        BCAgent["Behavior Cloning Agent"]:::core
        DiffusionQL["Diffusion-QL Agent"]:::core
        DiffusionOnly["Stand-alone Diffusion Agent"]:::core
    end

    %% Experiment Orchestration
    subgraph "Experiment Orchestration" 
        direction TB
        Main["Full Experiment Orchestrator\n(main.py)"]:::orchestration
        subgraph "Toy Experiments"
            direction TB
            ToyDir["Toy Experiment Scripts"]:::orchestration
            ToyHelpers["Toy Experiment Utilities"]:::orchestration
            RunToySH["run_toy.sh"]:::orchestration
            RunToyBC["run_toy_bc.py"]:::orchestration
            RunToyQL["run_toy_ql.py"]:::orchestration
        end
    end

    %% Utilities Layer
    subgraph "Utilities & Support" 
        direction TB
        Logger["Logging & Metrics Tracking"]:::utils
        PyTorchUtil["PyTorch Utilities"]:::utils
        GenericUtils["Generic Utilities"]:::utils
        Tabulate["Tabulated Results Formatter"]:::utils
    end

    %% External Services
    subgraph "External Services"
        direction TB
        PyTorch["PyTorch Framework"]:::external
        CUDA["CUDA GPU"]:::external
        MuJoCoSim["MuJoCo Simulator"]:::external
    end

    %% Connections - Data flow
    D4RL --> DataSampler
    DataSampler --> BCAgent
    DataSampler --> DiffusionOnly
    DataSampler --> DiffusionQL

    GymMuJoCo --> Main
    MuJoCoSim --> GymMuJoCo

    %% Training orchestration
    Main --> ModelDefs
    Main --> LossHelpers
    Main --> BCAgent
    Main --> DiffusionQL
    Main --> DiffusionOnly
    Main --> Logger
    Main --> PyTorchUtil
    Main --> PyTorch
    PyTorch --> CUDA

    %% Toy flow
    RunToySH --> ToyDir
    RunToyBC --> ToyDir
    RunToyQL --> ToyDir
    ToyDir --> ToyHelpers
    ToyHelpers --> BCAgent
    ToyHelpers --> DiffusionOnly
    ToyHelpers --> DiffusionQL

    %% Utilities usage
    BCAgent --> Logger
    DiffusionQL --> Logger
    DiffusionOnly --> Logger
    PyTorchUtil --> BCAgent
    PyTorchUtil --> DiffusionQL
    PyTorchUtil --> DiffusionOnly
    GenericUtils --> Main
    GenericUtils --> ToyHelpers
    Tabulate --> Main

    %% Click Events
    click DataSampler "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/utils/data_sampler.py"
    click GymMuJoCo "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/main.py"
    click ModelDefs "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/agents/model.py"
    click LossHelpers "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/agents/helpers.py"
    click BCAgent "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/agents/bc_diffusion.py"
    click DiffusionQL "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/agents/ql_diffusion.py"
    click DiffusionOnly "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/agents/diffusion.py"
    click Main "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/main.py"
    click ToyDir "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/tree/master/toy_experiments/"
    click ToyHelpers "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/toy_experiments/toy_helpers.py"
    click RunToySH "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/run_toy.sh"
    click RunToyBC "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/run_toy_bc.py"
    click RunToyQL "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/run_toy_ql.py"
    click Logger "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/utils/logger.py"
    click PyTorchUtil "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/utils/pytorch_util.py"
    click GenericUtils "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/utils/utils.py"
    click Tabulate "https://github.com/jamesd33064/diffusion-policies-for-offline-rl/blob/master/tabulate.py"

    %% Styles
    classDef data fill:#b2f2bb,stroke:#2f6627,color:#0f5132
    classDef core fill:#cfe2ff,stroke:#084298,color:#084298
    classDef orchestration fill:#ffe5b4,stroke:#cc8400,color:#663c00
    classDef utils fill:#e2cff1,stroke:#6f42c1,color:#511281
    classDef external fill:#d6d8db,stroke:#6c757d,color:#343a40
```
