How nvlabs/drail works
Main components
a. Configuration & CLI
– config.yaml & configs/ directory (WandB sweep specs)
– utils/wandb.sh and other shell scripts
– Entry script: drail/main.py
b. Environment Layer
– goal_prox module (custom MuJoCo & MiniGrid envs)
– d4rl module (Maze2D, Bullet, Carla, Hand‐Manipulation Suite)
– shape_env (custom Sine/Trajectory envs)
– External Sims: MuJoCo, Bullet, CARLA
c. Expert Dataset Storage
– ./expert_datasets (downloaded via utils/expert_data.sh)
d. Core Algorithm Layer
– DRAIL variants:
• Diffusion‐based discriminator: drail/ddpm/*
• Main DRAIL logic: drail/drail.py, drail/drail_un.py
• Policy extraction: drail/get_policy.py
– Baselines (via rl-toolkit/rlf/algos/il): GAIL, WAIL, BC, DP
– Additional RL routines (SAC in pytorch_sac)
e. Training & Orchestration
– drail/main.py invokes:
• Config loader → sets hyperparams → selects env & algorithm
• Loads expert dataset
• Constructs Diffusion model module & policy module
• Runs adversarial training loop (discriminator ↔ policy)
• Uses RL‐toolkit or PyTorch‐SAC for policy updates
f. Logging & Monitoring
– Weights & Biases (W&B) via utils/wandb.sh & logger.py
– Local saving of model checkpoints & plots
Relationships and data flows
– Configs → Main script → Environment instantiation & Expert‐data loader
– Expert Data + Environment interactions → Trajectories
– Trajectories feed both:
• Diffusion discriminator (drail/ddpm) → reward function
• Policy optimizer (RL‐toolkit or SAC)
– Discriminator outputs “diffusion reward” → policy update
– Policy rollouts → new trajectories → discriminator
– Throughout, hyperparameters & progress → W&B → Web UI
Architectural patterns
– Modular plugin pattern: new envs (goal_prox, d4rl, shape_env) and new algorithms (drail vs. GAIL vs. WAIL) swap in via configuration
– Pipeline pattern: data (expert & rollouts) flows uni‐ and bi‐directionally between discriminator and policy optimizer
– Layered separation:
• Env & Data Layer
• Algorithmic Layer (discriminator + optimizer)
• Orchestration Layer (main, config, logging)
Relevant technologies
– Python 3.8+, PyTorch, mujoco-py, Gym, Gym-Bullet, CARLA
– Weights & Biases (wandb)
– Shell scripts for setup & data preprocessing