How drcl-usc/ogmp works
Project Type and Purpose
• This is a research‐oriented reinforcement‐learning (RL) codebase for “Oracle Guided Multimodal Policies” (OGMP) used in agile robot control tasks (parkour, dive, etc.).
• It implements custom PPO algorithms (algos/), neural networks (nn/), environment interfaces (dtsd/), oracle modules (oracles/), training/test entry points (train.py, test.py), experiment configurations (exp_confs/), analysis scripts (analysis/), and logging infrastructure (logs/).
High-Level Components
A. Configuration Layer
• exp_confs/*.yaml – experiment parameters (environment, horizon, hyperparameters)
• dtsd/confs/*.yaml – environment‐specific parameters (robot model, terrain)
B. Core RL Pipeline
• train.py
– Reads YAML configs → Instantiates Environment (dtsd/envs & mujoco) → Builds Oracle (oracles/) and Policy/Critic (nn/) → Uses PPO implementation (algos/) to collect rollouts, update networks → Writes model checkpoints, TensorBoard logs (logs/)
• test.py
– Loads best model + config → Runs episodes in render mode → Saves videos/metrics
C. Environment Abstraction
• dtsd/envs/src/*.py – Gym‐style wrappers: action, observation, reward, logging, frame recording
• dtsd/envs/sim/*.py – Task definitions (dive.py, parkour.py), simulation base classes (mujoco_sim_base.py)
• dtsd/envs/rsc/models/* – Robot Mujoco XML/YAML & mesh assets
D. Neural Networks & Oracle Modules
• nn/ – Policy (actor.py), Value (critic.py), AE/VAE encoders (lstm_ae.py, lstm_vae.py), training utilities (fit.py)
• oracles/ – Interface for ground‐truth oracles (e.g., LQR, Li‐oracle), optimisers, utility functions
E. Algorithm Implementation
• algos/ – Custom PPO variants (ppo.py, extract.py) adapted for recurrent policies
F. Analysis & Results
• analysis/ – Scripts to compute metrics (rollout statistics, LMSR tests) using trained policies
• logs/ – Structured folders per experiment storing actor.pt, critic.pt, TensorBoard event files, param lists, encoders, reconstructions
• media/ – Visual artifacts (GIFs, images) for demonstration
Relationships & Data Flow
1. User invokes train.py/test.py with --exp_conf_path.
2. Script loads experiment config (exp_confs/) and environment config (dtsd/confs/).
3. Environment factory in dtsd/envs constructs a Mujoco or Gym‐like environment.
4. Oracle module is selected per config to provide privileged data or guidance.
5. Neural network models (nn/) are instantiated (actor, critic, optional autoencoder).
6. PPO algorithm (algos/) orchestrates:
– Interact with env → collect rollouts → optionally consult oracle → compute advantages → update nn weights
7. After each update, models are checkpointed in logs/, and metrics are published to TensorBoard.
8. Once training concludes, test.py can load checkpoints and run evaluation episodes (rendered).
9. analysis/ scripts read policies and logs to produce final plots in results/.
Architectural Patterns & Principles
• Modular separation:
• Configuration vs. code
• Environment abstraction (dtsd/) vs. algorithm (algos/) vs. model (nn/) vs. oracle (oracles/)
• Pipeline pattern: config → environment → data collection → model update → logging → analysis
• Dependency injection: experiment configs drive which env, oracle, and network variants are used
• Reusability:
• analysis/ reuses trained policies and env wrappers
• oracles/ can be swapped in with minimal code change
Detailed Capture of Essential Elements
• Do not collapse modules into a single “RL code” block. Show each folder as its own component.
• Where oracles intervene in that loop.
• Where multiprocessing is used in analysis/ (forked policy copies).