How baitingzbt/peda works
Project Type and Purpose
• This is an academic research codebase (“PEDA”) for offline multi-objective reinforcement learning experiments, not a classic full-stack app. It provides (a) dataset generation pipelines, (b) environment wrappers for MuJoCo tasks, (c) two model implementations (Decision Transformer “modt” and RVS “rvs”), and (d) an experiment orchestrator to run training & evaluation.
High-Level Components
a. Environment Layer
• Directory “environments/” and “data_generation/environments/”: XML assets + Python wrappers for MuJoCo tasks (Ant, HalfCheetah, Hopper, Swimmer, Walker2d, Humanoid).
b. Data Generation Pipeline
• “data_generation/”: scripts to collect and preprocess trajectories.
• Sub-folders “externals/baselines” and “externals/pytorch-a2c-ppo-acktr-gail”: embed open-source RL libraries to generate or replay trajectories.
c. Offline Dataset Storage
• Generated data lands in “data/” (pickled trajectories) and “data_generation/dummy”.
d. Model Implementations
• “modt/”: decision transformer code—models, trainers, evaluation, utilities.
• “rvs/”: return‐conditional VAE style model—data loading, policy definitions, training, analysis, visualization.
e. Experiment Orchestrator
• “experiment.py” + shell scripts (“all_env_uniform.sh”): parse CLI args, set up paths, launch training runs.
f. Pretrained Models & Outputs
• “lr_models/”: precomputed expert models for dataset generation.
• Logging directories under “experiment_runs/” (not listed) or specified output folders.
Relationships & Data Flow
• Start: Environment wrappers ← MuJoCo XML assets
↓
• Data Generation Module (scripts invoke external libraries baselines / pytorch)
↓
• Offline Dataset (“data/… .pkl”)
↓
• Two Training Pipelines in Parallel:
• modt/training → modt/models → modt/evaluation
• rvs/src/rvs/train.py → rvs/src/rvs/policies, dataset
↓
• Evaluation & Visualization modules produce metrics & plots
↓
• experiment.py orchestrates runs, collects logs, saves trained checkpoints
Architectural Patterns & Principles
• Pipeline Pattern: clear stages (data generation → storage → training → evaluation).
• Modularity: separate components for each model type (modt vs. rvs) and for environment wrappers vs. dataset code.
• Extensibility: “externals/” allows swapping in other RL libraries; one can plug new envs, new models.