How runzheyang/morl works
Project type and purpose
• This is a multi‐domain research codebase implementing a generalized multi‐objective reinforcement learning (MORL) algorithm and few‐shot policy adaptation, applied to three domains: synthetic toy environments (Deep Sea Treasure, Fruit Tree Navigation), a task‐oriented dialogue system (built on PyDial), and a multi‐objective version of SuperMario (gym‐super‐mario‐bros).
• The README describes training scripts, core algorithm modules (envelope MOQ‐learning), domain wrappers/environments, evaluation utilities, and supporting scripts.
File structure
• Top‐level directories:
• synthetic/ – PyTorch MORL implementation for toy domains, with subfolders for CRL (algorithm implementations), envs (domain definitions), scripts, tests, and utilities.
• pydial/ – Extension of the PyDial toolkit: dialog server, belief trackers, policy modules (including MORL envelope), semantic modules, user simulator, evaluation, tasks, tests, docs, configs, and training scripts.
• multimario/ – Multi‐objective SuperMario agent, environment wrapper, model definition, training and evaluation scripts, plus Slurm/job scripts.
• train.py, multimario/run_*.py, pydial/train and simulation entrypoints orchestrate experiments.
• Common patterns:
– “meta.py” under synthetic/crl/envelope implements core algorithm.
– Model architectures in “models/…”.
– Domain‐specific env wrappers under synthetic/envs, multimario/env.py, and PyDial’s DialogueServer.
– Scripts folder structures for local vs. cluster training.
Additional insights from README
• The “envelope” MORL algorithm is implemented in three places: synthetic/crl/envelope/meta.py, pydial/policy/envelope.py, and multimario/agent.py.
• Training is invoked via command‐line wrappers that parse hyperparameters, set up environments, initialize model & preference encodings, and run the generalized Bellman update.
• Evaluation utilities plot results, compute few‐shot adaptation metrics, and simulate rollouts.
a. Main components
• Core MORL Algorithm Module (“envelope”): parameterized policy & Bellman update logic
• Domain Interfaces / Environments:
– SyntheticEnv (DST, FTN)
– DialogueEnv (PyDial wrapper + DialogueServer)
– SuperMarioEnv (gym‐super‐mario‐bros wrapper)
• Model Architectures: linear, neural nets under CRL/models or multimario/model.py
• Training Orchestrator: train.py, pydial train CLI, multimario run scripts
• Replay Buffer / Memory (synthetic/replay_buffer_*, multimario sample logic)
• Evaluation & Visualization: synthetic/scripts, pydial/results/plot.py, multimario scripts
• Configuration & Experiment Management: bash scripts, Slurm cmds, config files under pydial/config and synthetic/scripts
b. Relationships & Interactions
– The Training Orchestrator reads a config or CLI flags → instantiates Domain Interface → Core MORL Module → Model Architecture + Replay Buffer → interacts with Domain to collect trajectories → updates policy parameters → periodically invokes Evaluation → writes logs & checkpoints.
– For dialogue: Training Orchestrator calls pydial train → DialogueServer + UserSimulator → runs episodes → PolicyManager (with envelope logic) → BeliefTracker, Semantic modules, NLG → sample actions → loop.
– Evaluation utilities read saved checkpoints and logged rewards → generate plots or adaptation metrics.
c. Architectural patterns & principles
• Modular separation: domain wrappers decoupled from algorithm logic
• Strategy pattern: different preference weighting strategies plugged into the same envelope update
• MVC‐like layering in PyDial: DialogueServer (controller), ontology & belief tracking (model), NLG templates & RNN (view)
• Reusable core library: same envelope/meta.py logic reused across three domains with minimal glue code
d. Key technologies & frameworks
– PyTorch (0.4.x or 1.1.0) for neural network policies
– gym interface for environments
– Python 2.7 for PyDial, Python 3.5 for synthetic & multimario
– bash/Slurm for job orchestration
– visdom (optional) for synthetic env logging