# runzheyang/morl architecture

> Architecture diagram of the GitHub repository runzheyang/morl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/runzheyang/morl
- Repository: https://github.com/runzheyang/morl

## Overview

##### Project type and purpose
- This is a multi‐domain research codebase implementing a generalized multi‐objective reinforcement learning (MORL) algorithm and few‐shot policy adaptation, applied to three domains: synthetic toy environments (Deep Sea Treasure, Fruit Tree Navigation), a task‐oriented dialogue system (built on PyDial), and a multi‐objective version of SuperMario (gym‐super‐mario‐bros).
- The README describes training scripts, core algorithm modules (envelope MOQ‐learning), domain wrappers/environments, evaluation utilities, and supporting scripts.

##### File structure
- Top‐level directories:
- synthetic/ – PyTorch MORL implementation for toy domains, with subfolders for CRL (algorithm implementations), envs (domain definitions), scripts, tests, and utilities.
- pydial/ – Extension of the PyDial toolkit: dialog server, belief trackers, policy modules (including MORL envelope), semantic modules, user simulator, evaluation, tasks, tests, docs, configs, and training scripts.
- multimario/ – Multi‐objective SuperMario agent, environment wrapper, model definition, training and evaluation scripts, plus Slurm/job scripts.
- train.py, multimario/run_*.py, pydial/train and simulation entrypoints orchestrate experiments.
- Common patterns:
– “meta.py” under synthetic/crl/envelope implements core algorithm.
– Model architectures in “models/…”.
– Domain‐specific env wrappers under synthetic/envs, multimario/env.py, and PyDial’s DialogueServer.
– Scripts folder structures for local vs. cluster training.

##### Additional insights from README
- The “envelope” MORL algorithm is implemented in three places: synthetic/crl/envelope/meta.py, pydial/policy/envelope.py, and multimario/agent.py.
- Training is invoked via command‐line wrappers that parse hyperparameters, set up environments, initialize model & preference encodings, and run the generalized Bellman update.
- Evaluation utilities plot results, compute few‐shot adaptation metrics, and simulate rollouts.

a. Main components
- Core MORL Algorithm Module (“envelope”): parameterized policy & Bellman update logic
- Domain Interfaces / Environments:
– SyntheticEnv (DST, FTN)
– DialogueEnv (PyDial wrapper + DialogueServer)
– SuperMarioEnv (gym‐super‐mario‐bros wrapper)
- Model Architectures: linear, neural nets under CRL/models or multimario/model.py
- Training Orchestrator: train.py, pydial train CLI, multimario run scripts
- Replay Buffer / Memory (synthetic/replay_buffer_*, multimario sample logic)
- Evaluation & Visualization: synthetic/scripts, pydial/results/plot.py, multimario scripts
- Configuration & Experiment Management: bash scripts, Slurm cmds, config files under pydial/config and synthetic/scripts

b. Relationships & Interactions
– The Training Orchestrator reads a config or CLI flags → instantiates Domain Interface → Core MORL Module → Model Architecture + Replay Buffer → interacts with Domain to collect trajectories → updates policy parameters → periodically invokes Evaluation → writes logs & checkpoints.
– For dialogue: Training Orchestrator calls pydial train → DialogueServer + UserSimulator → runs episodes → PolicyManager (with envelope logic) → BeliefTracker, Semantic modules, NLG → sample actions → loop.
– Evaluation utilities read saved checkpoints and logged rewards → generate plots or adaptation metrics.

c. Architectural patterns & principles
- Modular separation: domain wrappers decoupled from algorithm logic
- Strategy pattern: different preference weighting strategies plugged into the same envelope update
- MVC‐like layering in PyDial: DialogueServer (controller), ontology & belief tracking (model), NLG templates & RNN (view)
- Reusable core library: same envelope/meta.py logic reused across three domains with minimal glue code

d. Key technologies & frameworks
– PyTorch (0.4.x or 1.1.0) for neural network policies
– gym interface for environments
– Python 2.7 for PyDial, Python 3.5 for synthetic & multimario
– bash/Slurm for job orchestration
– visdom (optional) for synthetic env logging

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Training Orchestrators
    subgraph "Training Orchestrators"
        direction TB
        synthetic_train1["synthetic/train.py"]:::infra
        synthetic_train2["synthetic/roijers_train.py"]:::infra
        simulate["pydial/Simulate.py"]:::infra
        run_e3c["multimario/run_e3c.py"]:::infra
        run_a3c["multimario/run_a3c.py"]:::infra
        run_n3c["multimario/run_n3c.py"]:::infra
        adapt_e["multimario/adapt_e.py"]:::infra
        adapt_n["multimario/adapt_n.py"]:::infra
    end

    %% Domain Interfaces
    subgraph "Domain Interfaces"
        direction TB
        deep_sea["Deep Sea Treasure Env"]:::domain
        fruit_tree["Fruit Tree Navigation Env"]:::domain
        mo_env["Generic MO Env"]:::domain
        dialogue_server["DialogueServer"]:::domain
        mario_env["SuperMarioEnv"]:::domain
    end

    %% Core MORL Modules
    subgraph "Core MORL Module"
        direction TB
        envelope_syn["Envelope (Synthetic)"]:::algorithmCore
        envelope_dial["Envelope (PyDial)"]:::algorithmCore
        envelope_mario["Envelope (SuperMario)"]:::algorithmCore
    end

    %% Model Architectures
    subgraph "Model Architectures"
        direction TB
        linear_model["Linear Model"]:::domain
        mario_model["Mario Model"]:::domain
    end

    %% Replay Buffers
    subgraph "Replay Buffers"
        direction TB
        dial_buffer["PyDial ReplayBuffer"]:::domain
        mario_buffer["Mario Sampling Logic"]:::domain
    end

    %% Evaluation & Visualization
    subgraph "Evaluation & Visualization"
        direction TB
        plot_for_paper["plot_for_paper.py"]:::evaluation
        synthetic_scripts_eval["synthetic/scripts (eval)"]:::evaluation
        dial_plot["pydial/results/plot.py"]:::evaluation
        mario_eval_n3c["eval_n3c.py"]:::evaluation
        mario_eval_sh["eval.sh"]:::evaluation
    end

    %% Configuration & Experiment Mgmt
    subgraph "Config & Experiment Management"
        direction TB
        pydial_config1["pydial/config/MORL"]:::infra
        pydial_config2["pydial/_morlconfigs/"]:::infra
        synthetic_scripts_conf["synthetic/scripts (config)"]:::infra
        mario_local["multimario/scripts/_local/"]:::infra
        mario_ws["multimario/scripts/_workstation/"]:::infra
        mario_slurm["multimario/slurm/"]:::infra
    end

    %% Connections: Orchestrators -> Domain Interfaces
    synthetic_train1 -->|"launches"| deep_sea
    synthetic_train1 -->|"launches"| fruit_tree
    synthetic_train1 -->|"launches"| mo_env
    synthetic_train2 -->|"launches"| mo_env
    simulate -->|"launches"| dialogue_server
    run_e3c -->|"launches"| mario_env
    run_a3c -->|"launches"| mario_env
    run_n3c -->|"launches"| mario_env
    adapt_e -->|"launches"| mario_env
    adapt_n -->|"launches"| mario_env

    %% Orchestrators -> Core MORL
    synthetic_train1 -->|"preference vector"| envelope_syn
    synthetic_train2 -->|"preference vector"| envelope_syn
    simulate -->|"preference vector"| envelope_dial
    run_e3c -->|"preference vector"| envelope_mario
    run_a3c -->|"preference vector"| envelope_mario
    run_n3c -->|"preference vector"| envelope_mario
    adapt_e -->|"preference vector"| envelope_mario
    adapt_n -->|"preference vector"| envelope_mario

    %% Domain Interfaces -> Replay Buffers
    dialogue_server -->|"trajectory data"| dial_buffer
    mario_env -->|"trajectory data"| mario_buffer

    %% Core MORL -> Model & Buffer
    envelope_syn -->|"updates"| linear_model
    envelope_dial -->|"updates"| dial_buffer
    envelope_mario -->|"updates"| mario_model
    envelope_mario -->|"updates"| mario_buffer

    %% Evaluations: Core MORL -> Evaluation
    envelope_syn -->|"logs & checkpoints"| plot_for_paper
    envelope_syn -->|"logs & checkpoints"| synthetic_scripts_eval
    envelope_dial -->|"logs & checkpoints"| dial_plot
    envelope_mario -->|"logs & checkpoints"| mario_eval_n3c
    envelope_mario -->|"logs & checkpoints"| mario_eval_sh

    %% Config usage
    synthetic_train1 --> synthetic_scripts_conf
    synthetic_train2 --> synthetic_scripts_conf
    simulate --> pydial_config1
    simulate --> pydial_config2
    run_e3c --> mario_local
    run_a3c --> mario_local
    run_n3c --> mario_local
    adapt_e --> mario_ws
    adapt_n --> mario_ws
    run_e3c --> mario_slurm
    run_a3c --> mario_slurm
    run_n3c --> mario_slurm

    %% Styles
    classDef algorithmCore fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef domain fill:#90EE90,stroke:#000,stroke-width:1px
    classDef infra fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef evaluation fill:#FFA500,stroke:#000,stroke-width:1px

    %% Click Events
    click envelope_syn "https://github.com/runzheyang/morl/blob/master/synthetic/crl/envelope/meta.py"
    click envelope_dial "https://github.com/runzheyang/morl/blob/master/pydial/policy/DMORL/envelope.py"
    click envelope_mario "https://github.com/runzheyang/morl/blob/master/multimario/agent.py"
    click deep_sea "https://github.com/runzheyang/morl/blob/master/synthetic/envs/deep_sea_treasure.py"
    click fruit_tree "https://github.com/runzheyang/morl/blob/master/synthetic/envs/fruit_tree.py"
    click mo_env "https://github.com/runzheyang/morl/blob/master/synthetic/envs/mo_env.py"
    click dialogue_server "https://github.com/runzheyang/morl/blob/master/pydial/DialogueServer.py"
    click mario_env "https://github.com/runzheyang/morl/blob/master/multimario/env.py"
    click linear_model "https://github.com/runzheyang/morl/blob/master/synthetic/crl/envelope/models/linear.py"
    click mario_model "https://github.com/runzheyang/morl/blob/master/multimario/model.py"
    click synthetic_train1 "https://github.com/runzheyang/morl/blob/master/synthetic/train.py"
    click synthetic_train2 "https://github.com/runzheyang/morl/blob/master/synthetic/roijers_train.py"
    click simulate "https://github.com/runzheyang/morl/blob/master/pydial/Simulate.py"
    click run_e3c "https://github.com/runzheyang/morl/blob/master/multimario/run_e3c.py"
    click run_a3c "https://github.com/runzheyang/morl/blob/master/multimario/run_a3c.py"
    click run_n3c "https://github.com/runzheyang/morl/blob/master/multimario/run_n3c.py"
    click adapt_e "https://github.com/runzheyang/morl/blob/master/multimario/adapt_e.py"
    click adapt_n "https://github.com/runzheyang/morl/blob/master/multimario/adapt_n.py"
    click dial_buffer "https://github.com/runzheyang/morl/blob/master/pydial/policy/DRL/replay_buffer.py"
    click plot_for_paper "https://github.com/runzheyang/morl/blob/master/synthetic/test/plot_for_paper.py"
    click synthetic_scripts_eval "https://github.com/runzheyang/morl/tree/master/synthetic/scripts/"
    click dial_plot "https://github.com/runzheyang/morl/blob/master/pydial/results/plot.py"
    click mario_eval_n3c "https://github.com/runzheyang/morl/blob/master/multimario/eval_n3c.py"
    click mario_eval_sh "https://github.com/runzheyang/morl/blob/master/multimario/scripts/eval.sh"
    click pydial_config1 "https://github.com/runzheyang/morl/tree/master/pydial/config/MORL"
    click pydial_config2 "https://github.com/runzheyang/morl/tree/master/pydial/_morlconfigs/"
    click synthetic_scripts_conf "https://github.com/runzheyang/morl/tree/master/synthetic/scripts/"
    click mario_local "https://github.com/runzheyang/morl/tree/master/multimario/scripts/_local/"
    click mario_ws "https://github.com/runzheyang/morl/tree/master/multimario/scripts/_workstation/"
    click mario_slurm "https://github.com/runzheyang/morl/tree/master/multimario/slurm/"
```
