# zhaozijie2022/mg2l architecture

> Architecture diagram of the GitHub repository zhaozijie2022/mg2l, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhaozijie2022/mg2l
- Repository: https://github.com/zhaozijie2022/mg2l

## Overview

##### Project Type and Purpose
- This is a research‐oriented multiagent reinforcement learning (MARL) framework implementing the MG2L algorithm (Meta Learning Task Representation in Multi-Agent RL).
- It provides:
- Environment wrappers for MPE, MuJoCo, RWARE, MAgent
- A training loop (train.py + learner.py)
- Core algorithm implementations (mg2l, MAPPO, actor‐critic, etc.)
- Autoencoder‐based task encoders (global & local inference)
- Replay buffers, configuration management, utilities

##### Top-Level Components
- **train.py & learner.py** – orchestrate experiments, parse configs, launch training.
- **mg2l/algos/** – RL algorithm implementations and policy networks.
- **mg2l/ae/** – task encoder modules (global & local autoencoders).
- **mg2l/buffer/** – experience buffers (AE buffer, RL buffer).
- **mg2l/envs/** – environment factory + wrappers for different simulators.
- **mg2l/config/** – hierarchical configurations (algo, env, experiment).
- **mg2l/utils/** – shared helpers (PyTorch utilities, network builders, config loaders).

##### High-Level Data Flow & Interactions
1. **Configuration Loading**
- experiment and algorithm parameters loaded via OmegaConf from `mg2l/config/`.
2. **Environment Creation**
- `make_env.py` instantiates vectorized multiagent environments (MPE, MuJoCo, RWARE).
3. **Policy & Encoder Construction**
- `make_algo.py` builds policy nets, value nets, and attaches the MG2L encoders (`mg2l/ae/make_ae.py`).
4. **Experience Collection**
- Agents interact with envs; raw observations & actions stored in `rl_buffer.py`.
- Encoded representations (global & local) stored in `ae_buffer.py` for encoder training.
5. **Learning Loop**
- **RL Update**: sample from RL buffer → compute policy & value losses → backpropagate.
- **AE Update**: sample from AE buffer → compute mutual information objective → update encoder.
- **Parameter Sync**: updated parameters pushed back to interaction workers.
6. **Logging & Checkpointing**
- Utilities handle metric tracking, model saving.

##### Architectural Patterns & Design Principles
- **Modular Pipeline**: clear separation between envs, buffers, algorithm, and encoders.
- **Factory Methods**: `make_env.py`, `make_buffer.py`, `make_algo.py`, `make_ae.py`.
- **Config‐Driven**: experiments driven entirely by YAML configs (no hard‐coding).
- **Decoupled Encoders**: global vs local AE modules for meta‐learning.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraphs for main components

    subgraph "Configuration Management"
        direction TB
        AlgoConfig["algo_config/ mg2l.yaml"]:::config
        EnvConfig["env_config/ ...yaml"]:::config
        ExptConfig["expt_config/ default.yaml"]:::config
    end

    subgraph "Environment Suite"
        direction TB
        MakeEnv["make_env.py"]:::compute
        Wrappers["wrappers.py"]:::compute
        subgraph "MPE Envs"
            direction TB
            MPEEnv["mpe/ ..."]:::external
        end
        subgraph "MuJoCo Envs"
            direction TB
            MujocoEnv["mujoco/ ..."]:::external
        end
        RWAREEnv["rware_layout.py"]:::external
    end

    subgraph "Algorithm Core"
        direction TB
        MakeAlgo["make_algo.py"]:::compute
        MAPPO["mappo.py"]:::compute
        RAC["r_actor_critic.py"]:::compute
        subgraph "Algo Utils"
            direction TB
            Act["act.py"]:::compute
            CNN["cnn.py"]:::compute
            Dist["distributions.py"]:::compute
            MLP["mlp.py"]:::compute
            PopArt["popart.py"]:::compute
            RNN["rnn.py"]:::compute
            UtilAlgo["util.py"]:::compute
        end
    end

    subgraph "Encoder Module"
        direction TB
        MakeAE["make_ae.py"]:::compute
        Encoders["encoders.py"]:::compute
    end

    subgraph "Data Buffers"
        direction TB
        MakeBuffer["make_buffer.py"]:::compute
        RLBuffer["rl_buffer.py"]:::data
        AEBuffer["ae_buffer.py"]:::data
    end

    subgraph "Training Loop"
        direction TB
        TrainPy["train.py"]:::compute
        LearnerPy["learner.py"]:::compute
    end

    subgraph "Utilities & Logging"
        direction TB
        ConfigUtils["config_utils.py"]:::compute
        Networks["networks.py"]:::compute
        TorchUtils["pytorch_utils.py"]:::compute
        Util["util.py"]:::compute
        ValNorm["valuenorm.py"]:::compute
    end

    subgraph "External Dependencies"
        direction TB
        Gym["Gym"]:::external
        MuJoCoPy["mujoco_py"]:::external
        RWARESvc["RWARE"]:::external
    end

    %% Data Flow Connections
    AlgoConfig -->|loads| MakeEnv
    EnvConfig -->|loads| MakeEnv
    ExptConfig -->|loads| MakeEnv

    AlgoConfig -->|loads| MakeAlgo
    ExptConfig -->|loads| MakeAlgo
    AlgoConfig -->|loads| MakeAE

    MakeEnv -->|instantiates envs| RLBuffer
    MakeEnv -->|instantiates envs| AEBuffer

    RLBuffer -->|samples| MakeAlgo
    MakeAlgo -->|policy/value nets| TrainPy
    AEBuffer -->|samples| MakeAE
    MakeAE -->|encoders| TrainPy

    TrainPy -->|orchestrates| LearnerPy
    LearnerPy -->|update params| MakeAlgo
    LearnerPy -->|update params| MakeAE
    LearnerPy -->|logs/checkpoints| ConfigUtils

    MakeEnv -->|uses| Gym
    MakeEnv -->|uses| MuJoCoPy
    MakeEnv -->|uses| RWARESvc

    %% Click Events
    click TrainPy "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/train.py"
    click LearnerPy "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/learner.py"
    click MakeAlgo "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/make_algo.py"
    click MAPPO "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/mappo.py"
    click RAC "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/r_actor_critic.py"
    click Act "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/algo_utils/act.py"
    click CNN "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/algo_utils/cnn.py"
    click Dist "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/algo_utils/distributions.py"
    click MLP "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/algo_utils/mlp.py"
    click PopArt "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/algo_utils/popart.py"
    click RNN "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/algo_utils/rnn.py"
    click UtilAlgo "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/algos/algo_utils/util.py"
    click MakeAE "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/ae/make_ae.py"
    click Encoders "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/ae/encoders.py"
    click MakeBuffer "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/buffer/make_buffer.py"
    click RLBuffer "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/buffer/rl_buffer.py"
    click AEBuffer "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/buffer/ae_buffer.py"
    click MakeEnv "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/envs/make_env.py"
    click Wrappers "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/envs/wrappers.py"
    click MPEEnv "https://github.com/zhaozijie2022/mg2l/tree/master/mg2l/envs/mpe/"
    click MujocoEnv "https://github.com/zhaozijie2022/mg2l/tree/master/mg2l/envs/mujoco/"
    click RWAREEnv "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/envs/rware/rware_layout.py"
    click AlgoConfig "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/config/algo_config/mg2l.yaml"
    click EnvConfig "https://github.com/zhaozijie2022/mg2l/tree/master/mg2l/config/env_config/"
    click ExptConfig "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/config/expt_config/default.yaml"
    click ConfigUtils "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/utils/config_utils.py"
    click Networks "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/utils/networks.py"
    click TorchUtils "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/utils/pytorch_utils.py"
    click Util "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/utils/util.py"
    click ValNorm "https://github.com/zhaozijie2022/mg2l/blob/master/mg2l/utils/valuenorm.py"

    %% Styles
    classDef config fill:#bddfff,stroke:#333
    classDef compute fill:#b7e4c7,stroke:#333
    classDef data fill:#fff3b0,stroke:#333
    classDef external fill:#d3d3d3,stroke:#333
    classDef io fill:#ffd7d7,stroke:#333
```
