How zhaozijie2022/mg2l works
Project Type and Purpose
• This is a research‐oriented multiagent reinforcement learning (MARL) framework implementing the MG2L algorithm (Meta Learning Task Representation in Multi-Agent RL).
• It provides:
• Environment wrappers for MPE, MuJoCo, RWARE, MAgent
• A training loop (train.py + learner.py)
• Core algorithm implementations (mg2l, MAPPO, actor‐critic, etc.)
• Autoencoder‐based task encoders (global & local inference)
• Replay buffers, configuration management, utilities
Top-Level Components
• train.py & learner.py – orchestrate experiments, parse configs, launch training.
• mg2l/algos/ – RL algorithm implementations and policy networks.
• mg2l/ae/ – task encoder modules (global & local autoencoders).
• mg2l/buffer/ – experience buffers (AE buffer, RL buffer).
• mg2l/envs/ – environment factory + wrappers for different simulators.
• mg2l/config/ – hierarchical configurations (algo, env, experiment).
• mg2l/utils/ – shared helpers (PyTorch utilities, network builders, config loaders).
High-Level Data Flow & Interactions
1. Configuration Loading
• experiment and algorithm parameters loaded via OmegaConf from mg2l/config/.
2. Environment Creation
• make_env.py instantiates vectorized multiagent environments (MPE, MuJoCo, RWARE).
3. Policy & Encoder Construction
• make_algo.py builds policy nets, value nets, and attaches the MG2L encoders (mg2l/ae/make_ae.py).
4. Experience Collection
• Agents interact with envs; raw observations & actions stored in rl_buffer.py.
• Encoded representations (global & local) stored in ae_buffer.py for encoder training.
5. Learning Loop
• RL Update: sample from RL buffer → compute policy & value losses → backpropagate.
• AE Update: sample from AE buffer → compute mutual information objective → update encoder.
• Parameter Sync: updated parameters pushed back to interaction workers.
6. Logging & Checkpointing
• Utilities handle metric tracking, model saving.
Architectural Patterns & Design Principles
• Modular Pipeline: clear separation between envs, buffers, algorithm, and encoders.
• Factory Methods: make_env.py, make_buffer.py, make_algo.py, make_ae.py.
• Config‐Driven: experiments driven entirely by YAML configs (no hard‐coding).
• Decoupled Encoders: global vs local AE modules for meta‐learning.