How cdm1619/networked-mb-marl works
This project implements multiple reinforcement learning algorithms for large‐scale network control – including decentralized methods (e.g., DPPO, CPPO, DMPO, IA2C, IC3Net) and their corresponding model‐free and model‐based baselines – alongside several simulation environments (such as CACC Catchup, CACC Slowdown, ATSC Grid/Monaco/New York, Power-Grid, Real Power Net, Pandemic, and Large City). The file tree clearly indicates that the project is organized into several major components, each catering to either algorithm implementations, environment simulators, configuration files, or baseline methods.
Core Components
• RL Algorithms: Your primary learning algorithms are implemented in the “algorithms” directory. Notice separate submodules for agents, buffers, normalization, and the “main” runners for different simulation scenarios. The RL agents are built using an actor-critic style, with distinct modules that initialize different actors (policies) and critics (evaluations).
• Environment Simulators: The “algorithms/envs” directory contains multiple simulation environments. Examples include Sumo-based simulations for CACC, ATSC and Flow environments, as well as specialized simulators for PowerGrid, Real Power, and Pandemic networks. Each environment has its own configuration subdirectories and even further subdivisions (e.g., NCS, Flow, PandemicSimulator).
• Baseline Implementations: There are two baseline sets. One is in “commmunication-based-baselines” which covers communication-centric agents (e.g., CommNet, DIAL) and the other is “model-based-baselines” covering model-based approaches (e.g., MAG, Dreamer-based controllers and learners). These directories follow a similar structure, splitting agents, models, policies, and other utility modules.
• Configuration & Utilities: Notice the rich configuration files under “algorithms/config” and within environment-specific directories. They define key parameters such as communication radii and environment settings. Utility scripts (including logging via WandB) also play an important role in orchestrating experiments.
• Launcher & Evaluation: A central “launcher.py” serves as the entry point to choose the environment, algorithm, and device (e.g., “cuda:0”, “cpu”), and the project includes dedicated evaluation scripts (e.g., evaluate_atsc.py, evaluate_flow.py).
Relationships and Interactions
• The launcher connects the top-level process flow: it reads command-line arguments (environment, algorithm, device), selects the appropriate algorithm module (agent and policy) from “algorithms/algo/agent”, initializes the corresponding environment from “algorithms/envs” (or one of the baseline directories), and dispatches the training or evaluation process.
• Experiment configurations and simulation settings (e.g., Sumo settings) flow from configuration files into the environment modules, ensuring that the simulation, RL agent computations, and training hyperparameters are consistent.
• When using a baseline variant (communication-based or model-based), there is a clear dependency on separate submodules that encapsulate their own agent implementations while reusing environment definitions.
• Data flow: In training, the environment feeds state observations to the algorithm; the algorithm’s policies (actors) and value functions (critics) process these observations, generate actions, and update through experience stored in buffers. Training metrics and checkpoints are logged via WandB.
Architectural Patterns and Technologies
• Modular design: Independent modules for each algorithm, environment, baseline and configurations allow flexibility to switch between learning methods and simulation scenarios.
• Separation of concerns: Environment simulation (using Sumo for traffic, specialized code for Pandemic, etc.) is separate from agent learning code. In turn, algorithm specifics (actor, critic networks, model-based planning in DMPO) and baseline variants are neatly organized in separate directories.
• Integration with external tools: SUMO (for simulation) and WandB (for logging) are external dependencies that interface with the main control loop through dedicated integration points (e.g., environment variable initialization, logging modules).
Elements
• Launcher Module (launcher.py) as the entry point.
• RL Algorithms module: include sub-components for each learning algorithm (DPPO, CPPO, DMPO, etc.) with internal submodules for actors, critics, buffers.
• Environment Factory: nodes for each simulation type (e.g., Flow / SUMO, PandemicSimulator, PowerGrid, Real Power Net).
• Baseline Modules: separate nodes for communication-based and model-based baselines that interface with the primary algorithms.
• Configuration and Utilities: a node that aggregates config files (algorithms/config, environment.yml) and custom utilities (logging, normalization).
• External Dependencies: SUMO simulation environment, WandB logging service.
• The flow of parameters from the launcher to algorithm and environment modules.
• Data and feedback loops between environment outputs (observations, rewards) and algorithm inputs.
• Checkpoint and logging flows from training modules to WandB.
• Algorithm modules (perhaps in blue),
• Environment/simulation modules (in green),
• Baseline modules (in yellow),
• Configuration/utility components (in orange),
• External dependencies (in grey).
• Clear labels for each component, including the key parameters like “radius_v”, “radius_pi”, etc., where appropriate.