# cdm1619/networked-mb-marl architecture

> Architecture diagram of the GitHub repository cdm1619/networked-mb-marl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cdm1619/networked-mb-marl
- Repository: https://github.com/cdm1619/networked-mb-marl

## Overview

This project implements multiple reinforcement learning algorithms for large‐scale network control – including decentralized methods (e.g., DPPO, CPPO, DMPO, IA2C, IC3Net) and their corresponding model‐free and model‐based baselines – alongside several simulation environments (such as CACC Catchup, CACC Slowdown, ATSC Grid/Monaco/New York, Power-Grid, Real Power Net, Pandemic, and Large City). The file tree clearly indicates that the project is organized into several major components, each catering to either algorithm implementations, environment simulators, configuration files, or baseline methods.

##### Core Components
- RL Algorithms: Your primary learning algorithms are implemented in the “algorithms” directory. Notice separate submodules for agents, buffers, normalization, and the “main” runners for different simulation scenarios. The RL agents are built using an actor-critic style, with distinct modules that initialize different actors (policies) and critics (evaluations).
- Environment Simulators: The “algorithms/envs” directory contains multiple simulation environments. Examples include Sumo-based simulations for CACC, ATSC and Flow environments, as well as specialized simulators for PowerGrid, Real Power, and Pandemic networks. Each environment has its own configuration subdirectories and even further subdivisions (e.g., NCS, Flow, PandemicSimulator).
- Baseline Implementations: There are two baseline sets. One is in “commmunication-based-baselines” which covers communication-centric agents (e.g., CommNet, DIAL) and the other is “model-based-baselines” covering model-based approaches (e.g., MAG, Dreamer-based controllers and learners). These directories follow a similar structure, splitting agents, models, policies, and other utility modules.
- Configuration & Utilities: Notice the rich configuration files under “algorithms/config” and within environment-specific directories. They define key parameters such as communication radii and environment settings. Utility scripts (including logging via WandB) also play an important role in orchestrating experiments.
- Launcher & Evaluation: A central “launcher.py” serves as the entry point to choose the environment, algorithm, and device (e.g., “cuda:0”, “cpu”), and the project includes dedicated evaluation scripts (e.g., evaluate_atsc.py, evaluate_flow.py).

##### Relationships and Interactions
- The launcher connects the top-level process flow: it reads command-line arguments (environment, algorithm, device), selects the appropriate algorithm module (agent and policy) from “algorithms/algo/agent”, initializes the corresponding environment from “algorithms/envs” (or one of the baseline directories), and dispatches the training or evaluation process.
- Experiment configurations and simulation settings (e.g., Sumo settings) flow from configuration files into the environment modules, ensuring that the simulation, RL agent computations, and training hyperparameters are consistent.
- When using a baseline variant (communication-based or model-based), there is a clear dependency on separate submodules that encapsulate their own agent implementations while reusing environment definitions.
- Data flow: In training, the environment feeds state observations to the algorithm; the algorithm’s policies (actors) and value functions (critics) process these observations, generate actions, and update through experience stored in buffers. Training metrics and checkpoints are logged via WandB.

##### Architectural Patterns and Technologies
- Modular design: Independent modules for each algorithm, environment, baseline and configurations allow flexibility to switch between learning methods and simulation scenarios.
- Separation of concerns: Environment simulation (using Sumo for traffic, specialized code for Pandemic, etc.) is separate from agent learning code. In turn, algorithm specifics (actor, critic networks, model-based planning in DMPO) and baseline variants are neatly organized in separate directories.
- Integration with external tools: SUMO (for simulation) and WandB (for logging) are external dependencies that interface with the main control loop through dedicated integration points (e.g., environment variable initialization, logging modules).

##### Elements
- Launcher Module (launcher.py) as the entry point.
- RL Algorithms module: include sub-components for each learning algorithm (DPPO, CPPO, DMPO, etc.) with internal submodules for actors, critics, buffers.
- Environment Factory: nodes for each simulation type (e.g., Flow / SUMO, PandemicSimulator, PowerGrid, Real Power Net).
- Baseline Modules: separate nodes for communication-based and model-based baselines that interface with the primary algorithms.
- Configuration and Utilities: a node that aggregates config files (algorithms/config, environment.yml) and custom utilities (logging, normalization).
- External Dependencies: SUMO simulation environment, WandB logging service.
- The flow of parameters from the launcher to algorithm and environment modules.
- Data and feedback loops between environment outputs (observations, rewards) and algorithm inputs.
- Checkpoint and logging flows from training modules to WandB.
- Algorithm modules (perhaps in blue),
- Environment/simulation modules (in green),
- Baseline modules (in yellow),
- Configuration/utility components (in orange),
- External dependencies (in grey).
- Clear labels for each component, including the key parameters like “radius_v”, “radius_pi”, etc., where appropriate.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Launcher Module"]:::controller
    subgraph "RL Algorithms"
        B1["Agent Implementations"]:::algorithm
        B2["Buffer Module"]:::algorithm
        B3["Main Runner"]:::algorithm
        B4["Normalization Utils"]:::algorithm
    end
    subgraph "Environment Simulators"
        C1["General Environment"]:::environment
        C2["Pandemic Simulator"]:::environment
    end
    subgraph "Communication-Based Baselines"
        D1["Agent & Configurations"]:::baseline
    end
    subgraph "Model-Based Baselines"
        E1["Agent & Networks"]:::baseline
    end
    subgraph "Configuration & Utilities"
        F1["Algorithm Configs"]:::config
        F2["Utility Scripts"]:::config
    end
    G["SUMO Simulation Environment"]:::external
    H["WandB Logging Service"]:::external

    %% Connections from Launcher
    A -->|"reads"| F1
    A -->|"loads"| B1
    A -->|"initializes"| C1
    A -->|"selects"| D1
    A -->|"selects"| E1

    %% Configuration flows
    F1 -->|"provides"| B1
    F1 -->|"provides"| C1
    F1 -->|"provides"| D1
    F1 -->|"provides"| E1
    F2 -->|"supports"| B2

    %% Interaction between RL Algorithms and Environments
    B1 -->|"sendsActions"| C1
    C1 -->|"returnsObs"| B1

    %% Baseline modules interfacing with Environment
    D1 -->|"plugsInto"| C1
    E1 -->|"plugsInto"| C1

    %% Logging flows
    B1 -->|"logsMetrics"| H
    C1 -->|"logsMetrics"| H
    D1 -->|"logsMetrics"| H
    E1 -->|"logsMetrics"| H

    %% Environment interfacing with external simulation engine
    C1 -->|"interfaces"| G

    %% Click Events
    click A "https://github.com/cdm1619/networked-mb-marl/blob/master/launcher.py"
    click B1 "https://github.com/cdm1619/networked-mb-marl/tree/master/algorithms/algo/agent"
    click B2 "https://github.com/cdm1619/networked-mb-marl/blob/master/algorithms/algo/buffer.py"
    click B3 "https://github.com/cdm1619/networked-mb-marl/blob/master/algorithms/algo/main.py"
    click B4 "https://github.com/cdm1619/networked-mb-marl/blob/master/algorithms/algo/normalization_utils.py"
    click C1 "https://github.com/cdm1619/networked-mb-marl/tree/master/algorithms/envs"
    click C2 "https://github.com/cdm1619/networked-mb-marl/tree/master/algorithms/envs/PandemicSimulator"
    click D1 "https://github.com/cdm1619/networked-mb-marl/tree/master/commmunication-based-baselines"
    click E1 "https://github.com/cdm1619/networked-mb-marl/tree/master/model-based-baselines"
    click F1 "https://github.com/cdm1619/networked-mb-marl/tree/master/algorithms/config"
    click F2 "https://github.com/cdm1619/networked-mb-marl/blob/master/algorithms/utils.py"

    %% Styles
    classDef controller fill:#E6E6FA,stroke:#800080,stroke-width:2px;
    classDef algorithm fill:#ADD8E6,stroke:#0000FF,stroke-width:2px;
    classDef environment fill:#90EE90,stroke:#008000,stroke-width:2px;
    classDef baseline fill:#FFFF99,stroke:#FFD700,stroke-width:2px;
    classDef config fill:#FFDAB9,stroke:#FF8C00,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#808080,stroke-width:2px;
```
