# thu-rllab/cfcql architecture

> Architecture diagram of the GitHub repository thu-rllab/cfcql, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thu-rllab/cfcql
- Repository: https://github.com/thu-rllab/cfcql

## Overview

##### Overall Architecture Overview
- In both branches, the central common theme is offline multi-agent RL, where simulation environments, policy/algorithm modules, and training loops come together.

##### Main Components
a. Continuous Branch Components (under continuous/):
- Algorithms: Represent the algorithms (e.g., maddpg.py and maddpg_cc.py) as the core decision-making engines.
- Simulation Environments: Show two key environment folders:
▪ “multiagent-particle-envs” — this package contains core simulation (with components like environment, scenario definitions, and rendering for interacting agents).
▪ “multiagent_mujoco” — which holds assets (XML files for MuJoCo simulation) and simulation integration modules (mujoco_multi.py, multiagentenv.py).
- Main Entrypoint: Include the main.py script which likely manages the overall experiment or training loop for the continuous branch.

b. Discrete Branch Components (under discrete/):
- Setup and Run Scripts: Indicate scripts like clean.sh, run.sh, and setup directories (install/) for deployment and dependency setup.
- Source Code (“src”):
▪ Configurations: Mark the “config” folder (with YAML files for various algorithms and environment settings) that drives parameters for training and simulation.
▪ Components: Identify modules that encapsulate core components such as action selectors, episode buffers, epsilon schedules, etc.
▪ Learners: Show the diverse learners (e.g., awac_learner.py, iql_learner.py, etc.) which implement the specific learning algorithms.
▪ Controllers: Represent the “controllers” folder, where various policy controllers (e.g., basic_controller, conv_controller, etc.) are implemented.
▪ Environment Definitions: Show the “envs” folder including specialized environments (in particular, wrappers and interfacing for StarCraft II and custom environments) indicating that learning happens in simulated multi-agent settings.
▪ Modules: Highlight the “modules” folder that contains sub-modules for agents, critics, mixers, and layers – these likely form the building blocks for different network architectures and communication protocols among agents.
▪ Runners and Utilities: Include the “runners” and “utils” directories that contain the experiment runners (e.g., episode_runner) and supportive utilities (like logging, training utilities, data loaders).

##### Relationships and Data Flow
▪ Then show how “Agents” and “Controllers” interact with these buffers and feed into “Learners” that update policies.
▪ Finally the training loop (depicted in main entry scripts like main.py or discrete/src/main.py) coordinates these interactions.
- “Configuration” flows that drive parameter settings into both simulation environments and learner modules.
- Interactions from “Utils” that support network construction, noise handling, logging, and training tracking.

##### Architectural Patterns and Design Principles
- That both branches appear to follow a modular design which decouples simulation, learning algorithms, and utility functions.
- That the discrete branch is organized following a component-based architecture (with clear separation between configuration, control, simulation, and learning modules) while the continuous branch is a bit more lightweight but still follows a similar separation (algorithms vs. environment vs. utilities).
- Use Architecture patterns like MVC/MVVM in the sense that environment simulation (analogous to “model”), agents/layers (the “controller”) and the visualization/rendering (the “view”) are clearly separate.
- Make it clear that integration points (i.e., the training running scripts) serve to connect these components.

##### Final Remarks for the Principal Software Engineer
- Be detailed: list every logical layer (environment generation, algorithmic core, data buffering, utility services, etc.) and ensure that all inter-component relationships are clearly mapped.
- There’s no need to oversimplify—given the complexity of multi-agent RL and the separated nature of continuous versus discrete action implementations, it’s better to include as many components as possible.

## Diagram (Mermaid)

```mermaid
graph TD
    %% CFCQL System Architecture

    subgraph "Continuous Branch"
        ContinuousParticle["Particle Simulation Env"]:::simulation
        ContinuousMujoco["Mujoco Simulation Env"]:::simulation
        ContinuousAlgorithms["Algorithms (maddpg.py, maddpg_cc.py)"]:::algorithm
        ContinuousUtils["Utility Modules"]:::utility
        ContinuousTrainVAE["Train VAE"]:::utility
        ContinuousMain["Main Entrypoint"]:::default

        ContinuousParticle -->|"sendsData"| ContinuousAlgorithms
        ContinuousMujoco -->|"sendsData"| ContinuousAlgorithms
        ContinuousUtils -->|"supports"| ContinuousAlgorithms
        ContinuousTrainVAE -->|"assists"| ContinuousUtils
        ContinuousAlgorithms -->|"triggers"| ContinuousMain
    end

    subgraph "Discrete Branch"
        subgraph "Setup/Run Scripts"
            DiscreteClean["clean.sh"]:::setup
            DiscreteRun["run.sh"]:::setup
            DiscreteInstall["install"]:::setup
        end
        DiscreteConfigs["Configurations (YAML)"]:::config
        DiscreteComponents["Components"]:::component
        DiscreteLearners["Learners"]:::learner
        DiscreteControllers["Controllers"]:::controller
        DiscreteEnvDefs["Environment Definitions"]:::simulation
        DiscreteModules["Modules"]:::utility
        DiscreteRunners["Runners"]:::utility
        DiscreteUtils["Utilities"]:::utility
        DiscreteMain["Main Entrypoint"]:::default

        DiscreteClean --> DiscreteConfigs
        DiscreteRun --> DiscreteConfigs
        DiscreteInstall --> DiscreteConfigs

        DiscreteConfigs --> DiscreteComponents

        DiscreteComponents -->|"feedsTo"| DiscreteLearners
        DiscreteComponents -->|"feedsTo"| DiscreteControllers
        DiscreteComponents -->|"feedsTo"| DiscreteEnvDefs
        DiscreteComponents -->|"feedsTo"| DiscreteModules

        DiscreteLearners --> DiscreteRunners
        DiscreteControllers --> DiscreteRunners
        DiscreteEnvDefs --> DiscreteRunners
        DiscreteModules --> DiscreteRunners

        DiscreteLearners --> DiscreteUtils
        DiscreteControllers --> DiscreteUtils
        DiscreteEnvDefs --> DiscreteUtils
        DiscreteModules --> DiscreteUtils

        DiscreteRunners -->|"merges"| DiscreteMain
        DiscreteUtils -->|"merges"| DiscreteMain
    end

    %% Click Events for Continuous Branch
    click ContinuousAlgorithms "https://github.com/thu-rllab/cfcql/tree/main/continuous/algorithms"
    click ContinuousParticle "https://github.com/thu-rllab/cfcql/tree/main/continuous/multiagent-particle-envs"
    click ContinuousMujoco "https://github.com/thu-rllab/cfcql/tree/main/continuous/multiagent_mujoco"
    click ContinuousUtils "https://github.com/thu-rllab/cfcql/tree/main/continuous/utils"
    click ContinuousTrainVAE "https://github.com/thu-rllab/cfcql/blob/main/continuous/train_vae.py"
    click ContinuousMain "https://github.com/thu-rllab/cfcql/blob/main/continuous/main.py"

    %% Click Events for Discrete Branch
    click DiscreteClean "https://github.com/thu-rllab/cfcql/blob/main/discrete/clean.sh"
    click DiscreteRun "https://github.com/thu-rllab/cfcql/blob/main/discrete/run.sh"
    click DiscreteInstall "https://github.com/thu-rllab/cfcql/tree/main/discrete/install"
    click DiscreteConfigs "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/config"
    click DiscreteComponents "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/components"
    click DiscreteLearners "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/learners"
    click DiscreteControllers "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/controllers"
    click DiscreteEnvDefs "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/envs"
    click DiscreteModules "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/modules"
    click DiscreteRunners "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/runners"
    click DiscreteUtils "https://github.com/thu-rllab/cfcql/tree/main/discrete/src/utils"
    click DiscreteMain "https://github.com/thu-rllab/cfcql/blob/main/discrete/src/main.py"

    %% Styles
    classDef algorithm fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef simulation fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef utility fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef setup fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef config fill:#E6E6FA,stroke:#000,stroke-width:2px;
    classDef component fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef learner fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef controller fill:#F08080,stroke:#000,stroke-width:2px;
```
