# theohhhu/updet architecture

> Architecture diagram of the GitHub repository theohhhu/updet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theohhhu/updet
- Repository: https://github.com/theohhhu/updet

## Overview

##### Project Type and Purpose
- This is a research/reinforcement learning framework – an implementation of multi-agent RL with transformers – that builds on PyMARL and SMAC. Its purpose is to experiment and train transformer-based agents (including standard UPDeT and aggregation transformer models) in environments derived from StarCraft II battles.
- The README confirms that the project is for multi-agent reinforcement learning with emphasis on policy decoupling, using different algorithm configurations (e.g., VDN, QMIX, QTRAN, COMA) specified via YAML files.

##### Analyzing the File Structure
- Top-Level Files/Directories:
- Files such as LICENSE, README.md, docker-related files (Dockerfile, build shell script), installation scripts, and run scripts indicate both deployment and local execution support.
- The requirements.txt confirms dependency management.
- The src/ folder is the heart of the project and is subdivided into several key components:
- src/config: Contains configurations for algorithms and environments (e.g., different YAML files for each algorithm and environment), as well as a default configuration file that likely ties together global parameters.
- src/components: Contains helper modules such as action selectors, episode buffers, epsilon schedules, and transforms.
- src/controllers: Contains controller code (basic_controller.py) which likely manages agent actions and input processing in the simulation.
- src/envs: Contains the multi-agent environment wrapper (e.g., multiagentenv.py) to interface the agents with the SMAC/SC2 environment.
- src/learners: Contains learner implementations for different algorithms (e.g., coma_learner, q_learner, qtran_learner) which probably perform the optimization to update policies.
- src/modules: Split into agents, critics, and mixers subdirectories. Here you see implementations for different types of agents (e.g., rnn_agent, transformer_agg_agent, updet_agent) as well as network modules that combine agent outputs (mixers for QMix, VDN, QTran, etc.).
- src/runners: Houses the episode runner and parallel runner to coordinate training episodes and simulations.
- src/utils: Contains utility modules (logging, time helper, dictionary to named tuple conversion, etc.).
- There’s also a main entry point in src/main.py and src/run.py for launching experiments.

To capture the project’s architecture clearly, consider these main components and their relationships:
a. Core Components:
- Configuration module(s): Loads algorithm (e.g., VDN, QMIX) and environment (SC2) YAML files from src/config.
- Environment Interface: The src/envs folder provides a wrapper for the multi-agent environment (multiagentenv.py); this component communicates with the external SMAC/SC2 environment.
- Controllers: The basic_controller (in src/controllers) is responsible for processing observations and making decisions (especially those related to transformer input modifications).
- Learners: The learning modules (in src/learners) implement various training methods which update the networks (value functions, policy networks).
- Agent Modules: Inside src/modules/agents, you have different types of agents including UPDeT variants that define the transformation-based decision-making architecture.
- Critics and Mixers: Found under src/modules/critics and src/modules/mixers – these components compute value functions and perform the necessary mixing/factorization for multi-agent coordination.
- Runners: These orchestrate the overall training loop by managing episodes, parallellizing simulation if necessary.
- Supporting Components: Components like episode buffers, epsilon schedulers, and transforms (in src/components) support training dynamics and exploration.
b. Relationships and Interactions:
- The Configuration modules feed parameters/settings into almost every other component.
- The Runner initializes the environment, which in turn provides observations to the Controller.
- The Controller dispatches these observations to the appropriate Agent modules (transformer-based agents) which then collaborate with the Learners.
- Learners update agent policies based on rewards and feedback from the Environment.
- Critic and Mixer modules support learning by evaluating actions and combining the contributions of individual agents.
- Utility modules aid logging, time management, and parameter conversion throughout the training process.
c. Architectural Patterns & Design Principles:
- The project follows a modular design: discrete directories and modules for configuration, environment handling, learning, agent decision-making, and training orchestration.
- The design supports flexibility through configuration files (YAML) to swap between different reinforcement learning algorithms and agent types.
- Emphasis on separation of concerns (different directories for controllers, learners, and agent modules) enhances experiment reproducibility and extendability.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraphs
    subgraph "Configuration"
        config["Configuration Loader"]:::config
    end

    subgraph "Environment Interface"
        env["Environment Wrapper"]:::env
    end

    subgraph "Control"
        ctrl["Basic Controller"]:::controller
    end

    subgraph "Agents"
        agent1["Agent - RNN"]:::agent
        agent2["Agent - Transformer Aggregation"]:::agent
        agent3["Agent - UPDeT"]:::agent
    end

    subgraph "Learning"
        learner1["Learner - COMA"]:::learner
        learner2["Learner - Q-Learning"]:::learner
        learner3["Learner - QTRAN"]:::learner
    end

    subgraph "Critic & Mixers"
        critic["Critic - COMA"]:::critic
        mixer1["Mixer - QMIX"]:::mixer
        mixer2["Mixer - QTRAN"]:::mixer
        mixer3["Mixer - VDN"]:::mixer
    end

    subgraph "Runners"
        runner1["Episode Runner"]:::runner
        runner2["Parallel Runner"]:::runner
    end

    subgraph "Supporting Components"
        support["Components & Utilities"]:::utility
    end

    subgraph "Entry Points"
        entry1["Main Entry Point"]:::entry
        entry2["Run Script"]:::entry
    end

    %% Connections
    config -->|"feeds"| env
    config -->|"feeds"| ctrl
    config -->|"feeds"| learner1
    config -->|"feeds"| agent1
    config -->|"feeds"| agent2
    config -->|"feeds"| agent3

    env -->|"inits"| runner1
    env -->|"inits"| runner2

    runner1 -->|"orchestrates"| ctrl
    runner2 -->|"orchestrates"| ctrl

    ctrl -->|"dispatches"| agent1
    ctrl -->|"dispatches"| agent2
    ctrl -->|"dispatches"| agent3

    agent1 -->|"trains"| learner1
    agent2 -->|"trains"| learner2
    agent3 -->|"trains"| learner3

    learner1 -->|"evaluates"| critic
    learner2 -->|"evaluates"| mixer1
    learner3 -->|"evaluates"| mixer2
    learner3 -->|"evaluates"| mixer3

    learner1 -- "updates" --> agent1
    learner2 -- "updates" --> agent2
    learner3 -- "updates" --> agent3

    support --- config
    support --- env
    support --- runner1
    support --- runner2
    support --- learner1
    support --- learner2
    support --- learner3

    entry1 -->|"launches"| config
    entry1 -->|"launches"| runner1
    entry2 -->|"launches"| config
    entry2 -->|"launches"| runner2

    %% Click Events
    click config "https://github.com/theohhhu/updet/tree/main/src/config"
    click env "https://github.com/theohhhu/updet/blob/main/src/envs/multiagentenv.py"
    click ctrl "https://github.com/theohhhu/updet/blob/main/src/controllers/basic_controller.py"
    click learner1 "https://github.com/theohhhu/updet/blob/main/src/learners/coma_learner.py"
    click learner2 "https://github.com/theohhhu/updet/blob/main/src/learners/q_learner.py"
    click learner3 "https://github.com/theohhhu/updet/blob/main/src/learners/qtran_learner.py"
    click agent1 "https://github.com/theohhhu/updet/blob/main/src/modules/agents/rnn_agent.py"
    click agent2 "https://github.com/theohhhu/updet/blob/main/src/modules/agents/transformer_agg_agent.py"
    click agent3 "https://github.com/theohhhu/updet/blob/main/src/modules/agents/updet_agent.py"
    click critic "https://github.com/theohhhu/updet/blob/main/src/modules/critics/coma.py"
    click mixer1 "https://github.com/theohhhu/updet/blob/main/src/modules/mixers/qmix.py"
    click mixer2 "https://github.com/theohhhu/updet/blob/main/src/modules/mixers/qtran.py"
    click mixer3 "https://github.com/theohhhu/updet/blob/main/src/modules/mixers/vdn.py"
    click runner1 "https://github.com/theohhhu/updet/blob/main/src/runners/episode_runner.py"
    click runner2 "https://github.com/theohhhu/updet/blob/main/src/runners/parallel_runner.py"
    click support "https://github.com/theohhhu/updet/tree/main/src/components"
    click entry1 "https://github.com/theohhhu/updet/blob/main/src/main.py"
    click entry2 "https://github.com/theohhhu/updet/blob/main/src/run.py"

    %% Styles
    classDef config fill:#f7c6c7,stroke:#b0413e,stroke-width:2px;
    classDef env fill:#c6e2ff,stroke:#31708f,stroke-width:2px;
    classDef controller fill:#fde2a9,stroke:#d58512,stroke-width:2px;
    classDef agent fill:#d2f8d2,stroke:#2e8b57,stroke-width:2px;
    classDef learner fill:#d0ebff,stroke:#1c7ed6,stroke-width:2px;
    classDef critic fill:#fff3bf,stroke:#f08c00,stroke-width:2px;
    classDef mixer fill:#ffe3e3,stroke:#e03131,stroke-width:2px;
    classDef runner fill:#ffc9de,stroke:#a61e4d,stroke-width:2px;
    classDef utility fill:#e0e0e0,stroke:#6c757d,stroke-width:2px;
    classDef entry fill:#d3d3f9,stroke:#495aff,stroke-width:2px;
```
