# tomeu7/ao-marl architecture

> Architecture diagram of the GitHub repository tomeu7/ao-marl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tomeu7/ao-marl
- Repository: https://github.com/tomeu7/ao-marl

## Overview

##### Project Overview and Key Components
- The project implements adaptive optics control for telescopes using multi-agent reinforcement learning (MARL). Its purpose is to control a deformable mirror (and other optical parameters) via distributed RL agents, with the environment interacting with a simulation/control system (COMPASS via shesha).
- The README clearly shows that there are several “subsystems”: an RL training loop (in src/reinforcement_learning), a simulator/environment (ao_env.py), a supervisor interface (RlSupervisor.py) within the shesha package, and an auxiliary autoencoder module for denoising wavefront sensor images (in src/autoencoder). There is also an error budget analysis module and a guardians package that handles modifications related to ROKET.

##### Analyzing the File Structure
- Top-level directories:
– data: Contains simulation parameters, configuration files, and pre-generated simulation results.
– shesha: Contains modules that interface with COMPASS (e.g., supervisor scripts for the telescope simulation).
– guardians: Contains “ROKET” components, providing modifications and possibly extensions to the simulation/control system.
– src: Contains the core RL implementation including training modules, environment definition, helper functions for preprocessing (including delayed MDP), autoencoder, and error budgeting.
– main.py is used to start the training process.
- The RL code is organized into several components:
– reinforcement_learning: The MARL loop, environment wrapper (ao_env.py) that seems to “communicate” with COMPASS through the shesha supervisor interface.
– rpc_training: Implements a parallel training loop using Pytorch’s RPC framework.
– helper_functions: Preprocessing utilities (for state and action normalization, obtaining gains, etc.).
- The shesha package further includes the supervisor that bridges the RL environment and COMPASS.
- The autoencoder components and error_budget directory indicate additional analysis steps (for denoising sensor images and evaluating system performance).

##### Main Components
- RL Training Module:
- main.py as the entry point.
- src/reinforcement_learning modules including configuration, environment (ao_env.py), delayed MDP buffer, and SAC algorithm implementations (modified from external repository).
- The training loop in rpc_training which uses parallelism (via Pytorch RPC) for updating multiple agents.
- Environment and Simulator:
- The environment component built in ao_env.py that gathers state information.
- The interaction with shesha’s supervisor (e.g., RlSupervisor.py in shesha/supervisor) which communicates with COMPASS.
- Data/Configuration:
- The data folder that holds telescope parameter files and simulation layouts.
- Auxiliary Modules:
- Autoencoder (in src/autoencoder) for wavefront sensor image denoising, integrated optionally via an argument to the main training module.
- Error budget module (src/error_budget) for analyzing and plotting simulation results.
- Guardians package (with roket-related modifications) which appears to provide alternate or extended supervisory control.
- External Dependencies:
- Integration with COMPASS (via shesha package).
- Pytorch for reinforcement learning and SAC algorithms.
- Possibly external repositories such as the one from pranz24 for SAC, and the denoising autoencoder repository.

##### Interactions and Data Flows
- The flow from main.py to the training loop (src/reinforcement_learning/rpc_training/train_rpc.py) which orchestrates the learning process.
- Data flow from the RL environment (ao_env.py) to the training module where the state, action, reward tuples are processed (with help of the delayed MDP buffer).
- Communication between the RL training and the shesha supervisor components, which serve as an interface to the COMPASS simulation (e.g., the RlSupervisor).
- Integration of processed sensor images through the autoencoder module before being fed to the RL agent.
- Post-training evaluation and error analysis flowing into the error_budget module (saving to output/error_budget, generating h5 files, and producing analysis plots).

##### Architectural Patterns and Design Principles
- The project follows a modular design separating simulation data and configuration (data folder), environment interfacing (shesha/supervisor), main control logic (src/reinforcement_learning), and auxiliary utilities (autoencoder and preprocessing helpers). This separation reinforces a microservices-like modularity.
- Use of asynchronous/parallel processing in the training loop (RPC-based parallelism).
- That the design embodies external integration (with COMPASS) and extensibility (through modules such as guardians and autoencoder).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Control & Training Side
    subgraph "Control & Training Side"
        MainEntry["Main Entry (main.py)"]:::control
        RLTraining["RL Training Module"]:::control
        RPCLoop["RPC Training Loop (train_rpc.py)"]:::control
        RLConfig["Configuration & Environment Submodules"]:::control
        Autoencoder["Autoencoder Module"]:::aux
        Guardians["Guardians (ROKET)"]:::aux
    end

    %% Simulation & Environment Side
    subgraph "Simulation & Environment Side"
        AOEnv["Environment (ao_env.py)"]:::env
        Supervisor["COMPASS Interface (rlSupervisor.py)"]:::env
        Data["Data/Configuration (data/)"]:::data
        Params["Simulation Params (data/par/)"]:::data
        ErrorBudget["Error Budget Analysis (src/error_budget)"]:::aux
    end

    %% Data Flows and Relationships
    MainEntry -->|"start_training"| RLTraining
    RLTraining -->|"orchestrates"| RPCLoop
    RLTraining -->|"loads_config"| RLConfig
    RLConfig -->|"sends_state_observation"| AOEnv
    RPCLoop -->|"agent_interactions"| AOEnv
    AOEnv -->|"communicates_with"| Supervisor
    Supervisor -->|"feedback"| RLTraining
    Autoencoder -->|"processes_sensor_images"| RLConfig
    RLTraining -->|"outputs_results"| ErrorBudget
    Data -->|"provides_configuration"| RLConfig
    Data -->|"details_parameters"| Params
    Guardians -->|"modifies_control_flow"| Supervisor

    %% Click Events
    click MainEntry "https://github.com/tomeu7/ao-marl/blob/main/main.py"
    click RLTraining "https://github.com/tomeu7/ao-marl/tree/main/src/reinforcement_learning/"
    click RPCLoop "https://github.com/tomeu7/ao-marl/blob/main/src/reinforcement_learning/rpc_training/train_rpc.py"
    click AOEnv "https://github.com/tomeu7/ao-marl/blob/main/src/reinforcement_learning/environment/ao_env.py"
    click Supervisor "https://github.com/tomeu7/ao-marl/blob/main/shesha/supervisor/rlSupervisor.py"
    click Data "https://github.com/tomeu7/ao-marl/tree/main/data/"
    click Params "https://github.com/tomeu7/ao-marl/tree/main/data/par/"
    click Autoencoder "https://github.com/tomeu7/ao-marl/tree/main/src/autoencoder/"
    click ErrorBudget "https://github.com/tomeu7/ao-marl/tree/main/src/error_budget/"
    click Guardians "https://github.com/tomeu7/ao-marl/tree/main/guardians/"

    %% Styles
    classDef control fill:#AED6F1,stroke:#1F618D,stroke-width:2px
    classDef env fill:#F9E79F,stroke:#B7950B,stroke-width:2px
    classDef data fill:#A9DFBF,stroke:#1E8449,stroke-width:2px
    classDef aux fill:#F5B7B1,stroke:#922B21,stroke-width:2px
```
