How tomeu7/ao-marl works
Project Overview and Key Components
• The project implements adaptive optics control for telescopes using multi-agent reinforcement learning (MARL). Its purpose is to control a deformable mirror (and other optical parameters) via distributed RL agents, with the environment interacting with a simulation/control system (COMPASS via shesha).
• The README clearly shows that there are several “subsystems”: an RL training loop (in src/reinforcement_learning), a simulator/environment (ao_env.py), a supervisor interface (RlSupervisor.py) within the shesha package, and an auxiliary autoencoder module for denoising wavefront sensor images (in src/autoencoder). There is also an error budget analysis module and a guardians package that handles modifications related to ROKET.
Analyzing the File Structure
• Top-level directories:
– data: Contains simulation parameters, configuration files, and pre-generated simulation results.
– shesha: Contains modules that interface with COMPASS (e.g., supervisor scripts for the telescope simulation).
– guardians: Contains “ROKET” components, providing modifications and possibly extensions to the simulation/control system.
– src: Contains the core RL implementation including training modules, environment definition, helper functions for preprocessing (including delayed MDP), autoencoder, and error budgeting.
– main.py is used to start the training process.
• The RL code is organized into several components:
– reinforcement_learning: The MARL loop, environment wrapper (ao_env.py) that seems to “communicate” with COMPASS through the shesha supervisor interface.
– rpc_training: Implements a parallel training loop using Pytorch’s RPC framework.
– helper_functions: Preprocessing utilities (for state and action normalization, obtaining gains, etc.).
• The shesha package further includes the supervisor that bridges the RL environment and COMPASS.
• The autoencoder components and error_budget directory indicate additional analysis steps (for denoising sensor images and evaluating system performance).
Main Components
• RL Training Module:
• main.py as the entry point.
• src/reinforcement_learning modules including configuration, environment (ao_env.py), delayed MDP buffer, and SAC algorithm implementations (modified from external repository).
• The training loop in rpc_training which uses parallelism (via Pytorch RPC) for updating multiple agents.
• Environment and Simulator:
• The environment component built in ao_env.py that gathers state information.
• The interaction with shesha’s supervisor (e.g., RlSupervisor.py in shesha/supervisor) which communicates with COMPASS.
• Data/Configuration:
• The data folder that holds telescope parameter files and simulation layouts.
• Auxiliary Modules:
• Autoencoder (in src/autoencoder) for wavefront sensor image denoising, integrated optionally via an argument to the main training module.
• Error budget module (src/error_budget) for analyzing and plotting simulation results.
• Guardians package (with roket-related modifications) which appears to provide alternate or extended supervisory control.
• External Dependencies:
• Integration with COMPASS (via shesha package).
• Pytorch for reinforcement learning and SAC algorithms.
• Possibly external repositories such as the one from pranz24 for SAC, and the denoising autoencoder repository.
Interactions and Data Flows
• The flow from main.py to the training loop (src/reinforcement_learning/rpc_training/train_rpc.py) which orchestrates the learning process.
• Data flow from the RL environment (ao_env.py) to the training module where the state, action, reward tuples are processed (with help of the delayed MDP buffer).
• Communication between the RL training and the shesha supervisor components, which serve as an interface to the COMPASS simulation (e.g., the RlSupervisor).
• Integration of processed sensor images through the autoencoder module before being fed to the RL agent.
• Post-training evaluation and error analysis flowing into the error_budget module (saving to output/error_budget, generating h5 files, and producing analysis plots).
Architectural Patterns and Design Principles
• The project follows a modular design separating simulation data and configuration (data folder), environment interfacing (shesha/supervisor), main control logic (src/reinforcement_learning), and auxiliary utilities (autoencoder and preprocessing helpers). This separation reinforces a microservices-like modularity.
• Use of asynchronous/parallel processing in the training loop (RPC-based parallelism).
• That the design embodies external integration (with COMPASS) and extensibility (through modules such as guardians and autoencoder).