# riffe007/sacformer architecture

> Architecture diagram of the GitHub repository riffe007/sacformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/riffe007/sacformer
- Repository: https://github.com/riffe007/sacformer

## Overview

##### Main Components
- Agents: This folder contains the core SAC implementation – components such as the transformer-based policy (policy.py), twin Q-networks (q_network.py), replay buffer (replay_buffer.py), and the main SAC agent (sac_agent.py). Also included are training (train.py) and evaluation (evaluation.py) scripts that drive the learning process.
- Environments: Contains modules for integrating with different simulation systems – gym-based environments (gym_env.py), MuJoCo-specific setups (mujoco_envs.py), and vectorized environments (vec_env.py). These components represent the external world with which the agent interacts.
- Training Scripts: The training_scripts folder holds different training entry points – standard SAC training, multi-GPU training using Ray (sac_ray_train.py), and hyperparameter tuning (sac_hyperparam_search.py). This distinguishes between different running modes.
- Utilities: Provides support functions for configuration management (config.py), logging (logging.py with TensorBoard and WandB integration), and common helper routines (utils.py).
- Models & Logs: These directories handle saving trained models (models folder) and storing training logs (logs folder), ensuring persistent tracking of progress.
- Virtual Environment (sac-env): Although not a source-code component, it indicates environment setup for dependencies – which can be annotated as an “infrastructure” component.

##### Relationships and Data Flow
- The Training Scripts serve as the entry points that invoke the SAC agent.
- The Agent interacts with environments – receiving state inputs, executing actions, collecting experiences, and using the PER replay buffer to sample data.
- During training, models (SAC network weights and transformer parameters) are updated and periodically saved in the Models directory.
- Logs flow out from training (using logging.py) and are visualized by TensorBoard and WandB.
- Multi-GPU training integrates with Ray RLlib, indicating a distributed training layer that coordinates worker nodes and aggregates gradients.

##### Architectural Patterns
- Modular separation: Notice the clear division into “agents”, “environments”, “training_scripts”, and “utils” indicating a well-modularized design.
- Pipeline pattern: Data flows from environment simulation through the agent’s forward/backward passes, with iterative logging and checkpointing.
- Extensibility: The use of Transformer-based policies (replacing traditional MLPs) and the support for various training modes (standard, distributed, hyperparameter search) point to a design that can be easily extended.
- Multi-paradigm integration: Supports both traditional sequential training and parallelized, potentially distributed training with Ray.

##### Relevant Technologies, Frameworks, and Libraries
- Soft Actor-Critic (SAC) – Core RL algorithm enhanced with a Transformer-based actor.
- Transformer architectures – Replacing MLPs for better sequential decision-making.
- Gymnasium & MuJoCo – For high-quality physics simulation.
- Ray RLlib – For distributed, multi-GPU training.
- Stable-Baselines3 VecEnv – For rapid environment rollouts.
- TensorBoard & WandB – For real-time monitoring and logging.

##### Elements
- Use clear labels for each major component including “Agents”, “Environments”, “Training Scripts”, “Utilities”, “Models”, and “Logs”.
- The control flow from Training Scripts to Agents.
- Data flow between Agents and Environments (e.g., state inputs, actions, and rewards).
- Logging and model saving outputs to their respective directories.
- Core algorithm modules (e.g., policy, Q-networks, replay buffer).
- Environment interfaces (Gymnasium, MuJoCo, VecEnv).
- Distributed/parallel components (Ray RLlib integration).
- Supporting tools (logging, configuration, utilities).
- Optionally, include annotations that mention key design principles (modularity, extensibility, and parallelism).

##### Detailed Component Mapping
- Agents:
- SAC Agent (sac_agent.py) as the central controller.
- Transformer-based Policy (policy.py) for decision-making.
- Q-Networks (q_network.py) for value estimation.
- Replay Buffer (replay_buffer.py) for prioritized experience replay.
- Training and Evaluation scripts (train.py and evaluation.py).
- Environments:
- Gymnasium integration (gym_env.py).
- MuJoCo simulation wrapper (mujoco_envs.py).
- Vectorized environments (vec_env.py) for parallelism.
- Training Scripts:
- Standard training (sac_train.py).
- Distributed/multi-GPU training (sac_ray_train.py).
- Automated hyperparameter tuning (sac_hyperparam_search.py).
- Utilities & Infrastructure:
- Logging (logging.py) and configuration (config.py) for experiment management.
- Virtual Environment (sac-env) that ensures dependency isolation.
- Saved Models and Logs directories for outputs and checkpoints.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraph: Training Scripts
    subgraph "Training Scripts"
        Standard["Standard Training"]:::train
        Ray_Train["Ray Training"]:::train
        Hyper_Tune["Hyperparameter Tuning"]:::train
    end

    %% Subgraph: Agents
    subgraph "Agents"
        SAC_Agent["SAC Agent"]:::agent
        Policy["Transformer Policy"]:::agent
        Q_Network["Twin Q-Networks"]:::agent
        Replay_Buffer["Replay Buffer"]:::agent
        Agent_Train["Agent Training"]:::agent
        Evaluation["Evaluation Script"]:::agent
    end

    %% Subgraph: Environments
    subgraph "Environments"
        Gym_Env["Gym Env"]:::env
        Mujoco_Env["MuJoCo Env"]:::env
        Vec_Env["Vectorized Env"]:::env
    end

    %% Subgraph: Utilities
    subgraph "Utilities"
        Config["Config"]:::util
        Logging["Logging"]:::util
        Utils["Utils"]:::util
    end

    %% Other Components
    Models["Saved Models"]:::storage
    Logs["Training Logs"]:::storage
    Virtual_Env["Virtual Env"]:::infra

    %% External Dependencies
    Ray_RLlib["Ray RLlib"]:::external
    Gymnasium["Gymnasium"]:::external
    MuJoCo["MuJoCo"]:::external
    TensorBoard["TensorBoard"]:::external
    WandB["WandB"]:::external

    %% Relationships: Training Scripts to Agents
    Standard -->|"initiates"| SAC_Agent
    Ray_Train -->|"initiates"| SAC_Agent
    Hyper_Tune -->|"initiates"| SAC_Agent

    %% Relationships: Agents internal modules
    SAC_Agent -->|"uses"| Policy
    SAC_Agent -->|"uses"| Q_Network
    SAC_Agent -->|"uses"| Replay_Buffer
    SAC_Agent -->|"triggers"| Agent_Train
    SAC_Agent -->|"evaluated by"| Evaluation

    %% Relationships: Agents interacting with Environments
    SAC_Agent -->|"receives state"| Gym_Env
    SAC_Agent -->|"receives state"| Mujoco_Env
    SAC_Agent -->|"receives state"| Vec_Env
    Gym_Env -->|"simulated-by"| Gymnasium
    Mujoco_Env -->|"simulated-by"| MuJoCo

    %% Relationships: Agents output to Models and Logs
    SAC_Agent -->|"saves models"| Models
    SAC_Agent -->|"logs progress"| Logging
    Logging -->|"writes logs"| Logs
    Logging -->|"reports to"| TensorBoard
    Logging -->|"reports to"| WandB

    %% Relationship: Distributed training with Ray
    Ray_Train -->|"distributes via"| Ray_RLlib

    %% Virtual Environment supports core components (dotted connections)
    Virtual_Env --- SAC_Agent
    Virtual_Env --- Gym_Env
    Virtual_Env --- Config
    Virtual_Env --- Logging

    %% Class Definitions for Colors
    classDef agent fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef env fill:#ABEBC6,stroke:#229954,stroke-width:2px;
    classDef train fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef util fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef storage fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;
    classDef external fill:#FAD7A0,stroke:#AF601A,stroke-width:2px;
    classDef infra fill:#D6EAF8,stroke:#2E86C1,stroke-width:2px;

    %% Click Events for Agents
    click SAC_Agent "https://github.com/riffe007/sacformer/blob/main/agents/sac_agent.py"
    click Policy "https://github.com/riffe007/sacformer/blob/main/agents/policy.py"
    click Q_Network "https://github.com/riffe007/sacformer/blob/main/agents/q_network.py"
    click Replay_Buffer "https://github.com/riffe007/sacformer/blob/main/agents/replay_buffer.py"
    click Agent_Train "https://github.com/riffe007/sacformer/blob/main/agents/train.py"
    click Evaluation "https://github.com/riffe007/sacformer/blob/main/agents/evaluation.py"

    %% Click Events for Environments
    click Gym_Env "https://github.com/riffe007/sacformer/blob/main/environments/gym_env.py"
    click Mujoco_Env "https://github.com/riffe007/sacformer/blob/main/environments/mujoco_envs.py"
    click Vec_Env "https://github.com/riffe007/sacformer/blob/main/environments/vec_env.py"

    %% Click Events for Training Scripts
    click Standard "https://github.com/riffe007/sacformer/blob/main/training_scripts/sac_train.py"
    click Ray_Train "https://github.com/riffe007/sacformer/blob/main/training_scripts/sac_ray_train.py"
    click Hyper_Tune "https://github.com/riffe007/sacformer/blob/main/training_scripts/sac_hyperparam_search.py"

    %% Click Events for Utilities
    click Config "https://github.com/riffe007/sacformer/blob/main/utils/config.py"
    click Logging "https://github.com/riffe007/sacformer/blob/main/utils/logging.py"
    click Utils "https://github.com/riffe007/sacformer/blob/main/utils/utils.py"

    %% Click Events for Models, Logs, and Virtual Env
    click Models "https://github.com/riffe007/sacformer/tree/main/models/"
    click Logs "https://github.com/riffe007/sacformer/tree/main/logs/"
    click Virtual_Env "https://github.com/riffe007/sacformer/tree/main/sac-env/"
```
