How riffe007/sacformer works
Main Components
• Agents: This folder contains the core SAC implementation – components such as the transformer-based policy (policy.py), twin Q-networks (q_network.py), replay buffer (replay_buffer.py), and the main SAC agent (sac_agent.py). Also included are training (train.py) and evaluation (evaluation.py) scripts that drive the learning process.
• Environments: Contains modules for integrating with different simulation systems – gym-based environments (gym_env.py), MuJoCo-specific setups (mujoco_envs.py), and vectorized environments (vec_env.py). These components represent the external world with which the agent interacts.
• Training Scripts: The training_scripts folder holds different training entry points – standard SAC training, multi-GPU training using Ray (sac_ray_train.py), and hyperparameter tuning (sac_hyperparam_search.py). This distinguishes between different running modes.
• Utilities: Provides support functions for configuration management (config.py), logging (logging.py with TensorBoard and WandB integration), and common helper routines (utils.py).
• Models & Logs: These directories handle saving trained models (models folder) and storing training logs (logs folder), ensuring persistent tracking of progress.
• Virtual Environment (sac-env): Although not a source-code component, it indicates environment setup for dependencies – which can be annotated as an “infrastructure” component.
Relationships and Data Flow
• The Training Scripts serve as the entry points that invoke the SAC agent.
• The Agent interacts with environments – receiving state inputs, executing actions, collecting experiences, and using the PER replay buffer to sample data.
• During training, models (SAC network weights and transformer parameters) are updated and periodically saved in the Models directory.
• Logs flow out from training (using logging.py) and are visualized by TensorBoard and WandB.
• Multi-GPU training integrates with Ray RLlib, indicating a distributed training layer that coordinates worker nodes and aggregates gradients.
Architectural Patterns
• Modular separation: Notice the clear division into “agents”, “environments”, “training_scripts”, and “utils” indicating a well-modularized design.
• Pipeline pattern: Data flows from environment simulation through the agent’s forward/backward passes, with iterative logging and checkpointing.
• Extensibility: The use of Transformer-based policies (replacing traditional MLPs) and the support for various training modes (standard, distributed, hyperparameter search) point to a design that can be easily extended.
• Multi-paradigm integration: Supports both traditional sequential training and parallelized, potentially distributed training with Ray.
Relevant Technologies, Frameworks, and Libraries
• Soft Actor-Critic (SAC) – Core RL algorithm enhanced with a Transformer-based actor.
• Transformer architectures – Replacing MLPs for better sequential decision-making.
• Gymnasium & MuJoCo – For high-quality physics simulation.
• Ray RLlib – For distributed, multi-GPU training.
• Stable-Baselines3 VecEnv – For rapid environment rollouts.
• TensorBoard & WandB – For real-time monitoring and logging.
Elements
• Use clear labels for each major component including “Agents”, “Environments”, “Training Scripts”, “Utilities”, “Models”, and “Logs”.
• The control flow from Training Scripts to Agents.
• Data flow between Agents and Environments (e.g., state inputs, actions, and rewards).
• Logging and model saving outputs to their respective directories.
• Core algorithm modules (e.g., policy, Q-networks, replay buffer).
• Environment interfaces (Gymnasium, MuJoCo, VecEnv).
• Distributed/parallel components (Ray RLlib integration).
• Supporting tools (logging, configuration, utilities).
• Optionally, include annotations that mention key design principles (modularity, extensibility, and parallelism).
Detailed Component Mapping
• Agents:
• SAC Agent (sac_agent.py) as the central controller.
• Transformer-based Policy (policy.py) for decision-making.
• Q-Networks (q_network.py) for value estimation.
• Replay Buffer (replay_buffer.py) for prioritized experience replay.
• Training and Evaluation scripts (train.py and evaluation.py).
• Environments:
• Gymnasium integration (gym_env.py).
• MuJoCo simulation wrapper (mujoco_envs.py).
• Vectorized environments (vec_env.py) for parallelism.
• Training Scripts:
• Standard training (sac_train.py).
• Distributed/multi-GPU training (sac_ray_train.py).
• Automated hyperparameter tuning (sac_hyperparam_search.py).
• Utilities & Infrastructure:
• Logging (logging.py) and configuration (config.py) for experiment management.
• Virtual Environment (sac-env) that ensures dependency isolation.
• Saved Models and Logs directories for outputs and checkpoints.