# graliuce/sgcrl architecture

> Architecture diagram of the GitHub repository graliuce/sgcrl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/graliuce/sgcrl
- Repository: https://github.com/graliuce/sgcrl

## Overview

This project is a reinforcement learning (RL) application designed to train agents using a contrastive learning setup – specifically, it implements the Single-goal Contrastive RL (SGCRL) method. The project is tailored toward robotics or simulated robotics tasks (e.g., sawyer_bin, sawyer_box, sawyer_peg environments) and leverages contrastive methods to guide the training process.

##### Main Components of the System
- Entry Point & Execution Flow:
- The main entry script is “lp_contrastive.py” which serves as the launcher to initiate training.
- Other executable scripts like “default.py”, “distributional.py”, “env_utils.py”, and “point_env.py” indicate various runtime utilities and environment setups.
- Core RL Modules (inside the “contrastive” folder):
- agents.py: Defines the RL agents and their behavior.
- learning.py: Contains the learning loop and core RL algorithm logic (contrastive reinforcement learning).
- networks.py: Houses the definitions of neural network architectures employed by the agents.
- config.py: Stores configuration details that tailor the agent behavior and experimental setups.
- builder.py: Likely responsible for constructing the components (agents, networks, etc.) based on the configuration.
- distributed_layout.py: Implements support for distributed and possibly parallel execution across GPUs or CPUs.
- utils.py: Provides helper functions or common utilities used throughout the project.
- Additional Components:
- Training video resources: Videos (e.g., sawyer_bin.mp4, sawyer_box.mp4, etc.) and zipped training videos in the “training_videos” folder. These resources help demonstrate training results or provide visual feedback of the experiments.
- Environment wrappers: Files like “env_utils.py” and “point_env.py” help integrate with simulation environments or task-specific settings.

##### Out Relationships and Interactions
- Control Flow:
- The entry point “lp_contrastive.py” initiates execution by parsing command-line flags (e.g., environment type, algorithm type, number of steps) as defined in the README.
- It loads configurations (from config.py) and uses the builder (builder.py) to instantiate core components such as agents and policy networks.
- Data Flow & Dependencies:
- The agent (implemented in agents.py and supported by learning.py) interacts with the environment (initialized via env_utils.py or point_env.py) to gather experiences.
- Neural network modules (networks.py) process state and action information, feeding into the learning loop where gradients and updates are computed.
- In distributed setups (handled by distributed_layout.py), multiple sub-components may communicate for parallel computations.
- External Dependencies & Integrations:
- External libraries (e.g., JAX, PyTorch, dm-haiku, optax) are integrated as outlined in the README. Their roles in acceleration (GPU support) and numerical computation must be clearly indicated.
- Mujoco is a key library for rendering and simulation physics, linked via environment setup instructions.

##### Architectural Patterns and Design Principles
- Modularity: The code is organized into clear modules (agent behavior, configuration, model building, and distributed learning), making separation of concerns a prime consideration.
- Configurability: The configuration file (config.py) and command-line flags ensure the design supports several environments and algorithms.
- Scalability: Support for distributed layouts suggests a design that scales computation across multiple cores or GPUs.
- Extensibility: With distinct components for agents, networks, and utilities, it’s easy to extend or swap out parts of the system.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Control Flow Layer
    LP["Entry Point (lp_contrastive.py)"]:::control
    subgraph "Configuration & Builder"
        Config["Configuration (contrastive/config.py)"]:::control
        Builder["Builder (contrastive/builder.py)"]:::control
    end

    %% Core RL Module Layer
    subgraph "Agent Module"
        Agent["Agent (contrastive/agents.py)"]:::core
        Learning["Learning Loop (contrastive/learning.py)"]:::core
        Networks["Networks (contrastive/networks.py)"]:::core
        Utilities["Project Utilities (contrastive/utils.py)"]:::core
    end

    %% Environment and Runtime Layer
    subgraph "Environment Module"
        EnvUtils["Env Utilities (env_utils.py)"]:::infra
        PointEnv["Point Environment (point_env.py)"]:::infra
        Default["Default Runtime (default.py)"]:::infra
        Distributional["Distributional Runtime (distributional.py)"]:::infra
    end

    %% Distributed Training Support
    DistTrain["Distributed Training (contrastive/distributed_layout.py)"]:::infra

    %% External Dependencies
    External["External Dependencies\n(JAX,PyTorch,dm-haiku,optax,Mujoco)"]:::external

    %% Training Video Resources
    subgraph "Training Videos"
        T1["sawyer_bin.mp4"]:::resource
        T2["sawyer_box.mp4"]:::resource
        T3["sawyer_box_hard.mp4"]:::resource
        T4["sawyer_peg.mp4"]:::resource
        T5["training_videos/"]:::resource
    end

    %% Connections
    LP --> Config
    LP --> Builder
    Config --> Agent
    Builder --> Agent
    Agent -->|"sendsAction"| EnvUtils
    EnvUtils -->|"returnsObs"| Agent
    Learning -->|"triggersParallel"| DistTrain
    Builder -->|"configures"| DistTrain
    External -->|"supports"| Networks
    Learning -->|"savesVideos"| T1

    %% Styles
    classDef control fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef core fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef infra fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef external fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef resource fill:#D3D3D3,stroke:#000,stroke-width:2px;

    %% Click Events
    click LP "https://github.com/graliuce/sgcrl/blob/main/lp_contrastive.py"
    click Config "https://github.com/graliuce/sgcrl/blob/main/contrastive/config.py"
    click Builder "https://github.com/graliuce/sgcrl/blob/main/contrastive/builder.py"
    click Agent "https://github.com/graliuce/sgcrl/blob/main/contrastive/agents.py"
    click Learning "https://github.com/graliuce/sgcrl/blob/main/contrastive/learning.py"
    click Networks "https://github.com/graliuce/sgcrl/blob/main/contrastive/networks.py"
    click Utilities "https://github.com/graliuce/sgcrl/blob/main/contrastive/utils.py"
    click EnvUtils "https://github.com/graliuce/sgcrl/blob/main/env_utils.py"
    click PointEnv "https://github.com/graliuce/sgcrl/blob/main/point_env.py"
    click Default "https://github.com/graliuce/sgcrl/blob/main/default.py"
    click Distributional "https://github.com/graliuce/sgcrl/blob/main/distributional.py"
    click DistTrain "https://github.com/graliuce/sgcrl/blob/main/contrastive/distributed_layout.py"
    click T1 "https://github.com/graliuce/sgcrl/blob/main/sawyer_bin.mp4"
    click T2 "https://github.com/graliuce/sgcrl/blob/main/sawyer_box.mp4"
    click T3 "https://github.com/graliuce/sgcrl/blob/main/sawyer_box_hard.mp4"
    click T4 "https://github.com/graliuce/sgcrl/blob/main/sawyer_peg.mp4"
    click T5 "https://github.com/graliuce/sgcrl/tree/main/training_videos/"
```
