# heisnotanimposter/openmanus-rl architecture

> Architecture diagram of the GitHub repository heisnotanimposter/openmanus-rl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/heisnotanimposter/openmanus-rl
- Repository: https://github.com/heisnotanimposter/openmanus-rl

## Overview

##### Project’s Main Components
- SFT Module: This component encapsulates the supervised fine-tuning routines (found in openmanus_rl/sft.py) and provides an interface for tuning a model on curated datasets.
- GRPO Module: Represented by openmanus_rl/grpo.py, this part handles reinforcement learning training by optimizing rewards based on model performance. It highlights the reward functions (accuracy, format, tag_count, etc.) and the integration of RL strategies.
- Data Pipeline & Dataset: Based on the README and provided dataset descriptions, show the data collection from sources like AgentInstruct, Agent-FLAN, and AgentTraj-L. This includes data preprocessing, uniform formatting, and the creation of training and evaluation datasets.
- Reward Modeling: Outline the role of reward functions and reward model training (openmanus_rl/rewards.py) to inform the GRPO module.
- Environment and Benchmarks: Although the main library is in openmanus_rl, the extended file tree (especially the “webshop_tot” and related directories) suggests simulation environments and testing/journey flows (e.g., transfer, search_engine, tests, web_agent_site) that simulate real-world interactions.
- Configuration & Deployment: Include configuration files (pyproject.toml, requirements.txt, pre-commit and GitHub workflow files) that manage testing, continuous integration, and deployment aspects.
- External Integrations: Represent connections with external agents and benchmark environments (e.g., reasoning models like GPT-O1, Deepseek-R1, QwQ-32B) and integration with RL tuning frameworks (e.g., Verl, TinyZero, OpenR1, Trlx).

##### Architectural Patterns and Design Principles
- Layered Architecture: Show clear separations between data ingestion, core model training (both SFT and RL-based GRPO), and evaluation/simulation in agent environments.
- Modularity: Emphasize that components like the RL modules, dataset routines, and reward modeling are decoupled and can be updated independently.
- Extension Points: Indicate areas where additional reasoning models (e.g., ReAct, Tree-of-Thoughts, Monte Carlo Tree Search) may be incorporated, and where the project can integrate external RL frameworks.

##### Relevant Technologies and Frameworks
- Python packages (as indicated in requirements.txt), along with Hugging Face libraries (as suggested by model paths in the training commands).
- Accelerate: A separate module to handle distributed training (via accelerate launch commands).
- GitHub CI/CD: Reference workflow configurations (in .github/workflows) that are part of the automation and quality checks.
- External assets such as images and data stored in the assets and data directories.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing Layer
    subgraph "Data Ingestion & Preprocessing"
        DP1["Data (Dataset Directory)"]:::database
        DP2["Examples (Dataset Directory)"]:::database
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        SFT["Supervised Fine-Tuning Module"]:::module
        GRPO["Reinforcement Learning (GRPO) Module"]:::module
        RM["Reward Modeling Component"]:::module
    end

    %% Agent Environment & Benchmarking
    subgraph "Agent Environment & Benchmarking"
        ENV1["WebShop Simulation Environment"]:::environment
        ENV2["Web Agent Site"]:::environment
    end

    %% Configuration & Deployment
    subgraph "Configuration & Deployment"
        CFG1["pyproject.toml"]:::config
        CFG2["requirements.txt"]:::config
        CFG3[".github"]:::config
        CFG4[".pre-commit-config.yaml"]:::config
    end

    %% Data Pipeline feeds into Training Pipeline
    DP1 -->|"provides_data"| SFT
    DP1 -->|"provides_data"| GRPO
    DP2 -->|"provides_data"| SFT
    DP2 -->|"provides_data"| GRPO

    %% Training Modules link to Reward Modeling
    SFT -->|"may_use_reward"| RM
    GRPO -->|"uses_reward"| RM

    %% Environment Feedback to Training Pipeline
    ENV1 -->|"feedback"| SFT
    ENV1 -->|"feedback"| GRPO
    ENV2 -->|"feedback"| SFT
    ENV2 -->|"feedback"| GRPO

    %% Configuration & Deployment orchestrates modules
    CFG1 -->|"orchestrates"| SFT
    CFG1 -->|"orchestrates"| GRPO
    CFG1 -->|"orchestrates"| ENV1
    CFG1 -->|"orchestrates"| ENV2

    CFG2 -->|"orchestrates"| SFT
    CFG2 -->|"orchestrates"| GRPO
    CFG2 -->|"orchestrates"| ENV1
    CFG2 -->|"orchestrates"| ENV2

    CFG3 -->|"orchestrates"| SFT
    CFG3 -->|"orchestrates"| GRPO
    CFG3 -->|"orchestrates"| ENV1
    CFG3 -->|"orchestrates"| ENV2

    CFG4 -->|"orchestrates"| SFT
    CFG4 -->|"orchestrates"| GRPO
    CFG4 -->|"orchestrates"| ENV1
    CFG4 -->|"orchestrates"| ENV2

    %% Click Events
    click SFT "https://github.com/heisnotanimposter/openmanus-rl/blob/main/openmanus_rl/sft.py"
    click GRPO "https://github.com/heisnotanimposter/openmanus-rl/blob/main/openmanus_rl/grpo.py"
    click RM "https://github.com/heisnotanimposter/openmanus-rl/blob/main/openmanus_rl/rewards.py"
    click DP1 "https://github.com/heisnotanimposter/openmanus-rl/tree/main/data"
    click DP2 "https://github.com/heisnotanimposter/openmanus-rl/tree/main/examples"
    click ENV1 "https://github.com/heisnotanimposter/openmanus-rl/tree/main/webshop_tot"
    click ENV2 "https://github.com/heisnotanimposter/openmanus-rl/tree/main/__MACOSX/webshop_tot/web_agent_site"
    click CFG1 "https://github.com/heisnotanimposter/openmanus-rl/blob/main/pyproject.toml"
    click CFG2 "https://github.com/heisnotanimposter/openmanus-rl/blob/main/requirements.txt"
    click CFG3 "https://github.com/heisnotanimposter/openmanus-rl/blob/main/.github"
    click CFG4 "https://github.com/heisnotanimposter/openmanus-rl/blob/main/.pre-commit-config.yaml"

    %% Styles
    classDef module fill:#f9c,stroke:#333,stroke-width:2px;
    classDef database fill:#cfc,stroke:#333,stroke-width:2px;
    classDef environment fill:#ffc,stroke:#333,stroke-width:2px;
    classDef config fill:#ccf,stroke:#333,stroke-width:2px;
```
