# 0russwest0/agent-r1 architecture

> Architecture diagram of the GitHub repository 0russwest0/agent-r1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/0russwest0/agent-r1
- Repository: https://github.com/0russwest0/agent-r1

## Overview

##### Project Identification & Purpose
- Agent-R1 is a reinforcement learning framework that trains LLM-based agents using multiple RL algorithms (e.g., PPO, GRPO, REINFORCE++).
- Its key purpose is to support multi-turn tool calling, multi-tool coordination, process rewards, and even multi-modal inputs (vision + text).
- The README highlights that the framework is built for easy integration and extensibility: developers can add custom tools, reward functions, and environments.

##### File Structure Analysis & Architectural Components
- The core logic resides in the “agent_r1” directory, which is broken into several submodules:
a. llm_agent: Contains core agent functionalities (e.g., generation logic and tensor support via generation.py and tensor_helper.py).
b. src: Houses most of the training and RL logic:
- agent_dp_actor.py, agent_dp_critic.py: Likely handle policy evaluation and actor-critic mechanisms.
- agent_ray_trainer.py: Indicates usage of distributed training (possibly using Ray) for scalability.
- main_agent.py: Likely the launching point for training/inference.
- core_algos.py & fsdp_workers.py: Manage core reinforcement learning algorithms and possibly distributed training workers.
- The “reward_score” folder implements various reward function routines (e.g., qa_em_and_format.py, gsm8k.py) to gauge tool performance.
- Config files (agent_trainer.yaml) which play a role in parameterizing training and environment settings.
c. tool: Contains a base for custom tool creation (tool_base.py, tool_env.py) and pre-built tools (calculator_tool, search_tool, wiki_search_tool) which agents can call.
- The “docs” directory provides architecture background (algorithm, extending docs, quickstart guides) that may include suggestions on how the system’s modules interconnect.
- “examples” and “scripts” directories are useful for demonstrating use cases like data preprocessing (hotpotqa, gsm8k) and serving search functionalities (wiki search server).

a. Main Components to Represent:
- LLM Agent Layer (agent_r1/llm_agent): Handles natural language generation and core LLM functionalities.
- Reinforcement Learning Core (agent_r1/src): Comprising training algorithms, actor-critic modules, distributed training (Ray) via agent_ray_trainer, and configuration (YAML-based).
- Reward Evaluation Module (agent_r1/src/reward_score): Provides reward scoring functions to evaluate tool calls.
- Tool Integration Layer (agent_r1/tool): Handles tool definition, environment integration, and the actual tool implementations (calculator, search, wiki search).
- Data & Preprocessing Components (examples/data_preprocess): Prepares data for training and testing.
- Supporting Infrastructure:
- External Scripts: Under “scripts” for processes like hotpotqa search and wiki search API, indicating interaction with external data sources.
- Submodule “verl”: Provides additional extensibility and possibly interfacing with external RL model extensions.
- Documentation & Config (docs, agent_trainer.yaml) that guide system configuration.

b. Interactions & Data Flow:
- The main training loop from “main_agent.py” orchestrates interactions between the LLM agent and the RL core.
- Reward modules evaluate each action/tool call and feed outcome rewards into the training algorithm.
- The tool layer is used by the agent dynamically as part of its multi-turn interactions.
- Distributed execution is managed by components like agent_ray_trainer, calling lower-level functions (fsdp_workers, core_algos).
- External APIs (wiki search, calculator) accessible via scripts support real-time queries.

c. Architectural Patterns & Design Principles:
- Modular Design: Clearly separated folders indicate logical modules (LLM management, RL training, tool integrations).
- Extensibility & Configuration: Custom tools and rewards can be easily plugged into the framework using base classes and configuration files.
- Distributed / Scalable Training: Use of Ray (agent_ray_trainer.py) and FSDP (fsdp_workers.py) suggests a design mindful of scalability.
- Emphasis on multi-turn interactions: The design supports sequential tool calls as part of the learning process.

## Diagram (Mermaid)

```mermaid
graph TD
    %% LLM Agent Module
    subgraph "LLM Agent Module"
        LLM1["LLM: Generation"]:::llmAgent
        LLM2["LLM: Tensor Helper"]:::llmAgent
    end

    %% Reinforcement Learning Core
    subgraph "Reinforcement Learning Core"
        RL1["RL: Actor (DP)"]:::rlCore
        RL2["RL: Critic (DP)"]:::rlCore
        RL3["RL: Ray Trainer"]:::rlCore
        RL4["RL: Main Agent"]:::rlCore
        RL5["RL: Algorithms"]:::rlCore
        RL6["RL: FSDP Workers"]:::rlCore
        CFG["Config: Agent Trainer"]:::configModule
    end

    %% Reward Scoring Module
    subgraph "Reward Scoring Module"
        RW1["Reward: GSM8K"]:::reward
        RW2["Reward: QA_EM&Format"]:::reward
    end

    %% Tool Integration Layer
    subgraph "Tool Integration Layer"
        T1["Tool: Base"]:::tool
        T2["Tool: Env"]:::tool
        T3["Tool: Calculator"]:::tool
        T4["Tool: Search"]:::tool
        T5["Tool: Wiki Search"]:::tool
    end

    %% Data Preprocessing Components
    subgraph "Data Preprocessing"
        DT1["Data: 2WikiMultiHopQA"]:::data
        DT2["Data: GSM8K"]:::data
        DT3["Data: HotpotQA"]:::data
        DT4["Data: Musique"]:::data
    end

    %% External Scripts & Supporting Infrastructure
    subgraph "External & Supporting"
        EX1["Ext Script: HotpotQA Search"]:::external
        EX2["Ext Script: Wiki Search Server"]:::external
        EX3["Ext Module: Verl"]:::external
    end

    %% Documentation
    subgraph "Documentation"
        DOC["Docs"]:::docs
    end

    %% Data Flow and Interactions

    %% Data Preprocessing feeds into the RL Training Core
    DT1 --> RL4
    DT2 --> RL4
    DT3 --> RL4
    DT4 --> RL4

    %% Main RL Agent interacts with LLM Agent Module
    RL4 --> LLM1
    RL4 --> LLM2

    %% RL Core and Reward Scoring feedback loop
    RL4 -->|"reward_feedback"| RW1
    RL4 -->|"reward_feedback"| RW2
    RW1 -->|"reward_signal"| RL4
    RW2 -->|"reward_signal"| RL4

    %% RL Core calls Tool Integration Layer
    RL4 --> T1
    RL4 --> T3
    RL4 --> T4
    RL4 --> T5

    %% Distributed Training support within RL Core
    RL3 --> RL4
    RL5 --> RL4
    RL6 --> RL4
    CFG --> RL4

    %% External Scripts and Supporting Infrastructure feed into RL Core
    EX1 --> RL4
    EX2 --> RL4
    EX3 --> RL4

    %% Documentation supports configuration
    DOC --> CFG

    %% Styles
    classDef llmAgent fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef rlCore fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef configModule fill:#F4A460,stroke:#000,stroke-width:2px;
    classDef reward fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef tool fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef data fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef external fill:#AFEEEE,stroke:#000,stroke-width:2px;
    classDef docs fill:#DDA0DD,stroke:#000,stroke-width:2px;

    %% Click Events
    click LLM1 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/llm_agent/generation.py"
    click LLM2 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/llm_agent/tensor_helper.py"
    click RL1 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/agent_dp_actor.py"
    click RL2 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/agent_dp_critic.py"
    click RL3 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/agent_ray_trainer.py"
    click RL4 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/main_agent.py"
    click RL5 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/core_algos.py"
    click RL6 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/fsdp_workers.py"
    click CFG "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/config/agent_trainer.yaml"
    click RW1 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/reward_score/gsm8k.py"
    click RW2 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/src/reward_score/qa_em_and_format.py"
    click T1 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/tool/tool_base.py"
    click T2 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/tool/tool_env.py"
    click T3 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/tool/tools/calculator_tool.py"
    click T4 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/tool/tools/search_tool.py"
    click T5 "https://github.com/0russwest0/agent-r1/blob/main/agent_r1/tool/tools/wiki_search_tool.py"
    click DT1 "https://github.com/0russwest0/agent-r1/blob/main/examples/data_preprocess/2wikimultihopqa.py"
    click DT2 "https://github.com/0russwest0/agent-r1/blob/main/examples/data_preprocess/gsm8k.py"
    click DT3 "https://github.com/0russwest0/agent-r1/blob/main/examples/data_preprocess/hotpotqa.py"
    click DT4 "https://github.com/0russwest0/agent-r1/blob/main/examples/data_preprocess/musique.py"
    click EX1 "https://github.com/0russwest0/agent-r1/tree/main/scripts/hotpotqa_search"
    click EX2 "https://github.com/0russwest0/agent-r1/tree/main/scripts/wiki_search_server"
    click EX3 "https://github.com/0russwest0/agent-r1/tree/main/Verl"
    click DOC "https://github.com/0russwest0/agent-r1/tree/main/docs"
```
