# gair-nlp/deepresearcher architecture

> Architecture diagram of the GitHub repository gair-nlp/deepresearcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gair-nlp/deepresearcher
- Repository: https://github.com/gair-nlp/deepresearcher

## Overview

##### Project type and purpose
- DeepResearcher is a distributed reinforcement-learning framework for training LLM-based research agents in real-world web environments.
- It orchestrates end-to-end RL training (SFT, PPO/GRPO, RLHF), rollout generation, evaluation, and external web search interactions via a backend handler.

##### File structure
- Top-level “train_grpo.sh”, “evaluate.sh”, “docker/” and “pyproject.toml” imply CLI entrypoints and containerized deployments.
- “verl/” houses the core library: model definitions, distributed controllers (Ray, Megatron, FSDP, vLLM), trainers, dataset utilities, checkpointing, and worker implementations.
- “scrl/handler/” implements the backend search‐environment server (web search agents, reading agents, server_handler.py).
- “verl/trainer/” implements the RL algorithms (ppo, grpo, sft) entrypoints and YAML configuration.
- “verl/workers/” defines actor, critic, rollout, reward model, sharding manager, etc.
- “examples/” and “tests/” show usage patterns and end-to-end pipelines.

##### README for additional insights
- Uses Ray for distributed orchestration; Dockerfiles provided for specialized environments (Megatron, vLLM).
- Backend handler exposes LLM + web search operations to the training loop.
- Pipelines:
a. Launch Ray cluster →
b. Launch one or more server_handler instances (web search API + LLM client) →
c. Run the RL trainer (train_grpo.sh) →
d. Generate rollouts (evaluate.sh) →
e. Compute metrics.

a. Main components:
- CLI / Orchestrator (train_grpo.sh, evaluate.sh)
- Ray Cluster (Head + Worker nodes)
- Trainer Module (verl/trainer: SFT, PPO, GRPO engines)
- Controller Subsystem (verl/single_controller: Ray, Megatron, FSDP, vLLM)
- Worker Pools:
– Actor Workers (call policy to generate actions)
– Critic Workers (value estimation)
– Rollout Workers (trajectories via vLLM / HF)
– Reward-Model Workers (compute RL rewards)
– Sharding Managers (data/model parallelism)
- Model Registry & Checkpoint Manager (verl/utils/checkpoint)
- Data Storage (Parquet datasets under data/, HDFS via utils/hdfs_io.py)
- Backend Handler Service (scrl/handler):
– Web Search Agent (serper, Azure Bing)
– Reading Agent (cross-validation, self-reflection)
- External Services:
– OpenAI / Qwen API
– Serper / Azure Bing
- Evaluation Module (evaluate/cacluate_metrics.py)

b. Component Interactions & Data Flow:
1. CLI invokes train_grpo.sh → Ray Head startup → spawn Worker nodes
2. Trainer Module requests rollout from Worker Pools via Controller Subsystem
3. Rollout Workers call Backend Handler Service over HTTP/gRPC to perform web searches and get LLM responses
4. Agents generate actions; environment state updates; Reward-Model Workers compute RL rewards
5. Critic Workers evaluate value functions; gradients aggregated by Controller
6. Checkpoint Manager persists model shards; metrics logged
7. After training, CLI invokes evaluate.sh → rollout JSONs generated → metrics computed

c. Architectural Patterns & Design Principles:
- Microservices for backend handler (decouples web search environment from training loop)
- Distributed data‐ and model‐parallel training (Megatron, FSDP, vLLM)
- Plugin architecture for model definitions (verl/models/registry.py) and trainer algorithms (verl/trainer/config/*.yaml)
- Ray‐based actor model for scalable RL

d. Key Technologies & Frameworks:
- Python 3.10, PyTorch 2.4.0, FlashAttention
- Ray (distributed orchestration)
- Megatron-LM, vLLM (model parallelism / inference acceleration)
- Docker (container images for specialized runtimes)
- Parquet (dataset storage), HDFS support
- OpenAI / Azure / Serper APIs for web search

##### Detail
- Worker type (actor, critic, reward_model, rollout, sharding) as separate nodes.
- Plugin registry for models and trainer algorithms.
- Kubernetes/Docker orchestration if used (docker/Dockerfile.megatron, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Orchestrators
    CLI_TRAIN["CLI: train_grpo.sh"]:::cli
    CLI_EVAL["CLI: evaluate.sh"]:::cli

    %% Training Cluster
    subgraph "Training Cluster"
        direction TB
        RayHead["Ray Head Node"]:::controller
        subgraph "Trainer Module" 
            direction TB
            Trainer["Trainer (SFT, PPO, GRPO)"]:::trainer
        end
        subgraph "Controller Subsystem"
            direction TB
            Controller["Controller (Ray, Megatron, FSDP, vLLM)"]:::controller
        end
        subgraph "Worker Pools"
            direction TB
            Actor["Actor Worker\nverl/workers/actor/"]:::worker
            Critic["Critic Worker\nverl/workers/critic/"]:::worker
            Rollout["Rollout Worker\nverl/workers/rollout/"]:::worker
            RewardModel["Reward-Model Worker\nverl/workers/reward_model/"]:::worker
            Sharding["Sharding Manager\nverl/workers/sharding_manager/"]:::worker
        end
        Checkpoint["Checkpoint Manager\nverl/utils/checkpoint/checkpoint_manager.py"]:::storage
        DataStorage["Parquet & HDFS Storage\nverl/utils/dataset/, verl/utils/hdfs_io.py"]:::storage
    end

    %% Backend Handler Service
    subgraph "Backend Handler Service"
        direction TB
        ServerHandler["Server Handler\nscrl/handler/server_handler.py"]:::service
        WebAgent["Web Search Agent\nscrl/handler/web_search_agent/"]:::service
        ReadAgent["Reading Agent\nscrl/handler/reading_agent/"]:::service
    end

    %% External Services
    subgraph "External Services"
        direction TB
        OpenAI["OpenAI / Qwen API"]:::external
        Serper["Serper Search API"]:::external
        Bing["Azure Bing API"]:::external
    end

    %% Evaluation Module
    Evaluation["Evaluation Module\nevaluate/cacluate_metrics.py"]:::evaluation

    %% Data Flow
    CLI_TRAIN -->|"start cluster"| RayHead
    RayHead --> Trainer
    Trainer --> Controller
    Controller --> Actor
    Controller --> Critic
    Controller --> Rollout
    Controller --> RewardModel
    Controller --> Sharding
    Rollout -->|HTTP/gRPC calls| ServerHandler
    ServerHandler --> WebAgent
    ServerHandler --> ReadAgent
    WebAgent --> OpenAI
    WebAgent --> Serper
    WebAgent --> Bing
    Actor --> Trainer
    Critic --> Trainer
    Rollout --> Trainer
    RewardModel --> Trainer
    Sharding --> Trainer
    Trainer --> Checkpoint
    Trainer --> DataStorage
    CLI_EVAL --> Evaluation
    Evaluation --> DataStorage

    %% Click Events
    click CLI_TRAIN "https://github.com/gair-nlp/deepresearcher/blob/main/train_grpo.sh"
    click CLI_EVAL "https://github.com/gair-nlp/deepresearcher/blob/main/evaluate.sh"
    click Trainer "https://github.com/gair-nlp/deepresearcher/tree/main/verl/trainer/"
    click Controller "https://github.com/gair-nlp/deepresearcher/tree/main/verl/single_controller/"
    click Actor "https://github.com/gair-nlp/deepresearcher/tree/main/verl/workers/actor/"
    click Critic "https://github.com/gair-nlp/deepresearcher/tree/main/verl/workers/critic/"
    click Rollout "https://github.com/gair-nlp/deepresearcher/tree/main/verl/workers/rollout/"
    click RewardModel "https://github.com/gair-nlp/deepresearcher/tree/main/verl/workers/reward_model/"
    click Sharding "https://github.com/gair-nlp/deepresearcher/tree/main/verl/workers/sharding_manager/"
    click Checkpoint "https://github.com/gair-nlp/deepresearcher/blob/main/verl/utils/checkpoint/checkpoint_manager.py"
    click DataStorage "https://github.com/gair-nlp/deepresearcher/tree/main/verl/utils/dataset/"
    click DataStorage "https://github.com/gair-nlp/deepresearcher/blob/main/verl/utils/hdfs_io.py"
    click ServerHandler "https://github.com/gair-nlp/deepresearcher/blob/main/scrl/handler/server_handler.py"
    click WebAgent "https://github.com/gair-nlp/deepresearcher/tree/main/scrl/handler/web_search_agent/"
    click ReadAgent "https://github.com/gair-nlp/deepresearcher/tree/main/scrl/handler/reading_agent/"
    click Evaluation "https://github.com/gair-nlp/deepresearcher/blob/main/evaluate/cacluate_metrics.py"

    %% Styles
    classDef cli fill:#f9f,stroke:#333,stroke-width:1px
    classDef trainer fill:#bbf,stroke:#333,stroke-width:1px
    classDef controller fill:#bfb,stroke:#333,stroke-width:1px
    classDef worker fill:#ffb,stroke:#333,stroke-width:1px
    classDef storage fill:#fbf,stroke:#333,stroke-width:1px
    classDef service fill:#bff,stroke:#333,stroke-width:1px
    classDef external fill:#fdd,stroke:#333,stroke-width:1px
    classDef evaluation fill:#dfd,stroke:#333,stroke-width:1px
```
