How gair-nlp/deepresearcher works
Project type and purpose
• DeepResearcher is a distributed reinforcement-learning framework for training LLM-based research agents in real-world web environments.
• It orchestrates end-to-end RL training (SFT, PPO/GRPO, RLHF), rollout generation, evaluation, and external web search interactions via a backend handler.
File structure
• Top-level “train_grpo.sh”, “evaluate.sh”, “docker/” and “pyproject.toml” imply CLI entrypoints and containerized deployments.
• “verl/” houses the core library: model definitions, distributed controllers (Ray, Megatron, FSDP, vLLM), trainers, dataset utilities, checkpointing, and worker implementations.
• “scrl/handler/” implements the backend search‐environment server (web search agents, reading agents, server_handler.py).
• “verl/trainer/” implements the RL algorithms (ppo, grpo, sft) entrypoints and YAML configuration.
• “verl/workers/” defines actor, critic, rollout, reward model, sharding manager, etc.
• “examples/” and “tests/” show usage patterns and end-to-end pipelines.
README for additional insights
• Uses Ray for distributed orchestration; Dockerfiles provided for specialized environments (Megatron, vLLM).
• Backend handler exposes LLM + web search operations to the training loop.
• Pipelines:
a. Launch Ray cluster →
b. Launch one or more server_handler instances (web search API + LLM client) →
c. Run the RL trainer (train_grpo.sh) →
d. Generate rollouts (evaluate.sh) →
e. Compute metrics.
a. Main components:
• CLI / Orchestrator (train_grpo.sh, evaluate.sh)
• Ray Cluster (Head + Worker nodes)
• Trainer Module (verl/trainer: SFT, PPO, GRPO engines)
• Controller Subsystem (verl/single_controller: Ray, Megatron, FSDP, vLLM)
• Worker Pools:
– Actor Workers (call policy to generate actions)
– Critic Workers (value estimation)
– Rollout Workers (trajectories via vLLM / HF)
– Reward-Model Workers (compute RL rewards)
– Sharding Managers (data/model parallelism)
• Model Registry & Checkpoint Manager (verl/utils/checkpoint)
• Data Storage (Parquet datasets under data/, HDFS via utils/hdfs_io.py)
• Backend Handler Service (scrl/handler):
– Web Search Agent (serper, Azure Bing)
– Reading Agent (cross-validation, self-reflection)
• External Services:
– OpenAI / Qwen API
– Serper / Azure Bing
• Evaluation Module (evaluate/cacluate_metrics.py)
b. Component Interactions & Data Flow:
1. CLI invokes train_grpo.sh → Ray Head startup → spawn Worker nodes
2. Trainer Module requests rollout from Worker Pools via Controller Subsystem
3. Rollout Workers call Backend Handler Service over HTTP/gRPC to perform web searches and get LLM responses
4. Agents generate actions; environment state updates; Reward-Model Workers compute RL rewards
5. Critic Workers evaluate value functions; gradients aggregated by Controller
6. Checkpoint Manager persists model shards; metrics logged
7. After training, CLI invokes evaluate.sh → rollout JSONs generated → metrics computed
c. Architectural Patterns & Design Principles:
• Microservices for backend handler (decouples web search environment from training loop)
• Distributed data‐ and model‐parallel training (Megatron, FSDP, vLLM)
• Plugin architecture for model definitions (verl/models/registry.py) and trainer algorithms (verl/trainer/config/*.yaml)
• Ray‐based actor model for scalable RL
d. Key Technologies & Frameworks:
• Python 3.10, PyTorch 2.4.0, FlashAttention
• Ray (distributed orchestration)
• Megatron-LM, vLLM (model parallelism / inference acceleration)
• Docker (container images for specialized runtimes)
• Parquet (dataset storage), HDFS support
• OpenAI / Azure / Serper APIs for web search
Detail
• Worker type (actor, critic, reward_model, rollout, sharding) as separate nodes.
• Plugin registry for models and trainer algorithms.
• Kubernetes/Docker orchestration if used (docker/Dockerfile.megatron, etc.).