# alibaba-nlp/deepresearch architecture

> Architecture diagram of the GitHub repository alibaba-nlp/deepresearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alibaba-nlp/deepresearch
- Repository: https://github.com/alibaba-nlp/deepresearch

## Overview

##### Project Type and Purpose
- This is a research-framework project (“Tongyi DeepResearch”) for building, scaling, and benchmarking **agentic LLMs** specialized in long‐horizon, deep information‐seeking tasks.
- Key modules:
- Agent development & scaling (AgentFounder, AgentScaler)
- Multiple WebAgent prototypes (WebDancer, WebSailor, Webwalker, etc.)
- Inference & evaluation pipelines
- Synthetic data generation & on‐policy RL training

##### File‐Structure Analysis
- Top‐level directories:
- Agent/AgentFounder – bootstrap & design of new agents (data‐gen, initial RL setups)
- Agent/AgentScaler – scalable data synthesis and continued pre-training pipeline
- WebAgent/… – concrete agent implementations (WebDancer, WebSailor, etc.) each with assets, datasets, demos, scripts
- inference/ – scripts and tools for running trained agents at inference time (ReAct, multi-step planning, file/video tools)
- evaluation/ – benchmark scripts for measuring agent performance on standard datasets
- assets/, requirements.txt – shared resources and dependencies
- Patterns:
- Pipeline stages (founding → scaling → inference → evaluation)
- Tool‐oriented ReAct paradigm: agent core + external tool interfaces
- Modular agent prototypes under WebAgent/ for extensibility

##### README Insights
- Describes three major phases:
a) Synthetic data generation → supervised & RL fine-tuning (AgentFounder + AgentScaler)
b) Two inference paradigms (ReAct & Heavy mode) implemented under inference/
c) Benchmark evaluation under evaluation/
- Mentions external tools (search, python exec, file parser, video analysis)

a. Main Components
1) Synthetic Data Generation (AgentFounder)
2) Continual Pre-training & Scaling (AgentScaler)
3) Trained Model Artifact (Tongyi DeepResearch LLM)
4) Inference Engine (inference/react_agent.py + tool modules)
5) Evaluation Suite (evaluation/*.py)
6) WebAgent Prototype Suite (WebAgent/WebDancer, WebSailor, etc.)
b. Relationships & Interactions
- AgentFounder → produces initial agentic training data → feeds into AgentScaler
- AgentScaler → continual pre-training → outputs model checkpoints
- Inference Engine → loads model checkpoints, orchestrates ReAct loops, calls tool modules (search, file, python, video)
- WebAgent prototypes → use the same inference engine + custom prompts/tools for specialized experiments
- Evaluation Suite → consumes model outputs via inference scripts → computes metrics
c. Architectural Patterns & Principles
- Pipeline decomposition: clearly separated stages (data, pre-train, infer, eval)
- Modular “tool” interfaces for ReAct: each external capability wrapped in a Python module
- Extensibility: new agents drop in under WebAgent/, inheriting common inference and evaluation layers
d. Relevant Technologies
- Python 3.10, PyTorch (implied), HuggingFace transformers
- Shell scripts (run_react_infer.sh, deploy_model.sh)
- JSONL data format for datasets and benchmarks
- Custom RL framework (Group RPO)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Pipeline
    subgraph "Data Generation" 
        direction TB
        AF["AgentFounder (Data-Gen)"]:::pipeline
    end

    %% Scaling Pipeline
    subgraph "Continual Pre-training" 
        direction TB
        AS["AgentScaler (Continual PT)"]:::train
    end

    %% Model Artifact
    MC["Model Checkpoints"]:::data

    %% Core Engine
    subgraph "Core Inference Engine" 
        direction TB
        ENG["LLM + ReAct Controller"]:::inference
    end

    %% Tool Adapters
    subgraph "Tool Adapters" 
        direction TB
        TS["Web Search Adapter"]:::inference
        TV["Web Visit Adapter"]:::inference
        TF["File Parser Adapter"]:::inference
        TPY["Python Exec Adapter"]:::inference
        TSCH["Scholar API Adapter"]:::inference
    end

    %% Inference Utilities
    subgraph "Inference Utilities" 
        direction TB
        MR["Multi-Step Runner"]:::utility
        SH["Shell Launcher"]:::utility
        PINF["Prompt Definitions"]:::utility
    end

    %% WebAgent Prototypes
    subgraph "WebAgent Prototypes" 
        direction TB
        WD["WebDancer"]:::extension
        WS["WebSailor"]:::extension
        WWL["WebWalker"]:::extension
        WWT["WebWatcher"]:::extension
        WSP["WebShaper"]:::extension
        WR["WebResearcher"]:::extension
        WRS["WebResummer"]:::extension
        WWV["WebWeaver"]:::extension
    end

    %% Evaluation Suite
    subgraph "Evaluation Suite" 
        direction TB
        EDS["DeepSearch Benchmark"]:::evaluation
        EHL["HLE Benchmark"]:::evaluation
        EP["Eval Prompt Definitions"]:::evaluation
    end

    %% Pipeline Flow
    AF -->|"generates data"| AS
    AS -->|"produces checkpoints"| MC
    MC -->|"loads model"| ENG
    ENG -->|"calls adapter"| TS
    ENG -->|"calls adapter"| TV
    ENG -->|"calls adapter"| TF
    ENG -->|"calls adapter"| TPY
    ENG -->|"calls adapter"| TSCH
    ENG -->|"uses runner"| MR
    ENG -->|"uses shell"| SH
    ENG -->|"uses prompts"| PINF
    ENG -.->|extends| WD
    ENG -.->|extends| WS
    ENG -.->|extends| WWL
    ENG -.->|extends| WWT
    ENG -.->|extends| WSP
    ENG -.->|extends| WR
    ENG -.->|extends| WRS
    ENG -.->|extends| WWV
    MR -->|"evaluates model"| EDS
    MR -->|"evaluates model"| EHL
    EDS -->|"uses prompts"| EP
    EHL -->|"uses prompts"| EP

    %% Click Events
    click AF "https://github.com/alibaba-nlp/deepresearch/tree/main/Agent/AgentFounder"
    click AS "https://github.com/alibaba-nlp/deepresearch/tree/main/Agent/AgentScaler"
    click ENG "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/react_agent.py"
    click TS "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/tool_search.py"
    click TV "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/tool_visit.py"
    click TF "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/tool_file.py"
    click TPY "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/tool_python.py"
    click TSCH "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/tool_scholar.py"
    click MR "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/run_multi_react.py"
    click SH "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/run_react_infer.sh"
    click PINF "https://github.com/alibaba-nlp/deepresearch/blob/main/inference/prompt.py"
    click EDS "https://github.com/alibaba-nlp/deepresearch/blob/main/evaluation/evaluate_deepsearch_official.py"
    click EHL "https://github.com/alibaba-nlp/deepresearch/blob/main/evaluation/evaluate_hle_official.py"
    click EP "https://github.com/alibaba-nlp/deepresearch/blob/main/evaluation/prompt.py"
    click WD "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebDancer"
    click WS "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebSailor"
    click WWL "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebWalker"
    click WWT "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebWatcher"
    click WSP "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebShaper"
    click WR "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebResearcher"
    click WRS "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebResummer"
    click WWV "https://github.com/alibaba-nlp/deepresearch/tree/main/WebAgent/WebWeaver"

    %% Styles
    classDef pipeline fill:#DEEBFF,stroke:#0366D6,stroke-width:2px
    classDef train fill:#E6FFED,stroke:#28A745,stroke-width:2px
    classDef data fill:#FFF5B1,stroke:#D69D00,stroke-width:2px
    classDef inference fill:#FFEFE6,stroke:#D9534F,stroke-width:2px
    classDef utility fill:#E8F0FE,stroke:#1A73E8,stroke-width:2px
    classDef extension fill:#F5F5F5,stroke:#6A737D,stroke-width:2px,stroke-dasharray: 5 5
    classDef evaluation fill:#F3E8FF,stroke:#6F42C1,stroke-width:2px
```
