# alibaba-nlp/webagent architecture

> Architecture diagram of the GitHub repository alibaba-nlp/webagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/alibaba-nlp/webagent
- Repository: https://github.com/alibaba-nlp/webagent

## Overview

##### Project Type & Purpose
- This is a mixed ML/engineering repository combining:
- WebDancer: an agentic search-reasoning model (training pipeline + demo UI).
- WebWalker: a benchmark/multi-agent framework for web traversal (API + evaluation).
- It provides data (datasets/ assets), model deployment scripts, demo interfaces, and evaluation tooling.

##### File Structure
Top-level folders:
- WebDancer/
- assets, datasets (sample_qa.jsonl, sample_traj.jsonl)
- demos/ (agents, GUI, LLM adapters, tools, utils)
- scripts/ (deploy_model.sh, run_demo.sh)
- requirements.txt, readme.md
- WebWalker/
- assets/
- src/ (agent.py, app.py, evaluate.py, prompts.py, rag_system.py, utils.py)
- requirements.txt, README.md
- Shared: LICENSE, README.md, .gitignore, assets/

##### Extract Key Architectural Insights from the README
- Core pipelines: data construction → trajectory sampling → supervised fine-tuning → RL.
- Demo uses Gradio for interactive UI.
- Model served via sglang after downloading from Hugging Face.
- WebWalker runs a retrieval-augmented generation (RAG) system plus evaluation scripts.

##### Main Components & Layers
A. User Interfaces
1. CLI scripts (bash run_demo.sh, deploy_model.sh)
2. Gradio Web UI (WebDancer/demos/gui/web_ui.py + HTML decorators)
3. WebWalker API/App (WebWalker/src/app.py)
B. Orchestration & Business Logic
1. Agent modules (WebDancer/demos/agents, WebWalker/src/agent.py)
2. LLM adapters (WebDancer/demos/llm/oai.py, qwen_dashscope.py)
3. Tool wrappers (search, visit, cache in demos/tools)
4. RAG system (WebWalker/src/rag_system.py)
5. Prompt templates (prompts.py)
C. Model & Compute
1. WebDancer model (downloaded from Hugging Face, loaded via sglang)
2. Training pipelines (not in code but implied by README)
3. LLM inference (OpenAI/Qwen, via API keys)
D. Data Storage
1. Datasets folder (sample QA, trajectories)
2. Assets (images, logos, roadmap)
E. External Services
1. Google Search API
2. Jina API
3. Dashscope API
4. Hugging Face model hub

##### Relationships & Data-flow
- Users launch `run_demo.sh` → starts Gradio server → user inputs query.
- Web UI calls Agent orchestrator → Agent loads model via sglang → Agent issues search via Google/Jina/Dashscope tools → fetches web pages → LLM processes steps (ReAct) → returns answer.
- WebWalker’s `app.py` exposes endpoints for benchmarking LLM agents → orchestrates RAG retrieval and agent execution → evaluation script (`evaluate.py`) computes metrics on WebWalkerQA.
- Underneath, prompt templates guide both pipelines; cache_utils speeds up repeated calls.

##### Architectural Patterns & Principles
- Layered architecture: UI → orchestration/agents → model/inference → external tools.
- Modular separation: WebDancer vs WebWalker, and within each, split agents, tools, adapters.
- Plugin-style LLM adapters (easy to swap OpenAI/Qwen).
- Data-centric training pipeline (four-stage: data → SFT → RL).

##### Final Checklist
[ ] Clear labels on all components
[ ] External services distinctly grouped
[ ] Note API keys & configuration files (requirements.txt, scripts)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% WebDancer Subproject
    subgraph "WebDancer"
        direction TB
        %% UI Layer
        CLI["CLI Scripts"]:::ui
        Gradio["Gradio Web UI"]:::ui

        %% Orchestration Layer
        subgraph "Orchestration & Agents"
            direction TB
            AgentWD["Search Agent"]:::orches
            Assistant["Assistant Chat"]:::orches
            subgraph "LLM Adapters"
                direction TB
                OAI["OpenAI Adapter"]:::orches
                QWEN["Qwen Dashscope Adapter"]:::orches
            end
            subgraph "Tool Wrappers"
                direction TB
                SearchTool["Search Tool"]:::orches
                VisitTool["Visit Tool"]:::orches
                CacheUtil["Cache Utils"]:::orches
            end
            subgraph "Utilities"
                direction TB
                Logs["Logging Utils"]:::orches
                DateUtil["Date Utils"]:::orches
            end
        end

        %% Model & Compute
        subgraph "Model & Training"
            direction TB
            HFDownload["Hugging Face Model Hub"]:::external
            SGLang["sglang Inference"]:::model
            Training["4-Stage Training Pipeline"]:::model
        end

        %% Data Storage
        subgraph "Data Storage"
            direction TB
            Datasets["Datasets"]:::data
            AssetsWD["WebDancer Assets"]:::data
        end
    end

    %% WebWalker Subproject
    subgraph "WebWalker"
        direction TB
        %% UI Layer
        API["REST API/App"]:::ui

        %% Orchestration & Evaluation
        subgraph "Orchestration & Evaluation"
            direction TB
            AgentWW["WebWalker Agent"]:::orches
            RAG["RAG System"]:::orches
            Prompts["Prompt Templates"]:::orches
            Evaluate["Evaluation Scripts"]:::orches
        end

        %% Data Storage
        AssetsWW["WebWalker Assets"]:::data
    end

    %% Shared Base
    subgraph "Shared & Config"
        direction TB
        SharedAssets["Shared Assets"]:::data
        ReqWD["WebDancer requirements.txt"]:::orches
        ReqWW["WebWalker requirements.txt"]:::orches
        Docs["Docs & Config (README, LICENSE, .gitignore)"]:::orches
    end

    %% External Services
    subgraph "External Services"
        direction TB
        Google["Google Search API"]:::external
        Jina["Jina API"]:::external
        Dashscope["Dashscope API"]:::external
        HFHub["Hugging Face Hub"]:::external
    end

    %% User
    User["User"]:::external

    %% Connections - WebDancer
    User -->|"runs run_demo.sh"| CLI
    User -->|"opens UI"| Gradio
    CLI -->|"starts demo"| Gradio
    Gradio -->|"HTTP request"| AgentWD
    AgentWD -->|"loads model"| SGLang
    AgentWD -->|"calls OAI/QWEN"| OAI
    AgentWD -->|"calls OAI/QWEN"| QWEN
    AgentWD -->|"uses tools"| SearchTool
    AgentWD -->|"uses tools"| VisitTool
    AgentWD -->|"uses cache"| CacheUtil
    OAI -->|"LLM API call"| HFDownload
    QWEN -->|"LLM API call"| Dashscope
    SearchTool -->|"Search query"| Google
    SearchTool -->|"Search query"| Jina
    VisitTool -->|"fetch pages"| external
    SGLang -->|"model weights"| HFDownload
    Training -->|"uses data"| Datasets
    Gradio -->|"serves response"| User

    %% Connections - WebWalker
    User -->|"sends request"| API
    API -->|"orchestrates"| AgentWW
    AgentWW -->|"retrieval"| RAG
    RAG -->|"uses prompts"| Prompts
    AgentWW -->|"evaluation call"| Evaluate
    Evaluate -->|"metrics"| User
    API -->|"serves responses"| User

    %% Shared connections
    ReqWD --- WebDancer
    ReqWW --- WebWalker
    SharedAssets --- Datasets
    SharedAssets --- AssetsWD
    SharedAssets --- AssetsWW

    %% Click Events
    click CLI "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/scripts/run_demo.sh"
    click Gradio "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/gui/web_ui.py"
    click AgentWD "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/agents/search_agent.py"
    click Assistant "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/assistant_qwq_chat.py"
    click OAI "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/llm/oai.py"
    click QWEN "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/llm/qwen_dashscope.py"
    click SearchTool "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/tools/private/search.py"
    click VisitTool "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/tools/private/visit.py"
    click CacheUtil "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/tools/private/cache_utils.py"
    click Logs "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/utils/logs.py"
    click DateUtil "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/demos/utils/date.py"
    click API "https://github.com/alibaba-nlp/webagent/blob/main/WebWalker/src/app.py"
    click AgentWW "https://github.com/alibaba-nlp/webagent/blob/main/WebWalker/src/agent.py"
    click RAG "https://github.com/alibaba-nlp/webagent/blob/main/WebWalker/src/rag_system.py"
    click Prompts "https://github.com/alibaba-nlp/webagent/blob/main/WebWalker/src/prompts.py"
    click Evaluate "https://github.com/alibaba-nlp/webagent/blob/main/WebWalker/src/evaluate.py"
    click Datasets "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/datasets/sample_qa.jsonl"
    click Datasets "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/datasets/sample_traj.jsonl"
    click AssetsWD "https://github.com/alibaba-nlp/webagent/tree/main/WebDancer/assets/"
    click AssetsWW "https://github.com/alibaba-nlp/webagent/tree/main/WebWalker/assets/"
    click SharedAssets "https://github.com/alibaba-nlp/webagent/tree/main/assets/"
    click ReqWD "https://github.com/alibaba-nlp/webagent/blob/main/WebDancer/requirements.txt"
    click ReqWW "https://github.com/alibaba-nlp/webagent/blob/main/WebWalker/requirements.txt"
    click Docs "https://github.com/alibaba-nlp/webagent/blob/main/README.md"

    %% Styles
    classDef ui fill:#ADD8E6,stroke:#000
    classDef orches fill:#90EE90,stroke:#000
    classDef model fill:#FFA500,stroke:#000
    classDef data fill:#FFFF99,stroke:#000
    classDef external fill:#E0E0E0,stroke:#000
```
