How petergriffinjin/search-r1 works
Main Components
• Data Processing & Preparation:
– Scripts under “scripts/data_process/” (e.g., nq_search.py, nq_rag.py) process raw datasets (like NQ) and corpora.
– The “example” directory provides sample input files such as corpus.jsonl.
• Retrieval/Search Engine Component:
– Located within “search_r1/search”, this component contains scripts for building indexes (build_index.sh, index_builder.py) as well as the retrieval server (retrieval_server.py, retrieval_request.py).
– The “retrieval_launch.sh” script is used to start a local search engine server.
• LLM Agent and Inference:
– The “search_r1/llm_agent” subdirectory holds code (e.g., generation.py, tensor_helper.py) that likely encapsulates the logic allowing the LLM to interact with the search engine, generating queries and processing responses.
– The top-level “infer.py” facilitates model inference and provides a user interface to test or demonstrate the trained model.
• RL Training Pipeline:
– Training scripts (train_ppo.sh, train_grpo.sh) and training orchestration under “verl/trainer/” implement reinforcement learning (RL) for training the LLM to call the search engine.
– These scripts invoke various training configurations (PPO, SFT, etc.).
• Fundamental Model & Utility Libraries:
– The “verl” directory is extensive and contains:
o Model definitions and implementations (e.g., within “verl/models/llama”, “verl/models/transformers”) that adjust standard LLM architectures.
o Controllers (found in “verl/single_controller”) and worker implementations for distributed or parallel processing.
o Utility modules (within “verl/utils”) covering logging, configuration, performance, and distributed computing.
– Third-party integrations such as vllm (inside “verl/third_party/vllm”) are integrated to optimize model loading and inference.
Relationships and Data Flow
• Train Time:
– Data flows from raw input (e.g., datasets and corpora) processed via the data processing scripts.
– The training pipeline leverages processes orchestrated by RL training scripts and the trainer modules in “verl/trainer”.
– During RL training, the LLM (as defined in “verl/models”) interacts with the search engine component (which can be accessed locally via “search_r1/search/retrieval_server.py”) to simulate and optimize tool-calling.
• Inference / Deployment:
– User requests (questions) are input via the “infer.py” script.
– The LLM uses the llm_agent to generate queries and calls the local search service.
– Results flow back to the client, showcased via the inference outputs.
Architectural Patterns
• Separation of Concerns:
– Notice the clear separation between data processing, model training (RL), inference, and retrieval system function. Each major functionality is encapsulated in its own directory or module.
• Modular Design:
– Using separate submodules (e.g., “verl/models”, “verl/trainer”, “search_r1”) supports modular and scalable design, making it easier to replace components (such as the retriever) or extend functionality.
• Distributed & Parallel Processing:
– Several modules under “verl/single_controller” and “verl/workers” indicate that the project leverages distributed frameworks (sometimes via Ray) for scalable training and inference.
• Integration of External Services:
– The design explicitly supports launching a standalone search engine (retrieval_server.py) that the LLM interacts with through REST API calls.