How qwen-applications/march works
1. Repository shape
- This repo is split into two layers: verl/ is the main distributed RL framework, and quarl/ is the MARCH-specific experiment layer that plugs custom fact-checking logic into that framework.
- The top-level README and quarl/examples/train_march.sh make it clear that MARCH is not a standalone app; it is a training recipe built on VeRL.
2. MARCH-specific pipeline
- MARCH implements a multi-agent fact-checking loop around RL training: a Solver generates an answer, a Proposer decomposes it into verifiable claims, and a Checker validates claims against retrieved documents.
- The key architectural goal is “blind verification”: checker-side reward signals should not depend on the solver’s hidden reasoning, only on the generated content and evidence.
3. Entry points and orchestration
- quarl/quarl/main_rl.py is the MARCH entrypoint.
- quarl/quarl/trainer/ray_trainer_with_fact_check_march.py is the main trainer integration point for the MARCH variant.
- In the base framework, verl/verl/trainer/main_ppo.py, main_generation.py, and main_eval.py play the same orchestration role for standard RL/eval flows.
4. Training control plane
- The core training loop is distributed and Ray-driven. VeRL provides the scheduling and worker lifecycle; the MARCH layer specializes the trainer and reward path.
- verl/verl/trainer/ppo/ray_trainer.py and the verl/verl/single_controller/* modules define the generic distributed control plane used by many recipes.
- MARCH’s quarl/quarl/trainer/* code customizes that control plane for fact-checking reward computation and rollout handling.
5. Worker boundary and model execution
- Worker code is split by role and backend:
- verl/verl/workers/actor/*
- verl/verl/workers/critic/*
- verl/verl/workers/rollout/*
- verl/verl/workers/reward_model/*
- Backends include FSDP and Megatron for training, and vLLM / SGLang for rollout and inference-side generation.
- quarl/quarl/worker/fsdp_worker.py is the MARCH-side worker specialization, indicating that MARCH leans on the FSDP path.
6. Reward and fact-checking subsystem
- Reward logic is a first-class subsystem in both layers:
- verl/verl/workers/reward_manager/* contains reusable reward manager implementations.
- verl/verl/utils/reward_score/* contains task-specific scoring utilities such as math, GSM8K, Geo3K, search, and sandbox-fusion scoring.
- quarl/quarl/reward/* adds MARCH-specific reward base/manager abstractions.
- The README’s CUSTOM_RM_ARGS and USE_CHECKER_PPO show that MARCH extends the reward manager to support custom labels and zero-tolerance/fact-check reward policies.
7. Data flow
- Training data is expected as Parquet with at least prompt and label fields, where prompts contain the user query plus retrieved documents.
- Data enters through dataset utilities in VeRL (verl/verl/utils/dataset/*, verl/verl/experimental/dataset/*) and MARCH helper code in quarl/quarl/utils/data_utils.py.
- The flow is: dataset loading → rollout/generation → claim decomposition/checking → reward computation → policy update.
8. Model and checkpoint management
- VeRL separates model definition/weight conversion from training logic:
- verl/verl/models/* contains Hugging Face, Megatron, and mcore adapters.
- verl/verl/model_merger/* and verl/verl/utils/checkpoint/* handle checkpoint conversion/merging/saving.
- This matters architecturally because the repo supports multiple model families and sharding formats, not a single fixed checkpoint layout.
- MARCH inherits this machinery rather than implementing its own model stack.
9. Tooling and external inference services
- Rollout paths can call external or semi-external inference engines: SGLang and vLLM are both supported, and there are sharding managers for colocated/distributed rollout.
- Tooling for multi-agent or retrieval-assisted tasks lives in verl/verl/tools/* and includes search, GSM8K, Geo3K, sandbox-fusion, and MCP-based tools.
- This is important because MARCH’s checker/proposer design depends on evidence retrieval and tool-assisted verification, not just plain text generation.
10. Configuration and reproducibility
- Configuration is YAML-driven across the repo:
- verl/verl/trainer/config/* for framework-level training settings
- quarl/quarl/config/rl_config.yaml for MARCH-specific settings
- many recipe-specific YAMLs under verl/recipe/*/config
- Shell scripts in quarl/examples/ and verl/examples/ are the operational boundary for launching experiments, usually with node/rank/env-var-based distributed setup.
11. Broader recipe library
- verl/recipe/* is a catalog of training recipes beyond MARCH: PPO, GRPO, DAPO, PRIME, SPIN, SPPO, R1 distillation, entropy-based training, SFT, and agent-loop experiments.
- These recipes reuse the same core framework but specialize different algorithmic or data-processing paths.
- For architecture diagrams, this means the repo is a platform with many experiment “leaf nodes,” not one monolithic training job.
12. Validation and test surface
- The repository includes a large test matrix under verl/tests/* covering datasets, workers, rollout backends, distributed behavior, model utilities, and end-to-end recipes.
- This test layout reinforces the system boundaries: dataset logic, worker execution, rollout engines, reward managers, and full training recipes are independently testable.
- MARCH-specific confidence comes from the special_e2e and interaction/rollout tests that exercise tool-based and multi-turn flows.
13. Overall architectural diagram takeaway
- A useful diagram should show quarl as a MARCH customization layer sitting on top of verl.
- The central runtime path is: Parquet data + configs → distributed trainer (Ray) → actor/critic/rollout workers → proposer/checker/reward managers → checkpoint/logging outputs.
- The key architectural differentiator is the fact-checking reward loop, where evidence-grounded checker signals shape policy updates to reduce hallucination.
Main entry points: ray_trainer.py PPO trainer · ray_trainer_with_fact_check_march.py March trainer · base.py Workers · README.md Recipe catalog
Components 27
MARCH layer
MARCH-specific experiment code layered on top of VeRL.
- MARCH entry · Launches the MARCH RL workflow.
quarl/quarl/main_rl.py - March trainer · Ray trainer specialization for fact-checking reward and rollout handling.
quarl/quarl/trainer/ray_trainer_with_fact_check_march.py - March reward · MARCH-specific reward abstractions and manager hooks.
quarl/quarl/reward/manager.py - Reward base · Base reward interfaces used by the MARCH layer.
quarl/quarl/reward/base.py - Reward iface · Interface boundary for plugging MARCH reward logic into the framework.
quarl/quarl/interface/reward.py - Data iface · Interface boundary for dataset and label handling.
quarl/quarl/interface/data.py - Data utils · MARCH helpers for loading and shaping training data.
quarl/quarl/utils/data_utils.py - FSDP worker · MARCH-side worker specialization aligned with the FSDP execution path.
quarl/quarl/worker/fsdp_worker.py
VeRL core
Distributed RL framework, training control plane, workers, and utilities.
- PPO main · Standard PPO training entrypoint.
verl/verl/trainer/main_ppo.py - Gen main · Standard generation entrypoint.
verl/verl/trainer/main_generation.py - Eval main · Standard evaluation entrypoint.
verl/verl/trainer/main_eval.py - PPO trainer · Generic distributed trainer orchestration for PPO, generation, and eval flows.
verl/verl/trainer/ppo/ray_trainer.py - Single controller · Generic worker lifecycle, registration, and scheduling primitives.
verl/verl/single_controller/base/worker_group.py - Workers · Actor, critic, rollout, and reward-model workers.
verl/verl/workers/actor/base.py - Rollout · Generation backends and async rollout servers for HF, vLLM, and SGLang.
verl/verl/workers/rollout/sglang_rollout/sglang_rollout.py - Reward mgr · Reusable reward manager implementations and registry.
verl/verl/workers/reward_manager/registry.py - Score utils · Task-specific scoring utilities for math, GSM8K, Geo3K, search, and sandbox-fusion.
verl/verl/utils/reward_score/math.py - Datasets · RL, RM, SFT, multiturn, and experimental dataset loaders and samplers.
verl/verl/utils/dataset/rl_dataset.py - Models · Hugging Face, Megatron, and mcore model adapters and registries.
verl/verl/models/registry.py - Checkpoints · Checkpoint saving, loading, merging, and conversion utilities.
verl/verl/utils/checkpoint/checkpoint_manager.py - Tools · Search, GSM8K, Geo3K, sandbox-fusion, and MCP-based tools for agentic and retrieval-assisted flows.
verl/verl/tools/base_tool.py - Configs · YAML-driven trainer, worker, rollout, and recipe configuration.
verl/verl/trainer/config/config.py
Recipes
Algorithm and task recipes built on the shared framework.
- Recipe catalog · Algorithm-specific recipes such as PPO, GRPO, DAPO, PRIME, SPIN, SPPO, R1 distillation, SFT, and agent loops.
verl/recipe/README.md - DAPO · Recipe specialization for DAPO training.
verl/recipe/dapo/main_dapo.py - PRIME · Recipe specialization for PRIME training.
verl/recipe/prime/main_prime.py - R1 eval · Evaluation/distillation recipe for R1-style workflows.
verl/recipe/r1/main_eval.py
Other components
- README · Documents the repository as a MARCH recipe built on VeRL.
README.md
Connections 31
- README → MARCH entry: describes · Repository README frames MARCH as a VeRL-based recipe.
- MARCH entry → March trainer: uses · Entry point routes into the MARCH trainer.
- March trainer → PPO trainer: extends · MARCH customizes the generic Ray trainer control plane.
- March trainer → March reward: computes reward · Fact-checking reward is specialized in the MARCH layer.
- March reward → Reward base: builds on · Reward manager uses shared reward abstractions.
- March reward → Reward iface: implements · Interfaces expose the MARCH reward contract.
- March trainer → Data iface: consumes · Trainer consumes labeled prompt/document data through the interface.
- Data iface → Data utils: backed by · Data interface is supported by MARCH data helpers.
- March trainer → FSDP worker: dispatches · MARCH leans on the FSDP worker specialization.
- March trainer → Workers: orchestrates · Trainer schedules actor/critic/rollout worker roles.
- PPO trainer → Single controller: uses · Core trainer relies on single-controller scheduling primitives.
- PPO trainer → Workers: drives · Distributed training runs across worker roles.
- Workers → Rollout: includes · Rollout workers provide generation and inference backends.
- Workers → Reward mgr: includes · Reward manager is part of the worker execution boundary.
- Reward mgr → Score utils: scores with · Reward managers delegate to task-specific scorers.
- PPO trainer → Datasets: loads · Training loops begin with dataset ingestion.
- PPO trainer → Models: trains · Trainer operates over multiple model families and adapters.
- Models → Checkpoints: persists via · Model families share checkpoint conversion and saving machinery.
- Rollout → Tools: calls · Tool-assisted rollout and retrieval support agentic flows.
- PPO trainer → Configs: configured by · Framework behavior is YAML-driven.
- Recipe catalog → DAPO: contains · Recipe catalog includes DAPO.
- Recipe catalog → PRIME: contains · Recipe catalog includes PRIME.
- Recipe catalog → R1 eval: contains · Recipe catalog includes R1 evaluation/distillation.
- DAPO → PPO trainer: reuses · Recipe-specific entrypoints reuse the shared trainer stack.
- PRIME → PPO trainer: reuses · Recipe-specific entrypoints reuse the shared trainer stack.
- R1 eval → Datasets: reuses · Evaluation recipes reuse dataset and task utilities.
- March trainer → Reward mgr: overrides · MARCH extends reward management for blind verification policies.
- March trainer → Rollout: uses · MARCH rollout path produces solver answers and evidence-grounded outputs.
- PPO main → PPO trainer: launches · PPO entrypoint starts the distributed trainer.
- Gen main → PPO trainer: launches · Generation entrypoint starts the shared trainer stack.
- Eval main → PPO trainer: launches · Evaluation entrypoint starts the shared trainer stack.