How one-and-done2022/tau-retail-agentic-grpo-longhorizon works
This repository is a long-horizon retail tool-agent post-training project: synthetic τ-bench tasks provide customer requests, agents interact with a simulated retail environment, and the resulting experience supports SFT and GRPO training before pass-rate evaluation. The README also documents Vanilla, State-, Constraint-, and CS-GRPO variants and fixed-protocol confirmatory evaluation. The graph combines those documented capabilities with paths in the tree; implementation-level wiring for several project-specific stages was not sampled, so uncertain connections are omitted.
Main entry points: ray_trainer.py GRPO training · check_retail_confirmatory_eval.py Confirmatory evaluation · retail_cs_grpo.py Agent interaction · env.py Retail environment
Components 17
Retail simulation
- Retail tasks
tau-bench/tau_bench/envs/retail/tasks_train.py - Retail environment
tau-bench/tau_bench/envs/retail/env.py - Retail tools
tau-bench/tau_bench/envs/retail/tools - Agent interaction
agentic-grpo-longhorizon/src/envs/retail_cs_grpo.py
Training data
- SFT collection
agentic-grpo-longhorizon/scripts/train/sft/collect_sft_data.py - SFT dataset
agentic-grpo-longhorizon/src/training/sft_dataset.py - GRPO parquet
agentic-grpo-longhorizon/scripts/train/grpo/build_grpo_parquet.py - Tool configuration
agentic-grpo-longhorizon/scripts/train/grpo/gen_tool_config.py
Agent training
- SFT training
agentic-grpo-longhorizon/scripts/train/sft/sft_train.py - Policy inference
agentic-grpo-longhorizon/src/models/vllm_policy.py - GRPO training
verl/verl/trainer/ppo/ray_trainer.py - Reward scoring
verl/verl/experimental/reward/reward_manager.py - Model checkpoints
verl/verl/utils/checkpoint/fsdp_checkpoint_manager.py
Evaluation
- Pass-k evaluation
agentic-grpo-longhorizon/src/evaluation/pass_k_eval.py - Confirmatory evaluation
agentic-grpo-longhorizon/scripts/eval/check_retail_confirmatory_eval.py - Evaluation results
agentic-grpo-longhorizon/scripts/eval/summarize_retail_confirmatory.py
Other components
- Experimenter
Connections 19
- Experimenter → Retail tasks: selects tasks
- Retail tasks → Retail environment: supplies scenarios
- Retail environment → Retail tools: exposes tools
- Agent interaction → Retail environment: interacts with
- SFT collection → Agent interaction: collects trajectories
- SFT collection → SFT dataset: prepares data
- SFT dataset → SFT training: feeds training
- Experimenter → GRPO parquet: prepares runs
- GRPO parquet → GRPO training: feeds training
- Tool configuration → GRPO training: configures tools
- GRPO training → Policy inference: updates policy
- GRPO training → Agent interaction: runs rollouts
- GRPO training → Reward scoring: requests scores
- GRPO training → Model checkpoints: saves model
- Policy inference → Confirmatory evaluation: provides policy
- Confirmatory evaluation → Pass-k evaluation: evaluates trajectories
- Confirmatory evaluation → Agent interaction: runs evaluation
- Confirmatory evaluation → Evaluation results: summarizes results
- Experimenter → Confirmatory evaluation: launches evaluation