How yyhdbl/shopping-grpo-longhorizon works
Shopping GRPO is a reproducible post-training and evaluation system for long-horizon shopping agents. Its main workflow collects teacher demonstrations, filters them into action-focused SFT data, trains with LoRA and online GRPO in ShopSimulator, then evaluates models on a shared held-out benchmark. The embedded simulator covers search through purchase and terminal reward; evaluation separately produces rubric, trajectory-judge, deterministic, and reward results. The README documents several capabilities whose implementations were not sampled, so those are represented at supported module or directory paths without assuming internal call wiring.
Main entry points: rollout.py Evaluation rollout · client.py Shop API client · trajectory.py Trajectory judge · grpo Online GRPO
Components 23
Data collection
- Trajectory acceptance
src/shopping_grpo/collection/sft.py - SFT dataset
src/shopping_grpo/training/sft/dataset.py
Model training
- LoRA SFT
src/shopping_grpo/training/sft - Online GRPO
src/shopping_grpo/training/grpo - Agent loop
src/shopping_grpo/training/grpo/adapter/agent_loop.py - Environment session
src/shopping_grpo/training/grpo/adapter/session.py - Tool adapter
src/shopping_grpo/training/grpo/adapter/tools.py
Shopping environment
- Shop API client
src/shopping_grpo/environment/client.py - ShopSimulator service
environments/ShopSimulator/shop_env/shop_env/shop_agent.py - Shopping sessions
environments/ShopSimulator/shop_env/web_agent_site/app.py - Product search
environments/ShopSimulator/shop_env/web_agent_site/engine/search.py - Shopping engine
environments/ShopSimulator/shop_env/web_agent_site/engine/engine.py - Terminal reward
environments/ShopSimulator/shop_env/web_agent_site/envs/web_agent_text_env.py
Evaluation
- Benchmark tasks
src/shopping_grpo/evaluation/manifest.py - Evaluation rollout
src/shopping_grpo/evaluation/rollout.py - Rubric curation
src/shopping_grpo/evaluation/rubric.py - Trajectory judge
src/shopping_grpo/evaluation/trajectory.py - Hard checks
src/shopping_grpo/evaluation/metrics.py - Results and comparison
src/shopping_grpo/evaluation/results.py
Other components
- Experimenter
- Teacher model
- Trained actor
- Judge model service
Connections 32
- Experimenter → Teacher model: requests demonstrations
- Teacher model → Shop API client: provides actions
- Shop API client → ShopSimulator service: sends actions
- ShopSimulator service → Shop API client: returns observations
- Shop API client → Trajectory acceptance: records rollouts
- Trajectory acceptance → SFT dataset: filters and builds
- SFT dataset → LoRA SFT: trains on
- LoRA SFT → Trained actor: produces model
- Trained actor → Online GRPO: initializes training
- Online GRPO → Agent loop: runs trajectories
- Agent loop → Environment session: opens session
- Agent loop → Tool adapter: dispatches tools
- Environment session → Shop API client: leases environment
- Tool adapter → Shop API client: executes actions
- Online GRPO → Trained actor: updates policy
- Experimenter → Benchmark tasks: selects benchmark
- Benchmark tasks → Evaluation rollout: provides tasks
- Trained actor → Evaluation rollout: generates actions
- Evaluation rollout → Shop API client: runs evaluation
- Evaluation rollout → Rubric curation: provides task evidence
- Rubric curation → Judge model service: requests rubric curation
- Rubric curation → Trajectory judge: supplies frozen rubric
- Evaluation rollout → Trajectory judge: provides trajectories
- Trajectory judge → Judge model service: requests scoring
- Evaluation rollout → Hard checks: checks outcomes
- Evaluation rollout → Results and comparison: provides rollout results
- Trajectory judge → Results and comparison: provides judge scores
- Hard checks → Results and comparison: provides hard checks
- ShopSimulator service → Shopping sessions: serves interactions
- Shopping sessions → Product search: searches products
- Shopping sessions → Shopping engine: renders product actions
- Shopping engine → Terminal reward: evaluates purchase