How chutungduongg/react_iclr_2023_reproduction works
This repository is a command-line reproduction and benchmarking framework for seven reasoning methods on HotpotQA and FEVER. The main workflow is: a user starts a benchmark, the CLI resolves configuration and samples task examples, constructs a model and method-specific agent, and passes them to the experiment runner for prediction, evaluation, and artifact persistence. ReAct and Act agents use a Wikipedia environment; hybrid methods combine reasoning strategies. The map follows the README and sampled code; links for unsampled task and evaluation modules reflect documented capabilities, not inspected call sites.
Main entry points: cli.py CLI dispatcher · huggingface.py Hugging Face provider · react.py ReAct agent · act.py Act-only agent
Components 23
CLI and configuration
- Python entry point
main.py - CLI dispatcher
src/react_reproduction/cli.py - Run configuration
src/react_reproduction/config.py
Tasks and prompts
- HotpotQA loader
src/react_reproduction/datasets/hotpotqa.py - FEVER loader
src/react_reproduction/datasets/fever.py - Prompt suites
src/react_reproduction/prompts
Agents and retrieval
- Standard and CoT
src/react_reproduction/agents - CoT-SC voting
src/react_reproduction/agents/cot_sc.py - Act-only agent
src/react_reproduction/agents/act.py - ReAct agent
src/react_reproduction/agents/react.py - Hybrid agents
src/react_reproduction/agents/hybrid.py - Action parsing
src/react_reproduction/agents/parsing.py - Wikipedia environment
src/react_reproduction/tools/wikipedia.py
Model inference
- Hugging Face provider
src/react_reproduction/llm/huggingface.py
Benchmark outputs
- Benchmark runner
src/react_reproduction/experiments/runner.py - Task evaluators
src/react_reproduction/evaluation/evaluators.py - Metric calculations
src/react_reproduction/evaluation/metrics.py - Run artifacts
src/react_reproduction/experiments/artifacts.py - Saved benchmark runs
outputs
Other components
- Benchmark user
- MediaWiki API
- Hugging Face models
- Hugging Face datasets
Connections 31
- Benchmark user → Python entry point: starts benchmark
- Python entry point → CLI dispatcher: dispatches
- CLI dispatcher → Run configuration: loads settings
- CLI dispatcher → HotpotQA loader: loads examples
- CLI dispatcher → FEVER loader: loads examples
- HotpotQA loader → Hugging Face datasets: loads split
- FEVER loader → Hugging Face datasets: loads split
- CLI dispatcher → Hugging Face provider: constructs model
- Hugging Face provider → Hugging Face models: loads model
- CLI dispatcher → Standard and CoT: selects method
- CLI dispatcher → CoT-SC voting: selects method
- CLI dispatcher → Act-only agent: selects method
- CLI dispatcher → ReAct agent: selects method
- CLI dispatcher → Hybrid agents: selects method
- CLI dispatcher → Prompt suites: selects task prompts
- Standard and CoT → Hugging Face provider: generates answers
- CoT-SC voting → Hugging Face provider: samples generations
- Act-only agent → Hugging Face provider: generates actions
- ReAct agent → Hugging Face provider: generates turns
- Act-only agent → Action parsing: parses actions
- ReAct agent → Action parsing: parses turns
- Act-only agent → Wikipedia environment: executes actions
- ReAct agent → Wikipedia environment: executes actions
- Wikipedia environment → MediaWiki API: searches and fetches
- Hybrid agents → ReAct agent: invokes strategy
- Hybrid agents → CoT-SC voting: invokes strategy
- CLI dispatcher → Benchmark runner: runs benchmark
- Benchmark runner → Task evaluators: evaluates predictions
- Task evaluators → Metric calculations: computes scores
- Benchmark runner → Run artifacts: writes run records
- Run artifacts → Saved benchmark runs: persists files