How sail-sg/cpo works
Project Overview and Purpose
• The project, “Chain of Preference Optimization (CPO),” implements the ideas from the research paper on using Tree-of-Thought (ToT) techniques to optimize Chain-of-Thought (CoT) reasoning in large language models (LLMs).
• Its main goal is to fine-tune LLMs so that they learn to generate more deliberate and optimal reasoning paths by aligning CoT reasoning with preference information extracted from tree-search outputs.
Analyzing the File Structure
• At the top level, you have several key scripts:
• “clean_dataset.py” – for processing and cleaning collected data.
• “dpo_training.py” – for training the LLM using Direct Preference Optimization.
• “run_test.py” – for running tests, generating reasoning paths, and evaluating model output.
• “load_data.py” – likely handling data input routines.
• “requirement.txt” – detailing dependencies.
• The “tot” subdirectory is the heart of the project and includes:
• “llama_models.py” – integration/abstractions for LLM (possibly tailored for Llama).
• “methods/” – scripts like “bfs_test.py” that likely implement search or tree traversal algorithms.
• “prompts/” – containing specialized prompts (e.g., “bamboogle.py”, “fever.py”) for different tasks.
• “tasks/” – implementing the actual tasks (e.g., “bamboogle.py”, “fever.py”) and a shared “base.py” for common task definitions.
• Additionally, “Figures” is used to store graphical materials (like the provided introductory figure), which can be integrated if needed to illustrate key concepts.
Main Components
a. Data Collection & Tree-of-Thought Generation Module:
• Script: run_test.py (used with parameters for tasks and methods).
• Subcomponents: Task selection (e.g., “bamboogle”, “fever”) via corresponding scripts in tot/tasks.
• Methods and prompts are chosen from tot/methods and tot/prompts, respectively.
b. Data Cleaning Module:
• Script: clean_dataset.py processes the raw paired preference data output from ToT.
c. Model Training (Optimization) Module:
• Script: dpo_training.py fine-tunes the base LLM (e.g., Llama-2-7b-hf) using the cleaned and paired dataset.
• External logging / monitoring (e.g., wandb integrated as per the README instructions).
d. Testing / Inference Module:
• Running run_test.py with the “naive_run” mode for final testing.
e. Core LLM Integration & Utility Libraries:
• tot/llama_models.py – integration point for the base LLM.
• Other utility modules for task execution (from tot/tasks) and reasoning exploration (from tot/methods).
Relationships and Data Flow
• The data generated is then processed by the clean_dataset.py script.
• The resulting dataset is then fed into the dpo_training.py module to fine-tune the LLM.
• Post training, the updated model can be evaluated via a separate run_test.py invocation in testing mode.
• Each phase interacts with the LLM integration layer (tot/llama_models.py) to either generate reasoning paths or perform inference.
Architectural Patterns and Technology Stack
• The design follows a modular structure:
– Separation of data collection, processing, training, and testing.
– Clear separation of concerns within the “tot” package (models, methods, prompts, tasks).
• Key technologies include:
– Accelerate (for distributed or efficient model training/inference execution).
– LLMs (specifically models like Llama, as indicated by “Llama-2-7b-hf”).
– External tools for logging/visualization (e.g., wandb).
• The README hints at an external dependency on tree-search strategies (ToT) and direct preference optimization techniques (DPO), meaning parts of the code engage in search algorithms (e.g., BFS in “bfs_test.py”).