# sail-sg/cpo architecture

> Architecture diagram of the GitHub repository sail-sg/cpo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sail-sg/cpo
- Repository: https://github.com/sail-sg/cpo

## Overview

##### Project Overview and Purpose
- The project, “Chain of Preference Optimization (CPO),” implements the ideas from the research paper on using Tree-of-Thought (ToT) techniques to optimize Chain-of-Thought (CoT) reasoning in large language models (LLMs).
- Its main goal is to fine-tune LLMs so that they learn to generate more deliberate and optimal reasoning paths by aligning CoT reasoning with preference information extracted from tree-search outputs.

##### Analyzing the File Structure
- At the top level, you have several key scripts:
- “clean_dataset.py” – for processing and cleaning collected data.
- “dpo_training.py” – for training the LLM using Direct Preference Optimization.
- “run_test.py” – for running tests, generating reasoning paths, and evaluating model output.
- “load_data.py” – likely handling data input routines.
- “requirement.txt” – detailing dependencies.
- The “tot” subdirectory is the heart of the project and includes:
- “llama_models.py” – integration/abstractions for LLM (possibly tailored for Llama).
- “methods/” – scripts like “bfs_test.py” that likely implement search or tree traversal algorithms.
- “prompts/” – containing specialized prompts (e.g., “bamboogle.py”, “fever.py”) for different tasks.
- “tasks/” – implementing the actual tasks (e.g., “bamboogle.py”, “fever.py”) and a shared “base.py” for common task definitions.
- Additionally, “Figures” is used to store graphical materials (like the provided introductory figure), which can be integrated if needed to illustrate key concepts.

##### Main Components
a. Data Collection & Tree-of-Thought Generation Module:
- Script: run_test.py (used with parameters for tasks and methods).
- Subcomponents: Task selection (e.g., “bamboogle”, “fever”) via corresponding scripts in tot/tasks.
- Methods and prompts are chosen from tot/methods and tot/prompts, respectively.
b. Data Cleaning Module:
- Script: clean_dataset.py processes the raw paired preference data output from ToT.
c. Model Training (Optimization) Module:
- Script: dpo_training.py fine-tunes the base LLM (e.g., Llama-2-7b-hf) using the cleaned and paired dataset.
- External logging / monitoring (e.g., wandb integrated as per the README instructions).
d. Testing / Inference Module:
- Running run_test.py with the “naive_run” mode for final testing.
e. Core LLM Integration & Utility Libraries:
- tot/llama_models.py – integration point for the base LLM.
- Other utility modules for task execution (from tot/tasks) and reasoning exploration (from tot/methods).

##### Relationships and Data Flow
- The data generated is then processed by the clean_dataset.py script.
- The resulting dataset is then fed into the dpo_training.py module to fine-tune the LLM.
- Post training, the updated model can be evaluated via a separate run_test.py invocation in testing mode.
- Each phase interacts with the LLM integration layer (tot/llama_models.py) to either generate reasoning paths or perform inference.

##### Architectural Patterns and Technology Stack
- The design follows a modular structure:
– Separation of data collection, processing, training, and testing.
– Clear separation of concerns within the “tot” package (models, methods, prompts, tasks).
- Key technologies include:
– Accelerate (for distributed or efficient model training/inference execution).
– LLMs (specifically models like Llama, as indicated by “Llama-2-7b-hf”).
– External tools for logging/visualization (e.g., wandb).
- The README hints at an external dependency on tree-search strategies (ToT) and direct preference optimization techniques (DPO), meaning parts of the code engage in search algorithms (e.g., BFS in “bfs_test.py”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Collection & ToT Generation Module"
        A["run_test.py (Data Collection)"]:::execScript
        subgraph "ToT Components"
            B["Tasks Module (tot/tasks)"]:::utility
            C["Methods Module (tot/methods)"]:::utility
            D["Prompts Module (tot/prompts)"]:::utility
            D1["bamboogle.py"]:::utility
            D2["fever.py"]:::utility
        end
        B --> A
        C --> A
        D --> A
        D1 --> D
        D2 --> D
    end

    E["clean_dataset.py"]:::execScript
    F["dpo_training.py"]:::execScript
    G["run_test.py (Naive Run)"]:::execScript
    H["LLM Integration (tot/llama_models.py)"]:::utility

    A -->|"generate_data"| E
    E -->|"cleaned_data"| F
    F -->|"trained_model"| G

    A -->|"LLM_connection"| H
    F -->|"LLM_connection"| H
    G -->|"LLM_connection"| H

    click A "https://github.com/sail-sg/cpo/blob/main/run_test.py"
    click B "https://github.com/sail-sg/cpo/tree/main/tot/tasks/"
    click C "https://github.com/sail-sg/cpo/blob/main/tot/methods/bfs_test.py"
    click D1 "https://github.com/sail-sg/cpo/blob/main/tot/prompts/bamboogle.py"
    click D2 "https://github.com/sail-sg/cpo/blob/main/tot/prompts/fever.py"
    click E "https://github.com/sail-sg/cpo/blob/main/clean_dataset.py"
    click F "https://github.com/sail-sg/cpo/blob/main/dpo_training.py"
    click G "https://github.com/sail-sg/cpo/blob/main/run_test.py"
    click H "https://github.com/sail-sg/cpo/blob/main/tot/llama_models.py"

    classDef execScript fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef utility fill:#90EE90,stroke:#333,stroke-width:2px;
```
