How princeton-nlp/tree-of-thought-llm works
Main components
• Task Layer: This is where the specific problem tasks are defined (see “src/tot/tasks” with its task modules for game24, crosswords, and text). Also include the data files under “src/tot/data” which provide input instances (e.g., 24.csv, crossword JSON files, text data).
• Method Layer: Contains the algorithms/strategies for exploring “thoughts.” In “src/tot/methods”, you have the BFS (and by extension, DFS in crosswords). These methods implement the core algorithmic ideas from the paper.
• API/LLM Interaction: The system uses GPT-4 (or similar language models) via the OpenAI API. Make sure to mark the external dependency on the OpenAI service (and configuration via the OPENAI_API_KEY).
• Experiment Logs: The “logs” directory contains experimental outputs and trajectories from running experiments for various tasks, giving insight into the performance of different methods (BFS, DFS, Naive sampling, etc.).
• Utility & Deployment Scripts: There are shell scripts and notebooks (located in “scripts” and a Jupyter Notebook in “scripts/crosswords”) which trigger various experimental runs.
• Entry Point: The “run.py” acts as a simple executable that ties together the arguments (e.g., temperature, selection method parameters) and invokes the chosen algorithm on a specific task, showcasing the overall control flow.
• From the Task, show access to its corresponding “Data” (stored under src/tot/data) and “Prompts” (from src/tot/prompts) used to interact with the LLM.
• The “Method Layer” (with modules like bfs.py) receives these prompts and orchestrates the idea of state generation, evaluation, and selection. This layer includes key parameters (like n_generate_sample, n_evaluate_sample, etc.)
• Once the LLM returns outputs, these are passed back to the method layer for evaluation and processing.
• Finally, outputs (the deduced answer or trajectory) are stored in logs and optionally visualized or printed for further analysis.
Architectural Patterns and Principles
• Modular Design: The separation between tasks, methods, prompts, and data encapsulates specific functionalities.
• Separation of Concerns: Different directories show clear boundaries (e.g., scripts for experiments vs. the src directory for core logic).
Specific Elements
• A clear label for each component (“Task Selector”, “Data Loader”, “Prompt Manager”, “Algorithm Executor”, “LLM Service”, “Logger”).
• From “run.py” → “Task Module” (Game24Task, CrosswordsTask, TextTask).
• From the “Task Module” → “Data” & “Prompts”.
• From “Algorithm Module” (e.g., BFS) → “LLM API” (sending prompts and receiving thought outputs).
• From “LLM API” back to “Algorithm Module” → “Logger” (persisting the trails).
• Use rectangles for internal modules.
Final Advice
• Remember that the tree-of-thoughts system is an iterative chain-of-thought approach that leverages LLMs in a non-deterministic manner—so ensuring that the flow from configuration to experimental logging is clearly illustrated will help capture the design’s intent.
Happy diagramming!