How yetianjhu/alphallm works
Main Components
• Policy Services: These are started via a dedicated script (e.g., server/policy_server.py) and use Llama-2-70B models to generate policy decisions.
• Value Service: A separate service (e.g., server/value_server.py) evaluates the quality of the solution paths.
• Outcome Reward Model (ORM) and Process Reward Model (PRM) Services: These are instantiated using different variants of the step server (see server/step_server_70b.py) and provide reward signals.
• MCTS Engine: This is managed via scripts (like scripts/run_mcts.sh) and the search module (search/experiment.py and search/mcts.py) that control the tree search process.
• Models: Under the models directory, you have the files that likely handle specific model logic (e.g., rmgpt_model.py, value_model.py).
• Utility Modules: The utils directory provides support functionality (such as dataset handling and common utilities).
Out Architectural Relationships
• The MCTS component is the central orchestration module that interacts with the policy, value, ORM, and PRM services.
• The Policy Service generates potential actions or policies to expand nodes in the search tree.
• The Value Service, along with the reward models, is hit to evaluate the quality of various nodes/paths.
• The search module ties these services together by executing simulations through the MCTS algorithm.
• Communication between these components is likely implemented via network port bindings, as indicated by the service startup commands in the README.
• “Models” component as a support layer that is reused by multiple services (policy and value). This reflects that model files are used by the corresponding service implementations.
• A block for “Utilities” to indicate helper functions such as dataset management, which support the overall infrastructure.
Architectural Patterns and Technologies
• Pattern of “self-improving loop” where MCTS decisions are iteratively refined by integrating outputs from the policy and reward evaluations.
• Use of tensor or model parallelism (as indicated by parameters such as --tensor-parallel-size) when relevant.
• That the network interfaces (host and port) are critical connectors between the modules.