# yetianjhu/alphallm architecture

> Architecture diagram of the GitHub repository yetianjhu/alphallm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yetianjhu/alphallm
- Repository: https://github.com/yetianjhu/alphallm

## Overview

##### Main Components
- Policy Services: These are started via a dedicated script (e.g., server/policy_server.py) and use Llama-2-70B models to generate policy decisions.
- Value Service: A separate service (e.g., server/value_server.py) evaluates the quality of the solution paths.
- Outcome Reward Model (ORM) and Process Reward Model (PRM) Services: These are instantiated using different variants of the step server (see server/step_server_70b.py) and provide reward signals.
- MCTS Engine: This is managed via scripts (like scripts/run_mcts.sh) and the search module (search/experiment.py and search/mcts.py) that control the tree search process.
- Models: Under the models directory, you have the files that likely handle specific model logic (e.g., rmgpt_model.py, value_model.py).
- Utility Modules: The utils directory provides support functionality (such as dataset handling and common utilities).

##### Out Architectural Relationships
- The MCTS component is the central orchestration module that interacts with the policy, value, ORM, and PRM services.
- The Policy Service generates potential actions or policies to expand nodes in the search tree.
- The Value Service, along with the reward models, is hit to evaluate the quality of various nodes/paths.
- The search module ties these services together by executing simulations through the MCTS algorithm.
- Communication between these components is likely implemented via network port bindings, as indicated by the service startup commands in the README.

- “Models” component as a support layer that is reused by multiple services (policy and value). This reflects that model files are used by the corresponding service implementations.
- A block for “Utilities” to indicate helper functions such as dataset management, which support the overall infrastructure.

##### Architectural Patterns and Technologies
- Pattern of “self-improving loop” where MCTS decisions are iteratively refined by integrating outputs from the policy and reward evaluations.
- Use of tensor or model parallelism (as indicated by parameters such as --tensor-parallel-size) when relevant.
- That the network interfaces (host and port) are critical connectors between the modules.

## Diagram (Mermaid)

```mermaid
graph TD
    subgraph "MCTS Engine"
        mcts1("Run MCTS")
        mcts2("Search Experiment")
        mcts3("MCTS Logic")
    end

    subgraph "Services"
        ps("Policy Service (Llama-2-70B)")
        vs("Value Service")
        orm("ORM Service")
        prm("PRM Service")
    end

    subgraph "Support Layer"
        models("Models")
        utils("Utilities")
    end

    mcts1 -->|"sends_request"| ps
    mcts2 -->|"sends_request"| vs
    mcts3 -->|"sends_request"| orm
    mcts3 -->|"sends_request"| prm

    ps -->|"uses"| models
    vs -->|"uses"| models
    ps -->|"utilizes"| utils
    vs -->|"utilizes"| utils
    orm -->|"utilizes"| utils
    prm -->|"utilizes"| utils

    click ps "https://github.com/yetianjhu/alphallm/blob/main/scripts/start_policy.sh"
    click vs "https://github.com/yetianjhu/alphallm/blob/main/server/value_server.py"
    click orm "https://github.com/yetianjhu/alphallm/blob/main/server/step_server_70b.py"
    click prm "https://github.com/yetianjhu/alphallm/blob/main/scripts/start_step_reward_70b.sh"
    click mcts1 "https://github.com/yetianjhu/alphallm/blob/main/scripts/run_mcts.sh"
    click mcts2 "https://github.com/yetianjhu/alphallm/blob/main/search/experiment.py"
    click mcts3 "https://github.com/yetianjhu/alphallm/blob/main/search/mcts.py"
    click models "https://github.com/yetianjhu/alphallm/tree/main/models/"
    click utils "https://github.com/yetianjhu/alphallm/tree/main/utils/"

    class mcts1,mcts2,mcts3 mcts
    class ps,vs,orm,prm service
    class models model
    class utils utility

    classDef mcts fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef service fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef model fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef utility fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
