# lmarena/p2l architecture

> Architecture diagram of the GitHub repository lmarena/p2l, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lmarena/p2l
- Repository: https://github.com/lmarena/p2l

## Overview

##### Main Components
- P2L Model Training Module: Represent the training code under the “p2l” module (with scripts like train.py, eval.py, and various YAML configurations in “training_configs”) and the associated “deepspeed” setup. This module is focused on training large language models with different heads and loss functions.
- P2L Inference/Endpoint Module: The p2l/endpoint.py file and associated utilities (in “model.py”, “auto_evals.py”, etc.) create a service for running inference on trained models. Include details such as GPU/CPU selection and command-line arguments (e.g., control over model-type, head-type, and loss-type).
- Router / Cost Optimizer Service: Under the “route” directory, there is a separate service which wraps the P2L endpoints into an OpenAI compatible API. This component handles the cost-optimization logic, using different routing types (e.g., bt-endpoint for Bradley-Terry models and grk-endpoint for Grounded RK models) and cost optimizers (optimal-lp, simple-lp).
- Supporting Scripts and Configuration: Include the various requirement files (train_requirements.txt, serve_requirements.txt, route/requirements.txt) and configuration files (example_config.yaml, multiple training configuration YAML files) that define how each component is set up. Also include the “fast_runpod_setup.sh” and “.gitignore” for deployment and version control.
- AutoEval Suite: This component (exposed via p2l/auto_evals.py and auto_eval_utils.py) provides an evaluation pipeline for automated metrics collection and leaderboard generation.

##### Relationships and Interactions
- Training Pipeline: Inputs are training configuration files (YAML) and data (datasets specified by the user). The training module uses these configurations to train a P2L model via deepspeed and outputs model checkpoints.
- Inference Service: The p2l endpoint loads a model (CPU/GPU based on flags) and exposes endpoints that expect a conversation (prompt list) to produce coefficient output vectors. This component interacts with the training outputs.
- Router Service: Sits in front of (or alongside) the inference service to provide an OpenAI-compatible interface. It accepts user prompts and forwards them to the appropriate P2L inference endpoint, applying cost routing logic via internal configuration.
- External Clients: They interact with the Router and/or directly with the P2L endpoints via REST APIs, using provided examples in the README.
- Configuration/Management Layer: Each service (training, inference, router) is configured via its respective requirement files and configuration scripts. This ensures that dependencies, resource allocation, and environment-specific settings are captured.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Separate modules for training, inference, and routing make it easier to manage and scale components individually.
- Extensibility: The presence of multiple training config files and support for multiple model types (qwen2, Llama, etc.) suggests the design is meant to work across different model families and evaluation strategies.
- Configurability and Service Orientation: Notice how many command-line arguments are exposed for both the inference and router services; these service interfaces direct how components interact.

##### Relevant Technologies, Frameworks, and Libraries
- Python 3.10 (core language environment)
- Deepspeed for distributed training (indicated by “deepspeed” folder and usage in training commands)
- YAML for configurations (training_configs and example routing configuration)
- REST/HTTP for endpoint communication (the OpenAI compatibility in the router endpoint)
- External libraries like “uv” for fast environment setup, and requests (as shown in the example code)

##### What
- Training Module (p2l.train, training_configs, and deepspeed)
- Inference Module (p2l.endpoint, model.py, and associated utilities)
- Router Service (route/openai_server, routers, and cost_optimizers)
- AutoEval Suite (p2l.auto_evals and auto_eval_utils)
- Configuration/Data Files (requirements files, configuration YAMLs, etc.)
- Data flow from training outputs (model checkpoints) into the Inference Module
- API requests flowing from external clients to the Router Service, and from Router Service to the P2L Inference Service
- Administrative or update loops from configuration files influencing each service

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Client
    ExternalClient["External Client"]:::external

    %% Router Service Subgraph
    subgraph "Router Service"
        R_openai["openai_server.py"]:::router
        R_routers["routers.py"]:::router
        R_cost["cost_optimizers.py"]:::router
        R_example["example_config.yaml"]:::router
        R_reqs["requirements.txt"]:::router
    end

    %% Inference Module Subgraph
    subgraph "P2L Inference Module"
        I_endpoint["endpoint.py"]:::inference
        I_model["model.py"]:::inference
    end

    %% Training Module Subgraph
    subgraph "P2L Model Training Module"
        T_train["train.py"]:::training
        T_eval["eval.py"]:::training
        T_configs["training_configs (YAML)"]:::training
        T_deepspeed["deepspeed (config: zero1.json)"]:::training
    end

    %% AutoEval Suite Subgraph
    subgraph "AutoEval Suite"
        A_auto["auto_evals.py"]:::autoeval
        A_utils["auto_eval_utils.py"]:::autoeval
    end

    %% Supporting Scripts & Configuration Subgraph
    subgraph "Supporting Scripts & Config"
        S_fast["fast_runpod_setup.sh"]:::config
        S_git[".gitignore"]:::config
        S_train_req["train_requirements.txt"]:::config
        S_serve_req["serve_requirements.txt"]:::config
    end

    %% Connections
    ExternalClient -->|"REST_Request"| R_openai
    R_openai -->|"forward_prompt"| I_endpoint
    T_train -->|"model_checkpoint"| I_endpoint
    A_auto -->|"metrics_to"| T_train
    A_auto -->|"metrics_to"| I_endpoint
    S_fast ---|"deployment"| T_train
    S_fast ---|"deployment"| I_endpoint
    S_fast ---|"deployment"| R_openai

    %% Styles
    classDef training fill:#AED6F1,stroke:#2980B9,stroke-width:2px;
    classDef inference fill:#ABEBC6,stroke:#229954,stroke-width:2px;
    classDef router fill:#F9E79F,stroke:#F1C40F,stroke-width:2px;
    classDef autoeval fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
    classDef config fill:#E5E7E9,stroke:#95A5A6,stroke-width:2px;
    classDef external fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;

    %% Click Events for Training Module
    click T_train "https://github.com/lmarena/p2l/blob/main/p2l/train.py"
    click T_eval "https://github.com/lmarena/p2l/blob/main/p2l/eval.py"
    click T_configs "https://github.com/lmarena/p2l/tree/main/training_configs/"
    click T_deepspeed "https://github.com/lmarena/p2l/blob/main/deepspeed/zero1.json"

    %% Click Events for Inference Module
    click I_endpoint "https://github.com/lmarena/p2l/blob/main/p2l/endpoint.py"
    click I_model "https://github.com/lmarena/p2l/blob/main/p2l/model.py"

    %% Click Events for Router Service
    click R_openai "https://github.com/lmarena/p2l/blob/main/route/openai_server.py"
    click R_routers "https://github.com/lmarena/p2l/blob/main/route/routers.py"
    click R_cost "https://github.com/lmarena/p2l/blob/main/route/cost_optimizers.py"
    click R_example "https://github.com/lmarena/p2l/blob/main/route/example_config.yaml"
    click R_reqs "https://github.com/lmarena/p2l/blob/main/route/requirements.txt"

    %% Click Events for Supporting Scripts & Config
    click S_fast "https://github.com/lmarena/p2l/blob/main/fast_runpod_setup.sh"
    click S_git "https://github.com/lmarena/p2l/blob/main/.gitignore"
    click S_train_req "https://github.com/lmarena/p2l/blob/main/train_requirements.txt"
    click S_serve_req "https://github.com/lmarena/p2l/blob/main/serve_requirements.txt"

    %% Click Events for AutoEval Suite
    click A_auto "https://github.com/lmarena/p2l/blob/main/p2l/auto_evals.py"
    click A_utils "https://github.com/lmarena/p2l/blob/main/p2l/auto_eval_utils.py"
```
