# bigscience-workshop/petals architecture

> Architecture diagram of the GitHub repository bigscience-workshop/petals, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigscience-workshop/petals
- Repository: https://github.com/bigscience-workshop/petals

## Overview

##### Main Components
- Client Components: Under src/petals/client, you have modules responsible for connecting to a distributed network, handling API calls for inference, fine-tuning, remote generation, and even special routing logic (e.g., sequence management).
- Server Components: Found under src/petals/server and the CLI tools in src/petals/cli, these pieces are responsible for hosting model blocks, handling incoming requests, caching, managing tasks, and performing block-specific computations.
- Model Definitions: In src/petals/models you see different subdirectories (bloom, falcon, llama, mixtral) that encapsulate model-specific configurations, model blocks, and inference details.
- Utility Modules: Under src/petals/utils and additional helper functions across the codebase, these cover networking (e.g., DHT for peer discovery), logging, asynchronous operations, and caching.
- Supporting Artifacts: Besides the core source code, you have benchmarks, examples (notebooks for prompt tuning, etc.), CI/CD workflows (.github), Dockerfile, and configuration files which reflect deployment, testing, and integration practices.

##### Relationships and Interactions
- Client-to-Server Communication: The client components in src/petals/client “discover” and connect to servers running via the CLI (e.g., run_server.py) and execute methods like remote generation, inference sessions, and routing of sequential operations.
- Internal Server Flow: Inside the server, show subcomponents such as:
- The task pool and prioritizers (handling incoming work).
- The networking/DHT utilities (for peer discovery and coordination).
- Memory caches and block utilities (for local storage and computational optimizations).
- Specific handlers for distributed forward and backward propagation.
- Model-specific Processing:
- Model folder (BLOOM, Falcon, Llama, Mixtral) as pluggable modules. They interact with both client and server components to ensure that, depending on the loaded model, the correct block definitions and configurations are applied.

##### Architectural Patterns & Design Principles
- Distributed/Micro-services style: Although not separated into independent services physically (since it’s packaged as a single project), logically the system is divided into nodes that communicate over the network.
- Modular Design: With clear separations between client, server, and model-specific layers, the project follows a modular design that allows plugging in multiple architectures for different LLMs.
- API/Command-Line Interface: The use of CLI modules for launching services and the client wrapper (which behaves like a remote API) should be highlighted.
- Extensibility: The separation of routing, caching, and DHT functionalities in distinct modules indicates an architecture aimed at extensibility.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Distributed Inference System
    subgraph "Distributed Inference System"
        %% Client Components
        subgraph "Client Components"
            C1["Client: Inference & Routing"]:::client
        end

        %% Server Components
        subgraph "Server Components"
            S1["Server: Host Model Blocks"]:::server
            S2["CLI: run_server.py"]:::server
            S3["CLI: run_dht.py"]:::server
            S4["Task Pool & Caching"]:::server
            S5["Block Processing & DHT Utilities"]:::server
        end

        %% Model Definitions
        subgraph "Model Definitions"
            M1["BLOOM Model"]:::model
            M2["Falcon Model"]:::model
            M3["Llama Model"]:::model
            M4["Mixtral Model"]:::model
        end

        %% Utility Modules
        subgraph "Utility Modules"
            U1["Networking & DHT"]:::utility
            U2["Logging & Async Ops"]:::utility
        end

        %% Supporting Artifacts
        subgraph "Supporting Artifacts"
            A1["Benchmarks"]:::artifact
            A2["Examples/Notebooks"]:::artifact
            A3["CI/CD Workflows"]:::artifact
            A4["Dockerfile"]:::artifact
            A5["README"]:::artifact
        end
    end

    %% Connections between components
    C1 -->|"connects"| S1
    C1 -->|"initiates"| S2
    C1 -->|"initiates"| S3
    S1 -->|"manages"| S4
    S4 -->|"triggers"| S5

    %% Connections to Model Definitions
    S1 -->|"loads"| M1
    S1 -->|"loads"| M2
    S1 -->|"loads"| M3
    S1 -->|"loads"| M4

    %% Utility usage
    C1 -->|"uses"| U1
    S1 -->|"communicates"| U1
    S1 -->|"logs_to"| U2
    S2 -->|"runs_with"| U1
    S3 -->|"runs_with"| U1

    %% Distributed network communication between servers (via DHT)
    S1 <-->|"peer_comm"| U1

    %% Optional connection from Supporting Artifacts to core system (for deployment/integration context)
    A4 ---|"deploys"| S1
    A3 ---|"CI/CD_triggers"| S1

    %% Click Events for Component Mapping
    click C1 "https://github.com/bigscience-workshop/petals/tree/main/src/petals/client"
    click S1 "https://github.com/bigscience-workshop/petals/tree/main/src/petals/server"
    click S2 "https://github.com/bigscience-workshop/petals/blob/main/src/petals/cli/run_server.py"
    click S3 "https://github.com/bigscience-workshop/petals/blob/main/src/petals/cli/run_dht.py"
    click M1 "https://github.com/bigscience-workshop/petals/tree/main/src/petals/models/bloom"
    click M2 "https://github.com/bigscience-workshop/petals/tree/main/src/petals/models/falcon"
    click M3 "https://github.com/bigscience-workshop/petals/tree/main/src/petals/models/llama"
    click M4 "https://github.com/bigscience-workshop/petals/tree/main/src/petals/models/mixtral"
    click U1 "https://github.com/bigscience-workshop/petals/tree/main/src/petals/utils"
    click U2 "https://github.com/bigscience-workshop/petals/blob/main/src/petals/constants.py"
    click A1 "https://github.com/bigscience-workshop/petals/tree/main/benchmarks"
    click A2 "https://github.com/bigscience-workshop/petals/tree/main/examples"
    click A3 "https://github.com/bigscience-workshop/petals/tree/main/.github/workflows"
    click A4 "https://github.com/bigscience-workshop/petals/tree/main/Dockerfile"
    click A5 "https://github.com/bigscience-workshop/petals/blob/main/README.md"

    %% Styles
    classDef client fill:#9bf,stroke:#333,stroke-width:2px;
    classDef server fill:#afa,stroke:#333,stroke-width:2px;
    classDef model fill:#f9a,stroke:#333,stroke-width:2px;
    classDef utility fill:#c9f,stroke:#333,stroke-width:2px;
    classDef artifact fill:#ddd,stroke:#333,stroke-width:2px;
```
