How bigscience-workshop/petals works
Main Components
• Client Components: Under src/petals/client, you have modules responsible for connecting to a distributed network, handling API calls for inference, fine-tuning, remote generation, and even special routing logic (e.g., sequence management).
• Server Components: Found under src/petals/server and the CLI tools in src/petals/cli, these pieces are responsible for hosting model blocks, handling incoming requests, caching, managing tasks, and performing block-specific computations.
• Model Definitions: In src/petals/models you see different subdirectories (bloom, falcon, llama, mixtral) that encapsulate model-specific configurations, model blocks, and inference details.
• Utility Modules: Under src/petals/utils and additional helper functions across the codebase, these cover networking (e.g., DHT for peer discovery), logging, asynchronous operations, and caching.
• Supporting Artifacts: Besides the core source code, you have benchmarks, examples (notebooks for prompt tuning, etc.), CI/CD workflows (.github), Dockerfile, and configuration files which reflect deployment, testing, and integration practices.
Relationships and Interactions
• Client-to-Server Communication: The client components in src/petals/client “discover” and connect to servers running via the CLI (e.g., run_server.py) and execute methods like remote generation, inference sessions, and routing of sequential operations.
• Internal Server Flow: Inside the server, show subcomponents such as:
• The task pool and prioritizers (handling incoming work).
• The networking/DHT utilities (for peer discovery and coordination).
• Memory caches and block utilities (for local storage and computational optimizations).
• Specific handlers for distributed forward and backward propagation.
• Model-specific Processing:
• Model folder (BLOOM, Falcon, Llama, Mixtral) as pluggable modules. They interact with both client and server components to ensure that, depending on the loaded model, the correct block definitions and configurations are applied.
Architectural Patterns & Design Principles
• Distributed/Micro-services style: Although not separated into independent services physically (since it’s packaged as a single project), logically the system is divided into nodes that communicate over the network.
• Modular Design: With clear separations between client, server, and model-specific layers, the project follows a modular design that allows plugging in multiple architectures for different LLMs.
• API/Command-Line Interface: The use of CLI modules for launching services and the client wrapper (which behaves like a remote API) should be highlighted.
• Extensibility: The separation of routing, caching, and DHT functionalities in distinct modules indicates an architecture aimed at extensibility.