How alpa-projects/alpa works
Main Components
• Core API and Parallelization Layer – This is the “alpa” package itself (e.g., alpa/api.py, alpa/parallel_method.py, alpa/parallel_plan.py). It exposes the decorator (@parallelize) and other high-level interfaces used by the client code.
• Distributed Training Modules – Break these down further:
– Collective Communication Modules: The “alpa/collective” folder (plus subfolders for various collective groups like nccl, gloo, and XLA-specific implementations) represents the components that manage low-level communication across devices.
– Pipeline Parallel Modules: Located in “alpa/pipeline_parallel”, these modules (e.g., layer_construction, schedules, runtime_emitter) set up and run the pipelined execution of model layers.
– Shard Parallel Modules: Found under “alpa/shard_parallel”, these files manage auto and manual sharding of model parameters.
– Device Mesh & Execution: “alpa/device_mesh.py”, “alpa/mesh_executable.py”, and “alpa/mesh_profiling.py” are involved in mapping computations to the physical devices.
• Model Definitions and Utilities – The “alpa/model” folder contains example neural network models (BERT, GPT, Conformer, etc.) and related utilities that can be used in training.
• Serving Components – Under “alpa/serve”, components like the “controller”, “http_util”, and “run” files enable the system’s large-scale model serving capabilities.
• Integration with Frameworks – The “alpa/torch” folder is for integration with PyTorch, enabling support for Torch-based training/inference.
• Auxiliary Components – Other folders (e.g., “benchmark”, “docs”, “examples”, “tests”, “docker”, and “build_jaxlib”) support testing, benchmarking, documentation, containerized deployment, and building external dependencies like jaxlib.
Relationships and Interactions
• Client/Front-end Interface: Users write training or serving code that uses the Alpa API. Their code calls functions decorated with @parallelize.
• Parallel Execution Engine: This layer (composed of the parallel plan, collective modules, and device mesh components) takes a single-device computation, performs graph analysis and partitioning, and outputs a distributed execution plan.
• Serving Pipeline: For inference, the serving modules accept requests (e.g., via HTTP as indicated by “http_util” and “controller”) that are routed into the distributed backend for large model inference.
• Integration with Third-Party Libraries: Note the dependencies on JAX/XLA, Ray, and possibly PyTorch. These integrations should be represented as external systems that interact with the core engine.
Architectural Patterns and Design Principles
• Modular and Layered Architecture: The source code is organized into clear modules (collective, pipeline_parallel, shard_parallel, etc.) that separately handle distinct responsibilities.
• Separation of Concerns: There is a clear divide between training, serving, and supporting utilities (benchmarking, testing, Docker deployments, and JAXLib building).
• Distributed and Data-Parallel Patterns: The system leverages data, operator, and pipeline parallelism, with dedicated modules to handle sharding and device communication.
• Extensibility: The roadmap and multi-framework integration (JAX, PyTorch) suggest a design that can be extended to accommodate varied backend requirements.