# hpcaitech/colossalai architecture

> Architecture diagram of the GitHub repository hpcaitech/colossalai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hpcaitech/colossalai
- Repository: https://github.com/hpcaitech/colossalai

## Overview

Colossal-AI is a framework for making large-model training and inference more scalable through distributed execution, sharding, and memory management. The clearest user-to-result path in the available evidence is an inference request entering the LLM engine, which prepares and batches prompts, shards the model, and generates tokens. Separate branches represent training schedules and auto-parallel sharding. The source index and excerpts establish several internal links; other major capabilities are represented without claiming unshown wiring.

Main entry points: [`colossalai/inference/core/llm_engine.py`](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/llm_engine.py) (LLM Engine), [`colossalai/inference/core/rpc_engine.py`](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/rpc_engine.py) (RPC Engine), [`colossalai/shardformer/policies/base_policy.py`](https://github.com/hpcaitech/colossalai/blob/main/colossalai/shardformer/policies/base_policy.py) (Shardformer Policies), [`colossalai/pipeline/stage_manager.py`](https://github.com/hpcaitech/colossalai/blob/main/colossalai/pipeline/stage_manager.py) (Pipeline Stages)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Inference"]
  node_inference_config["Inference Config<br/>[config.py]"]
  node_llm_engine["LLM Engine<br/>[llm_engine.py]"]
  node_async_engine["Async Engine<br/>[async_engine.py]"]
  node_rpc_engine["RPC Engine<br/>[rpc_engine.py]"]
  node_request_batching["Request Batching<br/>[batch_bucket.py]"]
  node_request_state["Request State<br/>[struct.py]"]
  node_sampling["Token Sampling<br/>[sampler.py]"]
  node_kv_cache[("KV Cache<br/>[kvcache_manager.py]")]
  node_rpc_worker["RPC Worker<br/>[rpc_worker.py]"]
end

subgraph group_parallel["Parallel Execution"]
  node_shardformer["Shardformer Policies<br/>[base_policy.py]"]
  node_parallel_embedding["Embedding Sharding"]
  node_stage_manager["Pipeline Stages<br/>[stage_manager.py]"]
  node_process_groups["Process Groups"]
end

subgraph group_training["Training and Memory"]
  node_pipeline_schedule["Pipeline Schedule"]
  node_offload_manager["Offload Manager<br/>[region_manager.py]"]
  node_gemini_chunks["Gemini Chunks<br/>[manager.py]"]
end

subgraph group_foundation["Runtime Foundation"]
  node_accelerator["Accelerator API<br/>[__init__.py]"]
  node_checkpoint_io["Checkpoint IO"]
  node_logging["Distributed Logging<br/>[__init__.py]"]
end

node_client(("Inference Client"))

node_client -.->|"submits prompts"| node_llm_engine
node_llm_engine -->|"uses settings"| node_inference_config
node_llm_engine -->|"batches requests"| node_request_batching
node_llm_engine -->|"creates sequences"| node_request_state
node_llm_engine -->|"selects tokens"| node_sampling
node_llm_engine -.->|"allocates cache"| node_kv_cache
node_llm_engine -->|"shards model"| node_shardformer
node_async_engine -->|"delegates inference"| node_llm_engine
node_rpc_engine -->|"dispatches work"| node_rpc_worker
node_rpc_engine -->|"uses batching"| node_request_batching
node_rpc_engine -->|"uses policies"| node_shardformer
node_shardformer -->|"uses stage manager"| node_stage_manager
node_stage_manager -->|"uses process mesh"| node_process_groups
node_pipeline_schedule -->|"communicates stages"| node_process_groups
node_pipeline_schedule -.->|"schedules pipeline"| node_stage_manager
node_offload_manager -.->|"coordinates chunks"| node_gemini_chunks
node_llm_engine -->|"selects device"| node_accelerator
node_llm_engine -->|"logs runtime"| node_logging

click node_inference_config "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/config.py"
click node_llm_engine "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/llm_engine.py"
click node_async_engine "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/async_engine.py"
click node_rpc_engine "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/rpc_engine.py"
click node_request_batching "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/batch_bucket.py"
click node_request_state "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/struct.py"
click node_sampling "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/sampler.py"
click node_kv_cache "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/kv_cache/kvcache_manager.py"
click node_rpc_worker "https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/executor/rpc_worker.py"
click node_shardformer "https://github.com/hpcaitech/colossalai/blob/main/colossalai/shardformer/policies/base_policy.py"
click node_parallel_embedding "https://github.com/hpcaitech/colossalai/blob/main/colossalai/auto_parallel/tensor_shard/node_handler/embedding_handler.py"
click node_stage_manager "https://github.com/hpcaitech/colossalai/blob/main/colossalai/pipeline/stage_manager.py"
click node_process_groups "https://github.com/hpcaitech/colossalai/blob/main/colossalai/cluster/process_group_manager.py"
click node_pipeline_schedule "https://github.com/hpcaitech/colossalai/blob/main/colossalai/legacy/engine/schedule/_pipeline_schedule.py"
click node_offload_manager "https://github.com/hpcaitech/colossalai/blob/main/colossalai/auto_parallel/offload/region_manager.py"
click node_gemini_chunks "https://github.com/hpcaitech/colossalai/blob/main/colossalai/zero/gemini/chunk/manager.py"
click node_accelerator "https://github.com/hpcaitech/colossalai/blob/main/colossalai/accelerator/__init__.py"
click node_checkpoint_io "https://github.com/hpcaitech/colossalai/blob/main/colossalai/checkpoint_io/general_checkpoint_io.py"
click node_logging "https://github.com/hpcaitech/colossalai/blob/main/colossalai/logging/__init__.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_inference_config,node_llm_engine,node_async_engine,node_rpc_engine,node_request_batching,node_request_state,node_sampling,node_kv_cache,node_rpc_worker,node_client toneBlue
class node_shardformer,node_parallel_embedding,node_stage_manager,node_process_groups toneAmber
class node_pipeline_schedule,node_offload_manager,node_gemini_chunks toneMint
class node_accelerator,node_checkpoint_io,node_logging toneRose
```

## Components

### Inference

- [Inference Config](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/config.py): `colossalai/inference/config.py`
- [LLM Engine](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/llm_engine.py): `colossalai/inference/core/llm_engine.py`
- [Async Engine](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/async_engine.py): `colossalai/inference/core/async_engine.py`
- [RPC Engine](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/core/rpc_engine.py): `colossalai/inference/core/rpc_engine.py`
- [Request Batching](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/batch_bucket.py): `colossalai/inference/batch_bucket.py`
- [Request State](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/struct.py): `colossalai/inference/struct.py`
- [Token Sampling](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/sampler.py): `colossalai/inference/sampler.py`
- [KV Cache](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/kv_cache/kvcache_manager.py): `colossalai/inference/kv_cache/kvcache_manager.py`
- [RPC Worker](https://github.com/hpcaitech/colossalai/blob/main/colossalai/inference/executor/rpc_worker.py): `colossalai/inference/executor/rpc_worker.py`

### Parallel Execution

- [Shardformer Policies](https://github.com/hpcaitech/colossalai/blob/main/colossalai/shardformer/policies/base_policy.py): `colossalai/shardformer/policies/base_policy.py`
- [Embedding Sharding](https://github.com/hpcaitech/colossalai/blob/main/colossalai/auto_parallel/tensor_shard/node_handler/embedding_handler.py): `colossalai/auto_parallel/tensor_shard/node_handler/embedding_handler.py`
- [Pipeline Stages](https://github.com/hpcaitech/colossalai/blob/main/colossalai/pipeline/stage_manager.py): `colossalai/pipeline/stage_manager.py`
- [Process Groups](https://github.com/hpcaitech/colossalai/blob/main/colossalai/cluster/process_group_manager.py): `colossalai/cluster/process_group_manager.py`

### Training and Memory

- [Pipeline Schedule](https://github.com/hpcaitech/colossalai/blob/main/colossalai/legacy/engine/schedule/_pipeline_schedule.py): `colossalai/legacy/engine/schedule/_pipeline_schedule.py`
- [Offload Manager](https://github.com/hpcaitech/colossalai/blob/main/colossalai/auto_parallel/offload/region_manager.py): `colossalai/auto_parallel/offload/region_manager.py`
- [Gemini Chunks](https://github.com/hpcaitech/colossalai/blob/main/colossalai/zero/gemini/chunk/manager.py): `colossalai/zero/gemini/chunk/manager.py`

### Runtime Foundation

- [Accelerator API](https://github.com/hpcaitech/colossalai/blob/main/colossalai/accelerator/__init__.py): `colossalai/accelerator/__init__.py`
- [Checkpoint IO](https://github.com/hpcaitech/colossalai/blob/main/colossalai/checkpoint_io/general_checkpoint_io.py): `colossalai/checkpoint_io/general_checkpoint_io.py`
- [Distributed Logging](https://github.com/hpcaitech/colossalai/blob/main/colossalai/logging/__init__.py): `colossalai/logging/__init__.py`

### Other components

- **Inference Client**

## Connections

- Inference Client → LLM Engine: submits prompts
- LLM Engine → Inference Config: uses settings (evidence: `colossalai/inference/core/llm_engine.py`)
- LLM Engine → Request Batching: batches requests (evidence: `colossalai/inference/core/llm_engine.py`)
- LLM Engine → Request State: creates sequences (evidence: `colossalai/inference/core/llm_engine.py`)
- LLM Engine → Token Sampling: selects tokens (evidence: `colossalai/inference/core/llm_engine.py`)
- LLM Engine → KV Cache: allocates cache
- LLM Engine → Shardformer Policies: shards model (evidence: `colossalai/inference/core/llm_engine.py`)
- Async Engine → LLM Engine: delegates inference (evidence: `colossalai/inference/core/async_engine.py`)
- RPC Engine → RPC Worker: dispatches work (evidence: `colossalai/inference/core/rpc_engine.py`)
- RPC Engine → Request Batching: uses batching (evidence: `colossalai/inference/core/rpc_engine.py`)
- RPC Engine → Shardformer Policies: uses policies (evidence: `colossalai/inference/core/rpc_engine.py`)
- Shardformer Policies → Pipeline Stages: uses stage manager (evidence: `colossalai/shardformer/policies/base_policy.py`)
- Pipeline Stages → Process Groups: uses process mesh (evidence: `colossalai/pipeline/stage_manager.py`)
- Pipeline Schedule → Process Groups: communicates stages (evidence: `colossalai/legacy/engine/schedule/_pipeline_schedule.py`)
- Pipeline Schedule → Pipeline Stages: schedules pipeline
- Offload Manager → Gemini Chunks: coordinates chunks
- LLM Engine → Accelerator API: selects device (evidence: `colossalai/inference/core/llm_engine.py`)
- LLM Engine → Distributed Logging: logs runtime (evidence: `colossalai/inference/core/llm_engine.py`)
