How hpcaitech/colossalai works
Colossal-AI is a framework for making large-model training and inference more scalable through distributed execution, sharding, and memory management. The clearest user-to-result path in the available evidence is an inference request entering the LLM engine, which prepares and batches prompts, shards the model, and generates tokens. Separate branches represent training schedules and auto-parallel sharding. The source index and excerpts establish several internal links; other major capabilities are represented without claiming unshown wiring.
Main entry points: llm_engine.py LLM Engine · rpc_engine.py RPC Engine · base_policy.py Shardformer Policies · stage_manager.py Pipeline Stages
Components 20
Inference
- Inference Config
colossalai/inference/config.py - LLM Engine
colossalai/inference/core/llm_engine.py - Async Engine
colossalai/inference/core/async_engine.py - RPC Engine
colossalai/inference/core/rpc_engine.py - Request Batching
colossalai/inference/batch_bucket.py - Request State
colossalai/inference/struct.py - Token Sampling
colossalai/inference/sampler.py - KV Cache
colossalai/inference/kv_cache/kvcache_manager.py - RPC Worker
colossalai/inference/executor/rpc_worker.py
Parallel Execution
- Shardformer Policies
colossalai/shardformer/policies/base_policy.py - Embedding Sharding
colossalai/auto_parallel/tensor_shard/node_handler/embedding_handler.py - Pipeline Stages
colossalai/pipeline/stage_manager.py - Process Groups
colossalai/cluster/process_group_manager.py
Training and Memory
- Pipeline Schedule
colossalai/legacy/engine/schedule/_pipeline_schedule.py - Offload Manager
colossalai/auto_parallel/offload/region_manager.py - Gemini Chunks
colossalai/zero/gemini/chunk/manager.py
Runtime Foundation
- Accelerator API
colossalai/accelerator/__init__.py - Checkpoint IO
colossalai/checkpoint_io/general_checkpoint_io.py - Distributed Logging
colossalai/logging/__init__.py
Other components
- Inference Client
Connections 18
- Inference Client → LLM Engine: submits prompts
- LLM Engine → Inference Config: uses settings
colossalai/inference/core/llm_engine.py - LLM Engine → Request Batching: batches requests
colossalai/inference/core/llm_engine.py - LLM Engine → Request State: creates sequences
colossalai/inference/core/llm_engine.py - LLM Engine → Token Sampling: selects tokens
colossalai/inference/core/llm_engine.py - LLM Engine → KV Cache: allocates cache
- LLM Engine → Shardformer Policies: shards model
colossalai/inference/core/llm_engine.py - Async Engine → LLM Engine: delegates inference
colossalai/inference/core/async_engine.py - RPC Engine → RPC Worker: dispatches work
colossalai/inference/core/rpc_engine.py - RPC Engine → Request Batching: uses batching
colossalai/inference/core/rpc_engine.py - RPC Engine → Shardformer Policies: uses policies
colossalai/inference/core/rpc_engine.py - Shardformer Policies → Pipeline Stages: uses stage manager
colossalai/shardformer/policies/base_policy.py - Pipeline Stages → Process Groups: uses process mesh
colossalai/pipeline/stage_manager.py - Pipeline Schedule → Process Groups: communicates stages
colossalai/legacy/engine/schedule/_pipeline_schedule.py - Pipeline Schedule → Pipeline Stages: schedules pipeline
- Offload Manager → Gemini Chunks: coordinates chunks
- LLM Engine → Accelerator API: selects device
colossalai/inference/core/llm_engine.py - LLM Engine → Distributed Logging: logs runtime
colossalai/inference/core/llm_engine.py