How inclusionai/ling works
This repository is an open-source model delivery and inference/finetuning support repo for the Ling MoE LLM family, not a traditional application service. Its architecture is centered on a custom model implementation plus deployment adapters for multiple serving stacks and training examples for external fine-tuning frameworks.
Main architectural layers
1. Core model implementation
• models/
- configuration_bailing_moe.py
- modeling_bailing_moe.py
• This is the repository’s canonical model boundary.
• It defines the Ling / Bailing-MoE architecture as a Hugging Face-style model package: configuration plus model forward logic.
• Everything else in the repo assumes these model definitions exist and are compatible with downstream tooling.
2. Training and fine-tuning examples
• examples/sft/
- YAML configs for supervised fine-tuning, both full fine-tune and LoRA variants.
• examples/dpo/
- YAML configs for preference optimization, again with full and LoRA variants.
• examples/deepspeed/
- DeepSpeed ZeRO configuration files for distributed training and memory optimization.
• These files are not the training framework themselves; they are integration artifacts meant to be consumed by an external trainer, specifically Llama-Factory as described in the README.
• The training path is:
- dataset/instruction data -> external finetuning framework -> Ling model code -> tuned checkpoints.
3. Inference integrations
The repo supports two major serving/runtime targets, each with its own integration boundary.
A. vLLM integration
• inference/vllm/bailing_moe.patch
• This patch adapts vLLM to the Ling MoE architecture.
• The boundary here is the upstream vLLM runtime; Ling is not implemented inside vLLM natively, so the repository provides a compatibility patch.
• Runtime flow:
- Ling checkpoint + tokenizer/config -> patched vLLM -> offline generation or OpenAI-compatible server.
B. MindIE / Ascend NPU integration
• inference/mindie/
• This is the largest deployment subsystem and is split into model-adaptation, weight-conversion, and runtime configuration assets.
Key subparts:
• inference/mindie/atb_llm/
- Custom ATB-LLM model/router code for DeepSeek-style internals adapted for Ling.
- Files include router, input builder, config, model wrappers, flash attention/casual attention logic, weight utilities, and layer helpers.
- This directory represents the lowest-level model-runtime integration with the MindIE stack.
• inference/mindie/patch_atb_llm.sh
- Shell patching step that adjusts the MindIE ATB-LLM installation/runtime to accept the Ling model code.
• inference/mindie/convert_bin_to_safetensor.py
- Converts checkpoint weights into safetensors format for MindIE compatibility.
• inference/mindie/convert_bin_to_safetensor_base.py
- Same conversion path for base models.
• inference/mindie/lite/ and inference/mindie/plus/
- Separate runtime config bundles for the two model families.
- Each contains:
- config.json / config.base.json for the service runtime
- model_chat_config.json / model_base_config.json for model-specific config replacement
• This subsystem is meant to bridge model artifacts from Hugging Face/ModelScope into Ascend MindIE service deployment.
Model family split
• The repository explicitly supports two model families:
- Ling-lite
- Ling-plus
• It also distinguishes between:
- chat/instruct models
- base models
• That distinction matters at deployment time because different config files are selected for each combination, and MindIE treats the chat and base variants differently.
Data and artifact flows
1. Pretrained model consumption
• Source checkpoints are expected from Hugging Face or ModelScope.
• These artifacts include model weights, tokenizer files, and Hugging Face config files.
• The repo does not host the model weights; it hosts the adaptation layer around them.
2. Fine-tuning flow
• External trainer such as Llama-Factory loads:
- the Ling model implementation from models/
- training recipe from examples/sft/ or examples/dpo/
- distributed training settings from examples/deepspeed/ if needed
• Output is a tuned checkpoint that remains in the Hugging Face ecosystem format or a compatible derivative.
3. vLLM inference flow
• Input prompt/chat messages are tokenized using a Hugging Face tokenizer.
• The model is loaded by vLLM, with the repo’s patch enabling support for the Ling MoE structure.
• Serving is either:
- offline batch inference
- online API serving
• The important boundary is that all runtime logic stays in vLLM once the patch is applied; Ling-specific behavior is adapted at the model interface layer.
4. MindIE inference flow
• Model artifacts are downloaded locally, then adapted:
- optional conversion to safetensors for Ling-lite
- config replacement with repo-provided MindIE configs
- patching of ATB-LLM runtime components
• For Ling-plus, the focus is on distributed/multi-machine service deployment.
• Requests are served over MindIE’s HTTP endpoints:
- /v1/chat/completions for chat models
- /infer for base models
• This flow crosses several boundaries:
- model artifact format boundary
- hardware/runtime boundary (CPU host vs Ascend NPU container vs multi-node fabric)
- service boundary (MindIE daemon and HTTP API)
Operational/deployment boundaries
• The README shows that MindIE deployment depends on:
- Ascend drivers and device nodes
- containerized runtime
- cluster networking/rank-table configuration for multi-machine deployment
• This means the repo is not just model code; it also carries the deployment contract for Ascend hardware.
• The presence of rank tables, device IPs, and HCCL environment variables indicates that Ling-plus serving is distributed across multiple NPUs and machines.
Important repository boundaries for an architecture diagram
• Core model boundary:
- models/ is the canonical Ling/Bailing-MoE implementation.
• Training boundary:
- examples/ are adapter configurations for external training systems, not trainers themselves.
• vLLM boundary:
- upstream vLLM is patched to understand Ling.
• MindIE boundary:
- a separate serving stack with custom config, weight conversion, and ATB-LLM patches.
• Artifact boundary:
- source model checkpoints come from external registries; this repo provides compatibility glue and deployment recipes.
• Hardware boundary:
- MindIE deployment assumes Ascend NPUs and optionally multi-node HCCL communication.
In short, the repository’s architecture is a model-centric integration package with three functional planes:
• model definition,
• training recipes,
• deployment adapters.
The most important diagram would show the Ling model at the center, with separate outgoing paths to:
• external fine-tuning tooling via YAML configs,
• patched vLLM inference,
• MindIE/Ascend inference with weight conversion and runtime patching.
Main entry points: models Ling model · bailing_moe.patch vLLM patch · atb_llm ATB-LLM layer · convert_bin_to_safetensor.py Weight convert
Components 17
Core model
Canonical Ling / Bailing-MoE implementation
- Ling model · Canonical configuration and forward implementation for Bailing-MoE / Ling
models - Model config · Hugging Face-style architecture configuration
models/configuration_bailing_moe.py - Forward pass · Core model layers and generation logic
models/modeling_bailing_moe.py - Source checkpoints · Pretrained weights and tokenizer artifacts from Hugging Face or ModelScope
Training recipes
Adapters for external fine-tuning frameworks
- SFT recipes · Supervised fine-tuning configs consumed by external trainers
examples/sft - DPO recipes · Preference optimization configs consumed by external trainers
examples/dpo - DeepSpeed configs · ZeRO and offload settings for external training pipelines
examples/deepspeed - External trainer · Llama-Factory or similar fine-tuning system that consumes repo recipes
vLLM integration
Patch boundary for upstream vLLM runtime
- vLLM patch · Compatibility patch that teaches upstream vLLM the Ling MoE structure
inference/vllm/bailing_moe.patch - vLLM runtime · Upstream inference engine used after patching
MindIE deployment
Ascend serving integration and deployment assets
- ATB patch · Shell patch step for adapting MindIE ATB-LLM to Ling
inference/mindie/patch_atb_llm.sh - ATB-LLM layer · Lowest-level model/runtime bridge for MindIE
inference/mindie/atb_llm - Weight convert · Checkpoint conversion into MindIE-compatible safetensors
inference/mindie/convert_bin_to_safetensor.py - Base convert · Conversion path for base-model checkpoints
inference/mindie/convert_bin_to_safetensor_base.py - Lite configs · Runtime and model config bundle for Ling-lite deployment
inference/mindie/lite - Plus configs · Runtime and model config bundle for Ling-plus deployment
inference/mindie/plus - MindIE runtime · Ascend NPU service runtime and HTTP inference boundary
Connections 19
- Source checkpoints → Ling model: load · Pretrained weights and tokenizer artifacts feed the canonical model implementation
- Model config → Forward pass: define · Configuration defines architectural parameters used by the forward model
- Ling model → External trainer: used by · Training frameworks import the model package during fine-tuning
- SFT recipes → External trainer: configures · SFT YAML recipes parameterize the external trainer
- DPO recipes → External trainer: configures · DPO YAML recipes parameterize the external trainer
- DeepSpeed configs → External trainer: optimizes · DeepSpeed configs are consumed when distributed training is enabled
- External trainer → Ling model: fine-tunes · Trainer updates Ling checkpoints using the repo model code
- Ling model → vLLM patch: adapted for · Ling model structure is made compatible with patched vLLM
- vLLM patch → vLLM runtime: extends · Patch is applied to the upstream inference engine
- vLLM runtime → Source checkpoints: serves · Patched vLLM loads checkpoint artifacts for generation
- ATB patch → ATB-LLM layer: patches · Shell patch adjusts ATB-LLM runtime components for Ling
- Weight convert → Lite configs: prepares · Conversion path produces artifacts expected by the lite deployment bundle
- Base convert → Plus configs: prepares · Base conversion path supports the plus deployment bundle
- ATB-LLM layer → MindIE runtime: runs in · ATB-LLM acts as the model adapter inside MindIE service runtime
- Lite configs → MindIE runtime: configures · Lite config bundle shapes service startup and model selection
- Plus configs → MindIE runtime: configures · Plus config bundle shapes service startup and model selection
- Source checkpoints → Weight convert: convert · Checkpoint artifacts are transformed into MindIE-compatible safetensors
- Source checkpoints → Base convert: convert · Base model artifacts follow a separate conversion path
- MindIE runtime → Ling model: serves · MindIE deployment ultimately serves the Ling family model behavior