# inclusionai/ling architecture

> Architecture diagram of the GitHub repository inclusionai/ling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/inclusionai/ling
- Repository: https://github.com/inclusionai/ling

## Overview

This repository is an open-source model delivery and inference/finetuning support repo for the Ling MoE LLM family, not a traditional application service. Its architecture is centered on a custom model implementation plus deployment adapters for multiple serving stacks and training examples for external fine-tuning frameworks.

Main architectural layers

1. Core model implementation
- `models/`
  - `configuration_bailing_moe.py`
  - `modeling_bailing_moe.py`
- This is the repository’s canonical model boundary.
- It defines the Ling / Bailing-MoE architecture as a Hugging Face-style model package: configuration plus model forward logic.
- Everything else in the repo assumes these model definitions exist and are compatible with downstream tooling.

2. Training and fine-tuning examples
- `examples/sft/`
  - YAML configs for supervised fine-tuning, both full fine-tune and LoRA variants.
- `examples/dpo/`
  - YAML configs for preference optimization, again with full and LoRA variants.
- `examples/deepspeed/`
  - DeepSpeed ZeRO configuration files for distributed training and memory optimization.
- These files are not the training framework themselves; they are integration artifacts meant to be consumed by an external trainer, specifically Llama-Factory as described in the README.
- The training path is:
  - dataset/instruction data -> external finetuning framework -> Ling model code -> tuned checkpoints.

3. Inference integrations
The repo supports two major serving/runtime targets, each with its own integration boundary.

A. vLLM integration
- `inference/vllm/bailing_moe.patch`
- This patch adapts vLLM to the Ling MoE architecture.
- The boundary here is the upstream vLLM runtime; Ling is not implemented inside vLLM natively, so the repository provides a compatibility patch.
- Runtime flow:
  - Ling checkpoint + tokenizer/config -> patched vLLM -> offline generation or OpenAI-compatible server.

B. MindIE / Ascend NPU integration
- `inference/mindie/`
- This is the largest deployment subsystem and is split into model-adaptation, weight-conversion, and runtime configuration assets.

Key subparts:
- `inference/mindie/atb_llm/`
  - Custom ATB-LLM model/router code for DeepSeek-style internals adapted for Ling.
  - Files include router, input builder, config, model wrappers, flash attention/casual attention logic, weight utilities, and layer helpers.
  - This directory represents the lowest-level model-runtime integration with the MindIE stack.
- `inference/mindie/patch_atb_llm.sh`
  - Shell patching step that adjusts the MindIE ATB-LLM installation/runtime to accept the Ling model code.
- `inference/mindie/convert_bin_to_safetensor.py`
  - Converts checkpoint weights into safetensors format for MindIE compatibility.
- `inference/mindie/convert_bin_to_safetensor_base.py`
  - Same conversion path for base models.
- `inference/mindie/lite/` and `inference/mindie/plus/`
  - Separate runtime config bundles for the two model families.
  - Each contains:
    - `config.json` / `config.base.json` for the service runtime
    - `model_chat_config.json` / `model_base_config.json` for model-specific config replacement
- This subsystem is meant to bridge model artifacts from Hugging Face/ModelScope into Ascend MindIE service deployment.

Model family split
- The repository explicitly supports two model families:
  - Ling-lite
  - Ling-plus
- It also distinguishes between:
  - chat/instruct models
  - base models
- That distinction matters at deployment time because different config files are selected for each combination, and MindIE treats the chat and base variants differently.

Data and artifact flows

1. Pretrained model consumption
- Source checkpoints are expected from Hugging Face or ModelScope.
- These artifacts include model weights, tokenizer files, and Hugging Face config files.
- The repo does not host the model weights; it hosts the adaptation layer around them.

2. Fine-tuning flow
- External trainer such as Llama-Factory loads:
  - the Ling model implementation from `models/`
  - training recipe from `examples/sft/` or `examples/dpo/`
  - distributed training settings from `examples/deepspeed/` if needed
- Output is a tuned checkpoint that remains in the Hugging Face ecosystem format or a compatible derivative.

3. vLLM inference flow
- Input prompt/chat messages are tokenized using a Hugging Face tokenizer.
- The model is loaded by vLLM, with the repo’s patch enabling support for the Ling MoE structure.
- Serving is either:
  - offline batch inference
  - online API serving
- The important boundary is that all runtime logic stays in vLLM once the patch is applied; Ling-specific behavior is adapted at the model interface layer.

4. MindIE inference flow
- Model artifacts are downloaded locally, then adapted:
  - optional conversion to safetensors for Ling-lite
  - config replacement with repo-provided MindIE configs
  - patching of ATB-LLM runtime components
- For Ling-plus, the focus is on distributed/multi-machine service deployment.
- Requests are served over MindIE’s HTTP endpoints:
  - `/v1/chat/completions` for chat models
  - `/infer` for base models
- This flow crosses several boundaries:
  - model artifact format boundary
  - hardware/runtime boundary (CPU host vs Ascend NPU container vs multi-node fabric)
  - service boundary (MindIE daemon and HTTP API)

Operational/deployment boundaries

- The README shows that MindIE deployment depends on:
  - Ascend drivers and device nodes
  - containerized runtime
  - cluster networking/rank-table configuration for multi-machine deployment
- This means the repo is not just model code; it also carries the deployment contract for Ascend hardware.
- The presence of rank tables, device IPs, and HCCL environment variables indicates that Ling-plus serving is distributed across multiple NPUs and machines.

Important repository boundaries for an architecture diagram

- Core model boundary:
  - `models/` is the canonical Ling/Bailing-MoE implementation.
- Training boundary:
  - `examples/` are adapter configurations for external training systems, not trainers themselves.
- vLLM boundary:
  - upstream vLLM is patched to understand Ling.
- MindIE boundary:
  - a separate serving stack with custom config, weight conversion, and ATB-LLM patches.
- Artifact boundary:
  - source model checkpoints come from external registries; this repo provides compatibility glue and deployment recipes.
- Hardware boundary:
  - MindIE deployment assumes Ascend NPUs and optionally multi-node HCCL communication.

In short, the repository’s architecture is a model-centric integration package with three functional planes:
- model definition,
- training recipes,
- deployment adapters.

The most important diagram would show the Ling model at the center, with separate outgoing paths to:
- external fine-tuning tooling via YAML configs,
- patched vLLM inference,
- MindIE/Ascend inference with weight conversion and runtime patching.

Main entry points: [`models`](https://github.com/inclusionai/ling/tree/HEAD/models) (Ling model), [`inference/vllm/bailing_moe.patch`](https://github.com/inclusionai/ling/blob/HEAD/inference/vllm/bailing_moe.patch) (vLLM patch), [`inference/mindie/atb_llm`](https://github.com/inclusionai/ling/tree/HEAD/inference/mindie/atb_llm) (ATB-LLM layer), [`inference/mindie/convert_bin_to_safetensor.py`](https://github.com/inclusionai/ling/blob/HEAD/inference/mindie/convert_bin_to_safetensor.py) (Weight convert)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Core model"
  ling_model["Ling model<br/>hf model package"]
  ling_config["Model config"]
  ling_forward["Forward pass<br/>model code"]
  source_checkpoints[("Source checkpoints<br/>artifact boundary")]
end

subgraph "Training recipes"
  sft_recipes["SFT recipes<br/>yaml recipes"]
  dpo_recipes["DPO recipes<br/>yaml recipes"]
  deepspeed_recipes["DeepSpeed configs<br/>distributed training cfg"]
  external_trainer(("External trainer<br/>framework boundary"))
end

subgraph "vLLM integration"
  vllm_patch["vLLM patch<br/>runtime patch<br/>[bailing_moe.patch]"]
  vllm_runtime["vLLM runtime<br/>serving stack"]
end

subgraph "MindIE deployment"
  mindie_patch["ATB patch<br/>runtime patch<br/>[patch_atb_llm.sh]"]
  atb_llm["ATB-LLM layer<br/>runtime adapter"]
  weight_convert["Weight convert<br/>artifact tool"]
  weight_convert_base["Base convert<br/>artifact tool"]
  mindie_lite_cfg["Lite configs<br/>deployment cfg"]
  mindie_plus_cfg["Plus configs<br/>deployment cfg"]
  mindie_runtime["MindIE runtime<br/>serving stack"]
end

source_checkpoints -->|"load"| ling_model
ling_config -->|"define"| ling_forward
ling_model -->|"used by"| external_trainer
sft_recipes -->|"configures"| external_trainer
dpo_recipes -->|"configures"| external_trainer
deepspeed_recipes -->|"optimizes"| external_trainer
external_trainer -->|"fine-tunes"| ling_model
ling_model -->|"adapted for"| vllm_patch
vllm_patch -->|"extends"| vllm_runtime
vllm_runtime -->|"serves"| source_checkpoints
mindie_patch -->|"patches"| atb_llm
weight_convert -->|"prepares"| mindie_lite_cfg
weight_convert_base -->|"prepares"| mindie_plus_cfg
atb_llm -->|"runs in"| mindie_runtime
mindie_lite_cfg -->|"configures"| mindie_runtime
mindie_plus_cfg -->|"configures"| mindie_runtime
source_checkpoints -->|"convert"| weight_convert
source_checkpoints -->|"convert"| weight_convert_base
mindie_runtime -->|"serves"| ling_model

click ling_model "https://github.com/inclusionai/ling/tree/master/models"
click ling_config "https://github.com/inclusionai/ling/blob/master/models/configuration_bailing_moe.py"
click ling_forward "https://github.com/inclusionai/ling/blob/master/models/modeling_bailing_moe.py"
click sft_recipes "https://github.com/inclusionai/ling/tree/master/examples/sft"
click dpo_recipes "https://github.com/inclusionai/ling/tree/master/examples/dpo"
click deepspeed_recipes "https://github.com/inclusionai/ling/tree/master/examples/deepspeed"
click vllm_patch "https://github.com/inclusionai/ling/blob/master/inference/vllm/bailing_moe.patch"
click mindie_patch "https://github.com/inclusionai/ling/blob/master/inference/mindie/patch_atb_llm.sh"
click atb_llm "https://github.com/inclusionai/ling/tree/master/inference/mindie/atb_llm"
click weight_convert "https://github.com/inclusionai/ling/blob/master/inference/mindie/convert_bin_to_safetensor.py"
click weight_convert_base "https://github.com/inclusionai/ling/blob/master/inference/mindie/convert_bin_to_safetensor_base.py"
click mindie_lite_cfg "https://github.com/inclusionai/ling/tree/master/inference/mindie/lite"
click mindie_plus_cfg "https://github.com/inclusionai/ling/tree/master/inference/mindie/plus"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class ling_model,ling_config,ling_forward,source_checkpoints toneBlue
class sft_recipes,dpo_recipes,deepspeed_recipes,external_trainer toneAmber
class vllm_patch,vllm_runtime toneMint
class mindie_patch,atb_llm,weight_convert,weight_convert_base,mindie_lite_cfg,mindie_plus_cfg,mindie_runtime toneRose
```

## Components

### Core model

Canonical Ling / Bailing-MoE implementation

- [Ling model](https://github.com/inclusionai/ling/tree/HEAD/models): hf model package · Canonical configuration and forward implementation for Bailing-MoE / Ling · `models`
- [Model config](https://github.com/inclusionai/ling/blob/HEAD/models/configuration_bailing_moe.py): config · Hugging Face-style architecture configuration · `models/configuration_bailing_moe.py`
- [Forward pass](https://github.com/inclusionai/ling/blob/HEAD/models/modeling_bailing_moe.py): model code · Core model layers and generation logic · `models/modeling_bailing_moe.py`
- **Source checkpoints**: artifact boundary · Pretrained weights and tokenizer artifacts from Hugging Face or ModelScope

### Training recipes

Adapters for external fine-tuning frameworks

- [SFT recipes](https://github.com/inclusionai/ling/tree/HEAD/examples/sft): yaml recipes · Supervised fine-tuning configs consumed by external trainers · `examples/sft`
- [DPO recipes](https://github.com/inclusionai/ling/tree/HEAD/examples/dpo): yaml recipes · Preference optimization configs consumed by external trainers · `examples/dpo`
- [DeepSpeed configs](https://github.com/inclusionai/ling/tree/HEAD/examples/deepspeed): distributed training cfg · ZeRO and offload settings for external training pipelines · `examples/deepspeed`
- **External trainer**: framework boundary · Llama-Factory or similar fine-tuning system that consumes repo recipes

### vLLM integration

Patch boundary for upstream vLLM runtime

- [vLLM patch](https://github.com/inclusionai/ling/blob/HEAD/inference/vllm/bailing_moe.patch): runtime patch · Compatibility patch that teaches upstream vLLM the Ling MoE structure · `inference/vllm/bailing_moe.patch`
- **vLLM runtime**: serving stack · Upstream inference engine used after patching

### MindIE deployment

Ascend serving integration and deployment assets

- [ATB patch](https://github.com/inclusionai/ling/blob/HEAD/inference/mindie/patch_atb_llm.sh): runtime patch · Shell patch step for adapting MindIE ATB-LLM to Ling · `inference/mindie/patch_atb_llm.sh`
- [ATB-LLM layer](https://github.com/inclusionai/ling/tree/HEAD/inference/mindie/atb_llm): runtime adapter · Lowest-level model/runtime bridge for MindIE · `inference/mindie/atb_llm`
- [Weight convert](https://github.com/inclusionai/ling/blob/HEAD/inference/mindie/convert_bin_to_safetensor.py): artifact tool · Checkpoint conversion into MindIE-compatible safetensors · `inference/mindie/convert_bin_to_safetensor.py`
- [Base convert](https://github.com/inclusionai/ling/blob/HEAD/inference/mindie/convert_bin_to_safetensor_base.py): artifact tool · Conversion path for base-model checkpoints · `inference/mindie/convert_bin_to_safetensor_base.py`
- [Lite configs](https://github.com/inclusionai/ling/tree/HEAD/inference/mindie/lite): deployment cfg · Runtime and model config bundle for Ling-lite deployment · `inference/mindie/lite`
- [Plus configs](https://github.com/inclusionai/ling/tree/HEAD/inference/mindie/plus): deployment cfg · Runtime and model config bundle for Ling-plus deployment · `inference/mindie/plus`
- **MindIE runtime**: serving stack · Ascend NPU service runtime and HTTP inference boundary

## Connections

- Source checkpoints → Ling model: load · Pretrained weights and tokenizer artifacts feed the canonical model implementation
- Model config → Forward pass: define · Configuration defines architectural parameters used by the forward model
- Ling model → External trainer: used by · Training frameworks import the model package during fine-tuning
- SFT recipes → External trainer: configures · SFT YAML recipes parameterize the external trainer
- DPO recipes → External trainer: configures · DPO YAML recipes parameterize the external trainer
- DeepSpeed configs → External trainer: optimizes · DeepSpeed configs are consumed when distributed training is enabled
- External trainer → Ling model: fine-tunes · Trainer updates Ling checkpoints using the repo model code
- Ling model → vLLM patch: adapted for · Ling model structure is made compatible with patched vLLM
- vLLM patch → vLLM runtime: extends · Patch is applied to the upstream inference engine
- vLLM runtime → Source checkpoints: serves · Patched vLLM loads checkpoint artifacts for generation
- ATB patch → ATB-LLM layer: patches · Shell patch adjusts ATB-LLM runtime components for Ling
- Weight convert → Lite configs: prepares · Conversion path produces artifacts expected by the lite deployment bundle
- Base convert → Plus configs: prepares · Base conversion path supports the plus deployment bundle
- ATB-LLM layer → MindIE runtime: runs in · ATB-LLM acts as the model adapter inside MindIE service runtime
- Lite configs → MindIE runtime: configures · Lite config bundle shapes service startup and model selection
- Plus configs → MindIE runtime: configures · Plus config bundle shapes service startup and model selection
- Source checkpoints → Weight convert: convert · Checkpoint artifacts are transformed into MindIE-compatible safetensors
- Source checkpoints → Base convert: convert · Base model artifacts follow a separate conversion path
- MindIE runtime → Ling model: serves · MindIE deployment ultimately serves the Ling family model behavior
