# mlc-ai/mlc-llm architecture

> Architecture diagram of the GitHub repository mlc-ai/mlc-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mlc-ai/mlc-llm
- Repository: https://github.com/mlc-ai/mlc-llm

## Overview

MLC LLM compiles and deploys language models through a shared inference engine, exposed through Python, REST, and mobile APIs. A typical workflow prepares model weights and configuration, compiles a model for a target, then submits prompts and streams generated responses. The graph also preserves documented embedding and cross-platform client capabilities. Relationships are limited to README claims and the supplied source excerpts or index; several listed subsystems were not sampled, so their internal wiring is omitted.

Main entry points: [`cpp/serve/engine.cc`](https://github.com/mlc-ai/mlc-llm/blob/main/cpp/serve/engine.cc) (Serving Engine), [`python/mlc_llm/serve/engine.py`](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/serve/engine.py) (Python Engine API), [`python/mlc_llm/cli`](https://github.com/mlc-ai/mlc-llm/tree/main/python/mlc_llm/cli) (CLI Commands), [`cpp/json_ffi/json_ffi_engine.cc`](https://github.com/mlc-ai/mlc-llm/blob/main/cpp/json_ffi/json_ffi_engine.cc) (JSON FFI Engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Model Preparation"]
  node_cli["CLI Commands"]
  node_weights["Weight Conversion<br/>[convert_weight.py]"]
  node_model_arch["Model Architectures"]
  node_model_config["Model Configuration<br/>[gen_config.py]"]
end

subgraph group_compile["Compilation"]
  node_compiler["Compiler Passes<br/>[pipeline.py]"]
  node_compiled_model["Compiled Model"]
end

subgraph group_inference["Inference Runtime"]
  node_native_ffi["JSON FFI Engine<br/>[json_ffi_engine.cc]"]
  node_native_engine["Serving Engine<br/>[engine.cc]"]
  node_engine_state["Request State<br/>[engine_state.h]"]
  node_tokenizer["Tokenization<br/>[tokenizers.cc]"]
  node_generation["Generation Actions"]
  node_embedding["Embedding Engine"]
end

subgraph group_interfaces["APIs and Clients"]
  node_python_api["Python Engine API<br/>[engine.py]"]
  node_openai_protocol["OpenAI Protocol"]
  node_rest_api["REST Server"]
  node_mobile_clients["Mobile Clients<br/>[MLCEngine.kt]"]
end

node_developer(("Model Developer"))
node_platforms{{"Target Platforms"}}
node_user(("API Consumer"))

node_developer -->|"prepares models"| node_cli
node_cli -.->|"converts weights"| node_weights
node_cli -.->|"generates config"| node_model_config
node_cli -.->|"compiles models"| node_compiler
node_model_arch -.->|"supplies model code"| node_compiler
node_weights -.->|"provides weights"| node_compiled_model
node_compiler -.->|"produces"| node_compiled_model
node_model_config -.->|"configures"| node_compiled_model
node_user -->|"submits requests"| node_python_api
node_user -->|"sends requests"| node_rest_api
node_user -->|"sends requests"| node_mobile_clients
node_python_api -->|"uses protocol"| node_openai_protocol
node_python_api -.->|"invokes engine"| node_native_ffi
node_rest_api -.->|"serves requests"| node_python_api
node_mobile_clients -.->|"bridges requests"| node_native_ffi
node_native_ffi -->|"submits requests"| node_native_engine
node_native_engine -->|"updates state"| node_engine_state
node_native_engine -.->|"dispatches actions"| node_generation
node_native_ffi -.->|"streams tokens"| node_tokenizer
node_native_engine -->|"runs inference"| node_compiled_model
node_native_engine -->|"runs on"| node_platforms
node_python_api -->|"streams responses"| node_user
node_native_engine -.->|"supports embeddings"| node_embedding

click node_cli "https://github.com/mlc-ai/mlc-llm/tree/main/python/mlc_llm/cli"
click node_weights "https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/interface/convert_weight.py"
click node_model_arch "https://github.com/mlc-ai/mlc-llm/tree/main/python/mlc_llm/model"
click node_model_config "https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/interface/gen_config.py"
click node_compiler "https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/compiler_pass/pipeline.py"
click node_python_api "https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/serve/engine.py"
click node_openai_protocol "https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/protocol/openai_api_protocol.py"
click node_native_ffi "https://github.com/mlc-ai/mlc-llm/blob/main/cpp/json_ffi/json_ffi_engine.cc"
click node_native_engine "https://github.com/mlc-ai/mlc-llm/blob/main/cpp/serve/engine.cc"
click node_engine_state "https://github.com/mlc-ai/mlc-llm/blob/main/cpp/serve/engine_state.h"
click node_tokenizer "https://github.com/mlc-ai/mlc-llm/blob/main/cpp/tokenizers/tokenizers.cc"
click node_generation "https://github.com/mlc-ai/mlc-llm/tree/main/cpp/serve/engine_actions"
click node_embedding "https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/serve/embedding_engine.py"
click node_rest_api "https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/serve/entrypoints/openai_entrypoints.py"
click node_mobile_clients "https://github.com/mlc-ai/mlc-llm/blob/main/android/mlc4j/src/main/java/ai/mlc/mlcllm/MLCEngine.kt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_weights,node_model_arch,node_model_config toneBlue
class node_compiler,node_compiled_model toneAmber
class node_native_ffi,node_native_engine,node_engine_state,node_tokenizer,node_generation,node_embedding,node_user toneMint
class node_python_api,node_openai_protocol,node_rest_api,node_mobile_clients toneRose
class node_developer,node_platforms toneIndigo
```

## Components

### Model Preparation

- [CLI Commands](https://github.com/mlc-ai/mlc-llm/tree/main/python/mlc_llm/cli): `python/mlc_llm/cli`
- [Weight Conversion](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/interface/convert_weight.py): `python/mlc_llm/interface/convert_weight.py`
- [Model Architectures](https://github.com/mlc-ai/mlc-llm/tree/main/python/mlc_llm/model): `python/mlc_llm/model`
- [Model Configuration](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/interface/gen_config.py): `python/mlc_llm/interface/gen_config.py`

### Compilation

- [Compiler Passes](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/compiler_pass/pipeline.py): `python/mlc_llm/compiler_pass/pipeline.py`
- **Compiled Model**

### Inference Runtime

- [JSON FFI Engine](https://github.com/mlc-ai/mlc-llm/blob/main/cpp/json_ffi/json_ffi_engine.cc): `cpp/json_ffi/json_ffi_engine.cc`
- [Serving Engine](https://github.com/mlc-ai/mlc-llm/blob/main/cpp/serve/engine.cc): `cpp/serve/engine.cc`
- [Request State](https://github.com/mlc-ai/mlc-llm/blob/main/cpp/serve/engine_state.h): `cpp/serve/engine_state.h`
- [Tokenization](https://github.com/mlc-ai/mlc-llm/blob/main/cpp/tokenizers/tokenizers.cc): `cpp/tokenizers/tokenizers.cc`
- [Generation Actions](https://github.com/mlc-ai/mlc-llm/tree/main/cpp/serve/engine_actions): `cpp/serve/engine_actions`
- [Embedding Engine](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/serve/embedding_engine.py): `python/mlc_llm/serve/embedding_engine.py`

### APIs and Clients

- [Python Engine API](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/serve/engine.py): `python/mlc_llm/serve/engine.py`
- [OpenAI Protocol](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/protocol/openai_api_protocol.py): `python/mlc_llm/protocol/openai_api_protocol.py`
- [REST Server](https://github.com/mlc-ai/mlc-llm/blob/main/python/mlc_llm/serve/entrypoints/openai_entrypoints.py): `python/mlc_llm/serve/entrypoints/openai_entrypoints.py`
- [Mobile Clients](https://github.com/mlc-ai/mlc-llm/blob/main/android/mlc4j/src/main/java/ai/mlc/mlcllm/MLCEngine.kt): `android/mlc4j/src/main/java/ai/mlc/mlcllm/MLCEngine.kt`

### Other components

- **Model Developer**
- **Target Platforms**
- **API Consumer**

## Connections

- Model Developer → CLI Commands: prepares models (evidence: `README.md`)
- CLI Commands → Weight Conversion: converts weights
- CLI Commands → Model Configuration: generates config
- CLI Commands → Compiler Passes: compiles models
- Model Architectures → Compiler Passes: supplies model code
- Weight Conversion → Compiled Model: provides weights
- Compiler Passes → Compiled Model: produces
- Model Configuration → Compiled Model: configures
- API Consumer → Python Engine API: submits requests (evidence: `README.md`)
- API Consumer → REST Server: sends requests (evidence: `README.md`)
- API Consumer → Mobile Clients: sends requests (evidence: `README.md`)
- Python Engine API → OpenAI Protocol: uses protocol (evidence: `python/mlc_llm/serve/engine.py`)
- Python Engine API → JSON FFI Engine: invokes engine
- REST Server → Python Engine API: serves requests
- Mobile Clients → JSON FFI Engine: bridges requests
- JSON FFI Engine → Serving Engine: submits requests (evidence: `cpp/json_ffi/json_ffi_engine.cc`)
- Serving Engine → Request State: updates state (evidence: `cpp/serve/engine.cc`)
- Serving Engine → Generation Actions: dispatches actions (evidence: `cpp/serve/engine.cc`)
- JSON FFI Engine → Tokenization: streams tokens
- Serving Engine → Compiled Model: runs inference (evidence: `README.md`)
- Serving Engine → Target Platforms: runs on (evidence: `README.md`)
- Python Engine API → API Consumer: streams responses (evidence: `python/mlc_llm/serve/engine.py`)
- Serving Engine → Embedding Engine: supports embeddings
