How mlc-ai/mlc-llm works
MLC LLM compiles and deploys language models through a shared inference engine, exposed through Python, REST, and mobile APIs. A typical workflow prepares model weights and configuration, compiles a model for a target, then submits prompts and streams generated responses. The graph also preserves documented embedding and cross-platform client capabilities. Relationships are limited to README claims and the supplied source excerpts or index; several listed subsystems were not sampled, so their internal wiring is omitted.
Main entry points: engine.cc Serving Engine · engine.py Python Engine API · cli CLI Commands · json_ffi_engine.cc JSON FFI Engine
Components 19
Model Preparation
- CLI Commands
python/mlc_llm/cli - Weight Conversion
python/mlc_llm/interface/convert_weight.py - Model Architectures
python/mlc_llm/model - Model Configuration
python/mlc_llm/interface/gen_config.py
Compilation
- Compiler Passes
python/mlc_llm/compiler_pass/pipeline.py - Compiled Model
Inference Runtime
- JSON FFI Engine
cpp/json_ffi/json_ffi_engine.cc - Serving Engine
cpp/serve/engine.cc - Request State
cpp/serve/engine_state.h - Tokenization
cpp/tokenizers/tokenizers.cc - Generation Actions
cpp/serve/engine_actions - Embedding Engine
python/mlc_llm/serve/embedding_engine.py
APIs and Clients
- Python Engine API
python/mlc_llm/serve/engine.py - OpenAI Protocol
python/mlc_llm/protocol/openai_api_protocol.py - REST Server
python/mlc_llm/serve/entrypoints/openai_entrypoints.py - Mobile Clients
android/mlc4j/src/main/java/ai/mlc/mlcllm/MLCEngine.kt
Other components
- Model Developer
- Target Platforms
- API Consumer
Connections 23
- Model Developer → CLI Commands: prepares models
README.md - CLI Commands → Weight Conversion: converts weights
- CLI Commands → Model Configuration: generates config
- CLI Commands → Compiler Passes: compiles models
- Model Architectures → Compiler Passes: supplies model code
- Weight Conversion → Compiled Model: provides weights
- Compiler Passes → Compiled Model: produces
- Model Configuration → Compiled Model: configures
- API Consumer → Python Engine API: submits requests
README.md - API Consumer → REST Server: sends requests
README.md - API Consumer → Mobile Clients: sends requests
README.md - Python Engine API → OpenAI Protocol: uses protocol
python/mlc_llm/serve/engine.py - Python Engine API → JSON FFI Engine: invokes engine
- REST Server → Python Engine API: serves requests
- Mobile Clients → JSON FFI Engine: bridges requests
- JSON FFI Engine → Serving Engine: submits requests
cpp/json_ffi/json_ffi_engine.cc - Serving Engine → Request State: updates state
cpp/serve/engine.cc - Serving Engine → Generation Actions: dispatches actions
cpp/serve/engine.cc - JSON FFI Engine → Tokenization: streams tokens
- Serving Engine → Compiled Model: runs inference
README.md - Serving Engine → Target Platforms: runs on
README.md - Python Engine API → API Consumer: streams responses
python/mlc_llm/serve/engine.py - Serving Engine → Embedding Engine: supports embeddings