How mlc-ai/web-llm works
WebLLM is a browser-based LLM inference engine. An application creates and loads a model through the engine, then submits chat, completion, or embedding requests and receives results, optionally as a stream. The engine coordinates model assets, tokenization, and inference; worker variants move engine work off the main thread. The README also documents structured generation, model customization, and several browser cache backends. Resumable generation and storage links reflect the sampled engine and session code; other unsampled subsystem wiring is omitted.
Main entry points: engine.ts MLCEngine · session_store.ts Session Store · web_worker.ts Worker Handler · cache_util.ts Model Cache
Components 22
Inference API
- MLCEngine
src/engine.ts - OpenAI Protocols
src/openai_api_protocols/index.ts
Model Runtime
- Chat Pipeline
src/llm_chat.ts - Model Config
src/config.ts - Model Cache
src/cache_util.ts - Integrity Checks
src/integrity.ts - Tokenizer
src/cache_util.ts - WebGPU Runtime
Worker Execution
- Worker Handler
src/web_worker.ts - Worker Client
src/web_worker.ts - Service Worker Engine
src/service_worker.ts - Extension Worker
src/extension_service_worker.ts - Worker Messages
src/message.ts
Resumable Generation
- Resume Coordinator
src/resumable/coordinator.ts - Session Store
src/resumable/session_store.ts - OPFS File Store
src/resumable/opfs_file_store.ts - Journal Replay
src/resumable/replay.ts - Resume Probe
src/resumable/session_probe.ts - Checkpoint Writer
src/resumable/checkpoint_writer.ts
Other components
- Web Application
- Model Assets
- Browser Cache
Connections 23
- Web Application → MLCEngine: creates and requests
- MLCEngine → OpenAI Protocols: accepts and returns
- MLCEngine → Model Config: reads model settings
- MLCEngine → Model Cache: loads cached assets
- MLCEngine → Tokenizer: loads tokenizer
- MLCEngine → Integrity Checks: verifies assets
- MLCEngine → Chat Pipeline: runs generation
- MLCEngine → WebGPU Runtime: initializes inference
- Chat Pipeline → WebGPU Runtime: executes model operations
- Model Cache → Browser Cache: reads and writes
- Model Cache → Model Assets: fetches model files
- MLCEngine → Resume Coordinator: coordinates resume
- Resume Coordinator → Session Store: manages sessions
- Session Store → OPFS File Store: reads and writes
- Session Store → Journal Replay: loads replay state
- Session Store → Checkpoint Writer: reads checkpoints
- Resume Probe → Journal Replay: probes replay state
- Web Application → Worker Client: uses worker engine
- Worker Client → Worker Messages: sends requests
- Worker Messages → Worker Handler: dispatches requests
- Worker Handler → MLCEngine: calls engine
- Service Worker Engine → Worker Handler: extends handler
- Extension Worker → Worker Handler: extends handler