How osaurus-ai/vmlx-swift works
vmlx-swift is a Swift MLX inference stack for local language, vision-language, multimodal, embedding, and image-generation workloads. The main workflow is an application submitting text or chat input, loading a model and tokenizer, then receiving generated stream events; chat templates, generation controls, cache behavior, and tool/reasoning parsing shape that runtime. Major branches include media processing, distributed execution, and Flux image generation. The README describes the facade’s intended coverage; sampled implementation is limited, so edges for unsampled subsystems reflect documented responsibilities rather than verified call sites.
Main entry points: LLMModel.swift LLM Interface · BatchEngine.swift Batch Generation · VLMModel.swift VLM Runtime · DistributedModePlanner.swift Cluster Planning
Components 24
Application API
- VMLX Facade
Libraries/VMLX/VMLX.swift - Chat Sessions
Libraries/MLXLMCommon/ChatSession.swift - Text And Events
Libraries/MLXLMCommon/GenerationActivity.swift
Language Runtime
- LLM Interface
Libraries/MLXLLM/LLMModel.swift - Model Loading
Libraries/MLXLMCommon/Load.swift - Model Factory
Libraries/MLXLMCommon/ModelFactory.swift - Language Families
Libraries/MLXLLM/Models/Llama.swift - Tokenizer
Libraries/MLXLMCommon/Tokenizer.swift - Chat Templates
Libraries/MLXLMCommon/ChatTemplateResolution.swift - Batch Generation
Libraries/MLXLMCommon/BatchEngine/BatchEngine.swift - Generation Caches
Libraries/MLXLMCommon/Cache/CacheCoordinator.swift - Reasoning And Tools
Libraries/MLXLMCommon/Tool/ToolCallProcessor.swift - Generation Stream
Libraries/MLXLMCommon/GenerationStreamRouting.swift
Media Models
- VLM Runtime
Libraries/MLXVLM/VLMModel.swift - Media Processing
Libraries/MLXVLM/MediaProcessing.swift - Vision And Omni Models
Libraries/MLXVLM/Models/Qwen35.swift - Embedding Pipeline
Libraries/MLXEmbedders/Model2VecStaticEmbeddingPipeline.swift
Distributed Runtime
- Cluster Planning
Libraries/MLXDistributedCore/DistributedModePlanner.swift - Distributed Collectives
Libraries/MLXDistributedTP/Collectives.swift - Pipeline Transport
Libraries/MLXDistributedTransport/PipelineStageClient.swift
Image Generation
- Flux Engine
Libraries/vMLXFlux/FluxEngine.swift - Flux Model Families
Libraries/vMLXFluxModels/Exports.swift - Image Events
Libraries/vMLXFluxKit/Requests.swift
Other components
- Embedding Application
Connections 23
- Embedding Application → VMLX Facade: imports API
- VMLX Facade → Chat Sessions: exposes chat
- Chat Sessions → LLM Interface: submits turns
- LLM Interface → Model Loading: loads model
- Model Loading → Model Factory: constructs
- Model Factory → Language Families: selects family
- LLM Interface → Tokenizer: encodes input
- LLM Interface → Chat Templates: resolves template
- LLM Interface → Batch Generation: generates
- Batch Generation → Generation Caches: reads and writes
- Batch Generation → Reasoning And Tools: routes output
- Batch Generation → Generation Stream: emits events
- Generation Stream → Text And Events: yields events
- Embedding Application → VLM Runtime: submits media input
- VLM Runtime → Media Processing: prepares media
- VLM Runtime → Vision And Omni Models: runs model
- Embedding Application → Embedding Pipeline: requests vectors
- Embedding Application → Cluster Planning: plans cluster
- Cluster Planning → Distributed Collectives: configures execution
- Cluster Planning → Pipeline Transport: coordinates stages
- Embedding Application → Flux Engine: requests image
- Flux Engine → Flux Model Families: dispatches model
- Flux Engine → Image Events: streams events