How apple/coreai-models works
Core AI Models combines Python model-authoring and export tools with Swift on-device runtime libraries and command-line runners. Its principal workflow is selecting a supported model, exporting a Core AI asset bundle and resources, then loading them in an app or runner for inference and task-specific postprocessing. Major branches include language and vision inference, image/video diffusion, speech recognition, detection and segmentation, plus agent skills. Edges reflect README-documented workflows and sampled orchestration; unsampled module wiring is intentionally left out.
Main entry points: CoreAILanguageModel.swift Language model API · model_registry.py Model registry · pipeline.py Model export · primitives Python primitives
Components 21
Authoring and export
- Model recipes
models - Model registry
python/src/coreai_models/model_registry.py - Python primitives
python/src/coreai_models/primitives - Model export
python/src/coreai_models/export/pipeline.py - Diffusion export
python/src/coreai_models/diffusion/pipeline.py - Segmentation export
python/src/coreai_models/segmentation/video_pipeline.py
Language runtime
- Language model API
swift/Sources/CoreAILanguageModels/LanguageModel/CoreAILanguageModel.swift - Inference engines
swift/Sources/CoreAILanguageModels/InferenceEngines - Token decoding
swift/Sources/CoreAILanguageModels/DecodingStrategies - LLM server
swift/Sources/Tools/llm-server
Vision runtime
- Image segmentation
swift/Sources/CoreAIImageSegmenter - Video segmentation
swift/Sources/CoreAIVideoSegmenter - Object detection
swift/Sources/CoreAIObjectDetector
Media runtime
- Image diffusion
swift/Sources/CoreAIDiffusionPipeline/Pipelines - Video diffusion
swift/Sources/CoreAIVideoDiffusionPipeline/Pipelines - Speech recognition
swift/Sources/CoreAISpeech
Agent skills
- Agent skills
skills
Other components
- Model author
- Core AI bundles
- App or runner
- Inference results
Connections 29
- Model author → Model registry: selects model
- Model recipes → Model registry: lists supported
- Model registry → Model export: resolves model
- Model author → Python primitives: authors with
- Python primitives → Model export: supports export
- Model author → Diffusion export: exports pipeline
- Model author → Segmentation export: exports segmentation
- Model export → Core AI bundles: writes model
- Diffusion export → Core AI bundles: writes pipeline
- Segmentation export → Core AI bundles: writes bundle
- App or runner → Language model API: requests generation
- Language model API → Inference engines: runs inference
- Inference engines → Token decoding: generates tokens
- Token decoding → Language model API: returns output
- App or runner → LLM server: sends requests
- LLM server → Language model API: dispatches completion
- App or runner → Image segmentation: submits image
- Image segmentation → Inference results: returns masks
- App or runner → Video segmentation: submits video
- Video segmentation → Inference results: returns tracks
- App or runner → Object detection: submits image
- Object detection → Inference results: returns detections
- App or runner → Image diffusion: requests image
- Image diffusion → Inference results: returns image
- App or runner → Video diffusion: requests video
- Video diffusion → Inference results: returns frames
- App or runner → Speech recognition: submits audio
- Speech recognition → Inference results: returns transcript
- Model author → Agent skills: uses guidance